Varnish is a web accelerator written with performance and flexibility in mind. It's modern architecture gives it a significantly better performance than many of it's competing products. Varnish store web pages in memory so the web servers don't have to create the same web page over and over again. The web server only recreate a page when it is changed. Additionally Varnish can serve web pages much faster then any application server is capable of - giving the website a significant speed up.
Varnish is open source but also has a company behind it that develops it and have support contracts.
How fast it is? No one really knows, but for a basic configuration (a single core 2.2Ghz Opteron) it managed to get 27,000 requests/sec consistently.
On one of the latest tests on a rather high end machine (an Intel Xeon X5670 2.9Ghz 6 cores with hyperthreading) it got up to 275,000 requests/sec.
While Varnish may sound like magic there is significant technology written into it that makes it much more than just a web accelerator such as:
Varnish is being used by big companies such as Facebook and Twitter,
We replaced a non caching pricey hardware based load balancer at Aol Answers with Varnish and use it as a load balancer and a caching proxy in a high availability configuration and we LOVE it!