Thoughts on Varnish

Varnish is getting a lot of attention these days around the internet, and with good reason, it’s a nicely written and speedy cache, and has a nice DSL for caching. It has great features like hot reloading of cache rules and ESI.

One thing that’s really surprised me, though, is that Varnish uses one thread per connection. Most network programs designed for high number of connection don’t use one thread per connection anymore as it has serious drawbacks.

With slow clients, many of your threads are spending a lot of time doing nothing but blocking in write(). In all internet consumer apps, I believe, slow clients make up the majority of your connections. But even though the threads are doing nothing, the OS still has memory and scheduling overhead in dealing with them. You find yourself with an artificially low ceiling on the amount of users you can service with a single machine.

What makes a client slow, though? Both speed and latency. Cell phones, 56k modems, and users on high speed links but not geographically close to your data center can all be classified as ‘slow’.

One design that is more appropriate for dealing with the slow client problem uses a pool of worker threads or processes behind the scene and epoll / kqueue / event ports handling slow clients and telling the pool of workers that a socket is ready with a change notification. Your cost is still correlated with growth but at a much lower rate and the number of users you can service will dramatically increase.

So why does Varnish use this older, more troublesome model? Probably because most services aren’t going to notice the bottleneck; They simply don’t have enough concurrent connections to worry about using a few extra machines. If you’re never saturated a load balancer or firewall, you’ve probably never had to seriously consider the C10k issues involved.

Also, unfortunately, the way most people write load tests is that they are only testing the All Fast Clients scenario and not a mix of fast clients and slow clients. I’m guilty of this, too.

My executive summary: Varnish is a nice piece of software, and I hope they spend the time to make it useful for larger sites as well as smaller ones.