diff --git a/contents/writing/about/the-battle-of-the-caches/index.md b/contents/writing/about/the-battle-of-the-caches/index.md new file mode 100644 index 0000000..d4fb1ad --- /dev/null +++ b/contents/writing/about/the-battle-of-the-caches/index.md @@ -0,0 +1,95 @@ +--- +title: The Battle of the Caches +author: Brett Langdon +date: 2013-08-01 +template: article.jade +--- + +A co-worker and I set out to each build our own http proxy cache. +One of them was written in Go and the other as a C++ plugin for +Kyoto Tycoon. + +--- + +So, I know what most people are thinking: “Not another cache benchmark post, +with skewed or biased results.” But luckily that is not what this post is about; +there are no opinionated graphs showing that my favorite caching system happens +to be better than all the other ones. Instead, this post is about why at work we +decided to write our own API caching system rather than use Varnish +(a tested, tried and true HTTP caching system). + +Let us discuss the problem we have to solve. The system we have is a simple +request/response HTTP server that needs to have very low latency (a few +milliseconds, usually 2-3 on average) and we are adding a third-party HTTP API +call to almost every request that we see. I am sure some people see the issue +right away, any network call is going to add at least a half to a whole millisecond +to your processing time and that is if the two servers are in the same datacenter, +more if they are not. That is just network traffic, now we must rely on the +performance of the third-party API, hoping that they are able to maintain a +consistent response time under heavy load. If, in total, this third-party API call +is adding more than 2 milliseconds response time to each request that our system +is processing then that greatly reduces the capacity of our system. + +THE SOLUTION! Lets use Varnish. This is the logical solution, lets put a caching +system in front of the API. The content we are requesting isn’t changing very often +(every few days, if that) and it can help speed up the added latency from the API +call. So, we tried this but had very little luck; no matter what we tried we could +not get Varnish to respond in under 2 milliseconds per request (which is a main +requirement of solution we were looking for). That means Varnish is out, the next +solution is to write our own caching system. + +Now, before people start flooding the comments calling me a troll or yelling at me +for not trying this or that or some other thing, let me try to explain really why +we decided to write our own cache rather than spend extra days investing time into +Varnish or some other known HTTP cache. We have a fairly specific requirement from +our cache, very low and consistent latency. “Consistent” is the key word that really +matters to us. We decided fairly early on that getting no response on a cache miss +is better for our application than blocking and waiting for a response from the +proxy call. This is a very odd requirement and most HTTP caching systems do not +support it since it almost defeats their purpose (be “slow” 1-2 times so you can be +fast all the other times). As well, HTTP is not a requirement for us, that is, +from the cache to the API server HTTP must be used, but it is not a requirement +that our application calls to the cache using HTTP. Headers add extra bandwidth +and processing that are not required for our application. + +So we decided that our ideal cache would have 3 main requirements: +1. Must have a consistent response time, returning nothing early over waiting for a proper response +2. Support the Memcached Protocol +3. Support TTLs on the cached data + +This behavior works basically like so: Call to cache, if it is a cache miss, +return an empty response and queue the request to a background process to make the +call to the API server, every identical request coming in (until the proxy call +returns a result) will receive an empty response but not add the request to the +queue. As soon as the proxy call returns, update the cache and every identical call +coming in will yield the proper response. After a given TTL consider the data in +the cache to be old and re-fetch. + +This was then seen as a challenge between a co-worker, +Dan Crosta, and myself to see who +can write the better/faster caching system with these requirements. His solution, +entitled “CacheOrBust”, was a +Kyoto Tycoon plugin +written in C++ which simply used a subset of the memcached protocol as well as some +background workers and a request queue to perform the fetching. My solution, +Ferrite, is a +custom server written in Go +(originally written in C) that has the same functionality (except using +goroutines +rather than background workers and a queue). Both servers used +Kyoto Cabinet +as the underlying caching data structure. + +So… results already! As with most fairly competitive competitions it is always a +sad day when there is a tie. Thats right, two similar solutions, written in two +different programming languages yielded similar results (we probably have +Kyoto Cabinet to thank). Both of our caching systems were able to yield us the +results we wanted, **consistent** sub-millisecond response times, averaging about +.5-.6 millisecond responses (different physical servers, but same datacenter), +regardless of whether the response was a cache hit or a cache miss. Usually the +morale of the story is: “do not re-invent the wheel, use something that already +exists that does what you want,” but realistically sometimes this isn’t an option. +Sometimes you have to bend the rules a little to get exactly what your application +needs, especially when dealing with low latency systems, every millisecond counts. +Just be smart about the decisions you make and make sure you have sound +justification for them, especially if you decide to build it yourself.