Benchmarking Top CDN Providers

James Simpson

• 3 min read
Benchmarking Top CDN Providers

UPDATE: After speaking with CloudFlare's CTO, I've been informed that they don't cache audio files (even static ones) by default. I'm in the process of re-doing the large file test and will update the results accordingly.

UPDATE 2: The charts for the large file tests have been updated and CloudFlare has gone from dead last to fairly even with MaxCDN. The lesson is that if you are going to use CloudFlare, be aware that you need to enable caching of audio files using the Page Rules feature.

Nearly two years ago we ran a 24-hour benchmark comparing CloudFlare vs CloudFront when improving load times for our HTML5 MMORPG. Since then we've seen the emergence of HTTP/2 and a bevy of CDN's stepping up their games, prompting a new set of benchmarks on a broader scale.

We are again using the Node.js CDN benchmark CLI that we open-sourced previously, but this time we are benchmarking: CloudFlare, CloudFront, MaxCDN, CDN77, KeyCDN and CDNsun. These are some of the most well known and widely used providers that serve more than just the "enterprise."

They also offer HTTP/2 support now or in the near future and free SNI SSL support, which is a must-have for our use-case. Some even have baked-in Let's Encrypt support, which isn't vital but is a nifty nice-to-have.

The Test

In order to benchmark a CDN provider, you must record performance over time and not just after a handful of downloads. The main purpose of a CDN is global speed, so it is also important to put multiple POPs to the test when gathering your data.

cdn-bench combined with a few Digital Ocean droplets in different zones makes this trivial. Using the below command, we ran a 24-hour benchmark of all 6 CDNs from 3 locations around the world: Oklahoma City (in our office), London and Frankfurt.

node cdn-bench
    --files https://cloudflare.example.com/large,https://cloudfront.example.com/large,...
    --interval 600
    --limit 144
    --out results.csv

This downloads the listed files, one after the other, every 10 minutes over a 24 hour period. It then appends the download times to a CSV file for analysis. We loaded the same 3 files into each CDN: a small image (12 KB), a medium image (199 KB) and a large audio file (4.1 MB).

The Results

After the tests completed, we combined the CSV output from each server and averaged them into the below charts (click the legends to enable/disable each CDN). These charts help to visualize the relative performance of each CDN over a 24-hour period, which we found particularly useful when comparing the consistency and speed throughout the day.

We also calculated average download times and standard deviations for each. We found the standard deviations especially important, as it is indicative of how consistent our customer experience might be.

Conclusions

The results aren't quite as clear-cut as they were on our first test. CloudFlare seems to have fixed their consistency issues, especially on smaller files. However, they were consistently awful when it came to downloading the large file. CloudFlare appears to be the winner since last time, fixing their consistency issues and improving overall speed and standard deviation across the board. CloudFront (our previous winner) didn't show much difference, if only a little less consistent than before.

The overall winner appears to be one of the standout names in the industry: MaxCDN. They seem to be a solid choice for any type of workload, though we found their pricing to be somewhat deceptive and not nearly as competitive as some of the other providers (unless you have "enterprise" scale). They also don't yet support HTTP/2, though promise it'll be coming later in 2016.

In the end, we decided to move some of our small files back to CloudFlare (where bandwidth costs nothing) and everything else to KeyCDN. Besides free, KeyCDN's pricing was as good as it comes. They were also quite respectable on speed and consistency, include HTTP/2 support and include the aforementioned Let's Encrypt integration.

In the end, we decided to move everything back to CloudFlare from CloudFront. You can't beat free bandwidth, they already have full SSL and HTTP/2 support and the speeds across the board are on-par with the best.

Ultimately, you couldn't go wrong with most of these options depending on your workload and budget needs. This wasn't an all-encompassing test, so if there are other CDN's you think should be tested, we'd love to see your results in the comments!

Sign up for more like this.

Enter your email
Subscribe