Blogs on TurboBytes

CDNetworks joins TurboBytes' global Multi-CDN platform

Mon, 16 Oct 2017 00:00:00 +0000

We're excited to announce CDNetworks is joining our Multi-CDN platform! The businesses of our customers will benefit greatly from lower latency and higher availability across the globe and especially in Russia, the Middle-East and emerging markets in Asia. Furthermore, customers who need to deliver content to users in China, but don't have an ICP license, may expect huge performance gains.

Russia

CDNetworks has 15 POPs in Russia, including in Moscow, Yekaterinburg and Novosibirsk. The strong POP presence translates to excellent performance for the tens of millions of users in Russia.

The chart below shows the average CDN response times in Russia for several CDNs, based on our millions of real-world performance tests in the past week.

China

Need great performance in China but don’t have an ICP license? CDNetworks is your solution as they are the best CDN for delivery into mainland China from POPs outside the country.

ICP license holders can benefit from even better performance in China by utilizing CDNetworks’ many POPs in China mainland. This service is not yet available through TurboBytes, but we’re actively looking into this.

The two charts below show the reponse times and failratio of “EdgeCast China” (available through TurboBytes, ICP required), CDNetworks and a few other CDNs. “EdgeCast China” is best because content is delivered from POPs in Beijing and Shanghai, but CDNetworks outperforms all the other CDNs greatly. The average response time of CDNetworks is about 90 ms, while most other CDNs are >200 ms. More importantly, the reliability of CDNetworks is much higher, as shown in the failratio chart (lower is better).

Portugal

CDNetworks is one of the few CDNs that has edge servers in the country, resulting in 30% lower response times.

Turkey

The two POPs of CDNetworks in Ankara and Istanbul clearly make the difference:

The real impact of the DDoS against Dyn

Sat, 22 Oct 2016 18:47:00 +0000

Yesterday, October 21 2016, there was a large DDoS attack against Dyn, one of the leading authoritative DNS providers. The attack started around 11 AM UTC and lasted for hours, severely hurting the reachability of big name sites like Twitter, GitHub and PayPal. It hurt CDN Fastly too. Mainstream media logically picked up on the story and it seems the attack was done using Mirai IoT botnet. The exact nature and scale of this attack is currently not known to the public. 24 hours after the attack ended, it's still a much talked about topic on Twitter.

This article is not about what exactly happened, who was behind the attack or why they did it. We want to show the real impact of the attack on the performance of Dyn's authoritative DNS service globally, using our unique RUM for DNS data. TurboBytes monitors the real-world performance of authoritative DNS providers from across the globe, 24/7, by running tests in the browsers of millions of people that are connected to thousands of networks.

First wave: it was not just US-East and the impact was huge

Dyn official report: "On Friday October 21, 2016 at approximately 11:10 UTC, Dyn came under attack by a large Distributed Denial of Service (DDoS) attack against our Managed DNS infrastructure in the US-East region. Customers affected may have seen regional resolution failures in US-East and intermittent spikes in latency globally. Dyn's engineers were able to successfully mitigate the attack at approximately 13:20 UTC, and shortly after, the attack subsided."

This text tells very little about the level of pain Dyn customers had. You may read it as if only some customers' sites were unreachable for some users (in US-East) and there was an occasional slow response on other continents.
Our RUM for DNS data clearly shows it was pretty bad:

The chart above shows the Failratio of Dyn and other DNS providers as measured through recursive resolvers (Google Public DNS, OpenDNS and ISP resolvers) in the state of New York. Our tests are initiated in the browser and use a random subdomain to force the recursive to get the response from the authoritative.
During the two hour attack, Failratio averaged at 45% and peaked at 80%. In our book, that qualifies as an outage.

If Dyn servers responded, often it did so very slowly.

From our data we can confirm the problems in US were limited to US-East. But what happened outside US? Most countries were just fine, but many users in Germany and France definitely noticed Dyn failing during the first wave of the attack:

Second wave: 5 hours of big pain across continents

Dyn official report: "At roughly 15:50 UTC a second DDoS attack began against the Managed DNS platform. This attack was distributed in a more global fashion. Affected customers may have seen intermittent resolution issues as well as increased global latency. At approximately 17:00 UTC, our engineers were again able to mitigate the attack and service was restored."
That end time of 17:00 is maybe a typo, as the original status post states the attack lasted much longer and the incident was marked as Resolved at 22:17 UTC.
Our data shows the second wave indeed started at roughly 15:50 UTC and ended at ~ 20:30 UTC.

In US the Failratio averaged 32% and again the peak was 80%. Dyn performance was different in other countries, like United Kingdom and France:

In both major markets in Europe, the Failratio peaked at 100% and this peak lasted 30 minutes. This means no query to Dyn got a response! After the peak the Failratio dropped to ~ 10% in United Kingdom but in France it stabilized at a still very high 60%. The France chart also shows performance of other DNS providers was degraded for two hours. After taking a closer look at our data it became clear this pattern showed only on AS3215, which belongs to Orange, the biggest consumer ISP. Dyn was completely down from AS3215. Failratio was stable at 100%. Ouch. This must have impacted millions of people!
Interestingly, for two hours the other major DNS providers also suffered on the network of Orange. Is this related to the attack against Dyn? Maybe. We don't know. Also in Brasil the websites of Dyn customers suffered.

Pulse: instant diagnostics from eyeball networks

During the second wave, we ran some Pulse tests to view the real-world DNS behaviour from 80 machines around the globe, most connected to consumer ISP networks.

The first test ran at 17:33:44 UTC and showed a 70.1% error rate when our Pulse agents tried to query for soundcloud.com directly from Dyn's servers (ns1.p20.dynect.net., ns2.p20.dynect.net., ns3.p20.dynect.net., ns4.p20.dynect.net.).

The second test ran at 17:35:10 UTC. This time the Pulse agents queried for soundcloud.com via Google Public DNS, OpenDNS and their ISP resolvers. The error rate was 40.6%.

The record in question has a TTL of 600 (10 minutes), so logically the error rate was lower when querying recursive resolvers because some could serve the response from cache and did not have to reach out to Dyn. Another possible explanation for the difference in error rates is that in effort to mitigate attack traffic, Dyn may have blocked end user IPs from hitting their nameservers in order to give recursives a higher chance of reaching Dyn. In hindsight, we should have run more Pulse tests using random subdomains to conclusively test recursives' reachability to Dyn.

Closing remarks

This attack was a big one. The impact was tremendous because it lasted for several hours and it was against a leading DNS provider, bringing down many popular websites and online services in nations on multiple continents. TurboBytes is in the unique position to see the real-world impact for users on consumer ISP networks. Our RUM for DNS data does not lie.

April 2 2015, we wrote a blog post Why You Should Use Two DNS Providers. This explains how recursive resolvers work and why using more than one DNS provider makes your website or online service much more reliable and resilient against an attack like this one against Dyn.

The TurboBytes tool Pulse can come in handy in case you want to diagnose DNS performance on eyeball networks worldwide. Next time you think something may be wrong, visit https://pulse.turbobytes.com/.

Consider using OpenDNS as your primary resolver, from home and the office. Their founder David Ulevitch tweeted yesterday about a cool feature that helps you experience the Internet as you expect it to be even if a DNS provider like Dyn is down:

Pro-tip: OpenDNS users generally see the Internet as they should. We do a good job of handing "last known good" IPs when we can't resolve.
— ☁ David Ulevitch ☁ (@davidu) October 21, 2016

Level3 and SwiftServe boost TurboBytes' Multi-CDN performance

Tue, 16 Feb 2016 16:00:00 +0000

We're excited to announce two great CDN providers are joining our Multi-CDN platform: Level3 and SwiftServe. The businesses of our customers will benefit greatly from lower latency and higher availability across the globe and especially in Russia, Latin-America, the Middle-East and emerging markets in Asia. Together, Level3 and SwiftServe add 75 new POP locations to our Multi-CDN network map.

Level3

Level3 owns and operates the worlds largest Tier-1 network and provides a range of products and services, including CDN. Level3 CDN is used by companies like Apple and Netflix to deliver web objects, downloads and video to users worldwide. A big strongpoint of Level3 CDN is the high number of POPs in locations where most other CDNs are not present.

The chart below shows the daily median response time (TTFB) of our 5 CDNs in Russia, South-Africa, Qatar and Chili, based on our millions of real-world performance tests in the past weeks.

SwiftServe

SwiftServe is a high-growth CDN provider with offices in Singapore and UK and a strong focus on Asia. They have partnerships with major ISPs in Vietnam, Thailand, Indonesia and United Arab Emirates and this results in best-in-class performance. SwiftServe is launching new POPs soon in mainland China and India.

CloudFront outage in the UK on BT lasted 6.5 hours

Tue, 25 Aug 2015 13:00:00 +0000

CloudFront had a bad day in the UK yesterday, causing many websites to be unavailable to the millions of people on the BT network. The outage lasted 6.5 hours, during the day. Ouch.
We'll show you the problem was in DNS, and how you - a CloudFront user/customer - can easily and quickly do diagnostics on your CDN, from many consumer ISP networks across the globe.

It's not the first time CloudFront had DNS related availability problems. On Nov 27 2014 CloudFront had a major global outage lasting ~ 90 minutes. Yesterday's CloudFront outage in Great-Brittain was different, because it occurred on a single ISP network only, but it lasted for a very long time. What happened? In short: the DNS resolvers of BT sent 'empty' responses for <something>.cloudfront.net queries, resulting in browsers and apps not being able to connect to a CloudFront CDN server.

RUM does not lie

Here at TurboBytes, we constantly monitor performance of CDNs with RUM (Real User Monitoring) from within browsers of people at home and at work, everywhere in the world. We use this data to power our Multi-CDN service. Our non-blocking JS executes after page load and then silently in the background fetches a 15 KB object from a few CDNs, and beacons the load time details to our servers. If the 15 KB object failed to load within 5 seconds, we beacon a Fail.

In the chart below (UTC time zone), a vertical blue line was drawn for every test that passed (browser fetched 15 KB object from CDN within 5000 ms) and a vertical red line was drawn for every test that failed. Before and after the CloudFront outage it’s clear there is a lot more blue than red. During the outage it’s almost all red.

The Failratio jumps around 10:10 UTC and you can clearly see it’s a hard ‘break’. In the following 6.5 hours some ‘ok’ beacons do come in (likely because some users on BT have configured their machines to use Google Public DNS or OpenDNS), but by far most beacons show failure. Around 16:40 UTC the problem was fixed.

AWS status page

AWS first reported about the problem circa two and a half hours after it started. That’s not great. Also, and surprisingly, it was classified as ‘Informational message’ and not as ‘Performance issues’ or ‘Service disruption’.

How Pulse helped us quickly diagnose the problem

We first heard about the CloudFront outage from a tweet by @OpenRent, who responded to a site visitor and linked to Pulse test results. We then ran a HTTPS test on Pulse against their CloudFront endpoint and all seemed fine. The @OpenRent person gave a good hint to why that HTTPS test did not show the CloudFront problem: “… or the Agent is using a different public DNS”. Ah, yes, that must be it! We then quickly ran a DNS test (scroll down to agent 128-Lee-Armstrong, located in Portsmouth) and this gave insight into what was going on: BT resolvers sent a NOERROR response without an ANSWER section, meaning the client (browser/app) gets no IP address to connect to.

Searching Twitter for “Cloudfront BT” led us to some tweets by @Lovell, a nice guy from London who runs a image resizing web service. Lovell’s tweets one and two gave more insight into what was going on with CloudFront on BT:

We could have (and should have) run Pulse tests to verify Lovell’s findings about that single NS being unavailable, but we didn’t, so let’s safely assume Lovell is right.

Shameless plug

TurboBytes Pulse has grown from 10 to 80+ agents (test locations) in a few months time, and we’re always looking for more agents. Agent hosts get access to the Pulse API so if you want that and you can install the Pulse software on a Linux machine or Raspberry Pi that is connected to a consumer ISP network, please reach out to us.

Introducing TurboBytes Pulse

Mon, 08 Jun 2015 12:20:00 +0000

Is DNS working correctly everywhere for your domains? Are your servers and CDNs serving the desired responses? Are your users taking efficient network paths to your sites and apps?
If you regularly need these questions answered, you're going to love our new service.

TurboBytes Pulse enables you to easily & quickly collect DNS, HTTP(S) and Traceroute responses from computers around the world. Most of these 'agents' are connected to consumer ISP networks. Pulse is free and open-source !

How Pulse works

Pulse is a collection of test machines (we call these agents) and the CNC, the command and control center. Users send a test request to the CNC and agents are then instructed to run the test and send back the results. All communication between the CNC and the agents happens encrypted over TLS.
Learn more about the DNS, HTTP and Traceroute tests in the Pulse FAQ.

Currently tests can only be initiated from the Pulse website, but soon (end of June 2015) we’ll release the API to TurboBytes customers and everybody who hosts an agent.

Pulse currently has 20+ agents, including in San Francisco, Seattle, Vancouver, New York, Manchester, Goteborg, Utrecht, Taipei and Sydney. New agents come online every week and we expect to grow to 100 agents within a few months.

Why we developed Pulse

Pulse is built primarily as a service to our Multi-CDN customers: enable them to quickly find out if the CDNs are behaving correctly everywhere.
For example: a TurboBytes customer receives an email from a website visitor about not being able to load the site. Some questions that will come to mind are: is the CDN completely failing in that user’s country? Is it failing everywhere? Is one particular file not served correctly? Pulse makes it easy to quickly get those questions answered and gain insight in the behavior of the CDNs at that time.

The secondary purpose of Pulse is internal: as a Multi-CDN provider, we need a way to know when new and updated configs have provisioned globally across all the CDN POPs. If the API of CDN X tells us provisioning is done, has it then really completed across all their POPs? We’ve found out it’s best to run some verification checks against all POPs …

So why did we build our own? Couldn’t we have better integrated with an existing service like Pingdom or Catchpoint? We had two good reasons to build Pulse:

Real-World: we don’t want to test from datacenters, but from homes and offices connected to consumer ISP networks. That is where your users are, right?
Control: we have strong views on what Pulse must be able to do and what not.

Future

Pulse is a baby and will grow to maturity over time. We have big plans set out for our new precious !

More agents

Brasil, Spain, Hong Kong, Israel, South Africa. These are just a few countries where we will have Pulse agents soon. Our goal is to have at least one agent connected to each of the top 5 consumer ISP networks in all major countries. In the United States, agents will live in many states. Just one agent on Comcast is not enough, right?
Do you want to have full access to all of Pulse? Host an agent!

API

We always build the API first and then interfaces on top of it, so the Pulse API already exists. Giving users access to Pulse via API is high on our todo list, but we first need to polish a bit and implement things like key management, rate limiting and queueing. We expect to have the API ready in June.
Important: API access is a feature we will make available only to TurboBytes customers and Pulse agent hosts. Everybody else can use Pulse freely via the Web ui at pulse.turbobytes.com.

Features

We don’t want to tell you everything, but lots of good and useful things are coming to Pulse. Something is currently in the making for HTTP/2 …
Stay tuned via @TurboBytesPulse on Twitter.

We always welcome your thoughts, ideas and feedback. Please share below in the comments section or send an email to pulse@turbobytes.com.

Join us at Velocity 2015 in Santa Clara

Mon, 20 Apr 2015 10:00:00 +0000

Velocity conference, organized by O'Reilly, is the place to be to learn how to make your site or app faster and stronger, through attending talks by industry experts and by networking at the evening events, lunch or any other time really. We've been to Velocity many times and it's always super useful and great fun.

This year's first Velocity takes place May 27-29 in sunny Santa Clara and we can't wait to be there. Speakers this year include performance experts Tammy Everts (SOASTA), Ilya Grigorik (Google) and Mark Zeman (SpeedCurve). Dozens of other experts will be on stage sharing their knowledge and learnings.

... and TurboBytes will on stage too! We're honored to give a talk Preparing for CDN Failure: Why and How, together with our friends at Mobify.

The discount code 20TURBO gives you 20% off: register now!

We hope to see you in May.

Global DNS outage at Zerigo: the real-world perspective

Tue, 14 Apr 2015 13:00:00 +0000

On April 11 2015, during US daytime, Zerigo suffered a global DNS outage due to a DDoS attack. The Zerigo status page informs us that the attack hit their origin nameservers ^[1], and gives the impression the problems started before 15:24 UTC and ended at 18:30 UTC. But Zerigo customers kept complaining on Twitter even hours later. What happened, really? Our data tells the story.

Here at TurboBytes we closely monitor the real-world response time and availability of many authoritative DNS providers, including Zerigo, with our RUM for DNS. Every day we run millions of measurements from across the globe, and our data for April 11 clearly shows: Zerigo DNS performance was very poor for about 8 hours globally.

The real-world response times and fail ratio

The DDoS attack hit the Zerigo origin nameservers hard worldwide, resulting in very slow responses and poor availability:

Browsers beacon a Fail when the authoritative was too slow, down or sent a bad response.
Fail Ratio = % of measurements that failed. More info.

14:40 UTC: it all starts

Zerigo first mentions the DDoS attack at 15:24 UTC, but it started well before that, at ~ 14:40 UTC. Fail Ratio jumped and response times went sky high. Between 14:45 and 16:40 the median response time was well over 4 seconds (!) and Fail Ratio was between 40% and 60%.

In reality the Fail Ratio was actually even higher, because the number of beacons we received from browsers running the performance tests for Zerigo dropped by ~ 12%. This is easily explained by the fact that our tests don’t have a set timelimit and if a test takes a long time, some users will navigate to the next page before the test completes or fails.

16:50 UTC: all is well again, but not for long

Two hours after Zerigo DNS peformance went bad, suddenly performance is back to normal. The Zerigo status page: “Valued customers, please be advised that the DDOS attack has been mitigated and all Zerigo services are now restored”. However, within 10 minutes performance goes bad again, and it gets even worse than before.

17:20 UTC: Zerigo DNS completely unavailable

About 30 minutes after performance was back to normal levels, Zerigo DNS is completely down. Fail Ratio hits the 100% mark and response times are very, very high.

18:00 UTC: performance improved

Luckily for Zerigo customers, performance starts to improve soon and at ~ 18:00 UTC response times are much lower and the Fail Ratio is down to ~ 25%.

18:45 UTC: response times back to normal

Global median response time is ~ 260 ms and this is normal for Zerigo. But Fail Ratio is still a whopping 25% and it stays like that for over 2 hours.

21:05 UTC: everything looks good again, but not for long

Fail Ratio is back to normal … but only for about 10 minutes. At 21:30, it’s above 20%.

22:30 UTC: stable

Finally, all problems are gone and Zerigo DNS performance is back to normal levels.

Performance not the same on all networks

So far we’ve taken a global view on Zerigo’s DNS performance. The charts for individual countries look more or less the same as the ones we showed above. When browsing through our data some more, we did spot some interesting things though.

Perhaps one would expect performance to be the same on all networks, because the DDoS attack hit Zerigo origin nameservers directly. If those nameservers are down due to an overload in traffic, this impacts performance on all networks the same, right? Well, this turns out not to be the case.

US - AS7922 (Comcast)

In the global chart we saw that after ~ 18:00 UTC the Fail Ratio was at ~ 25%. On AS7922 however, Zerigo DNS remains almost completely unavailable until 22:30. Fail Ratio does not get below 60%, except for that short 10 minute dip at ~ 21:05.
On some networks in other countries we see the same.

From our data it is clear that resolvers on AS7922 had a harder time getting a response from Zerigo’s nameservers than for example Google Public DNS resolvers (see below). Why? We don’t know.

US - AS15169 (Google)

AS15169 is Google’s network. The Google Public DNS resolvers live on this network. After 18:00 UTC, the Fail Ratio stays relatively low and well below that 20% - 25% we see on AS7922. This is what our data shows for most networks.

Takeaways

Don’t expect timely and insightful information from your provider

Zerigo could have done a better job at informing their customers during the DNS performance problems. Many tweets of customers were not answered and the info on the status page was not awesome. We give a shoutout here to DNSimple. They suffered from a global DDoS attack in December and communicated about this in a proactive, professional way during the outage and wrote a lengthy post-mortem.

All DNS providers have performance issues

DDoS attacks happen and will continue to happen. It’s likely Zerigo will be hit again, and it’s likely other DNS providers will be under attack too. And then there are all sorts of other causes of poor DNS performance (BGP route leaks, broken peering links, …). What can you as a customer of managed DNS do? One thing: use two DNS providers.

Consider using two DNS providers

Here at TurboBytes we use two DNS providers (NSONE and AWS Route53) for extra reliability and speed. Read our blog post about Why You Should Use Two DNS Providers.

Notes:

Since February 2015, Zerigo uses CloudFlare Virtual DNS, which means CloudFlare proxies the DNS traffic and caches the results. If CloudFlare receives a query for an expired or uncached record, it will query the Zerigo origin. Many DNS records have a TTL lower than one 5 minutes (source), so if the origin nameservers are down, that will cause real-world problems for Zerigo customers.
TurboBytes measures authoritative DNS performance using a wildcard A record, so with Zerigo, we always hit their origin nameservers.

April 14, 13:50 UTC: our data shows Zerigo DNS performance has been steadily getting worse in the past 48 hours, with Fail Ratio peaking at 10% at 23:00 UTC on April 13. We’ll keep a close eye out and update this blog post when it makes sense.

We always welcome your thoughts, ideas and feedback. Please share below in the comments section and don't forget to check out our Authoritative DNS Performance Reports.

Why You Should Use Two DNS Providers

Thu, 02 Apr 2015 15:55:00 +0000

Most domains have one DNS provider configured as authoritative. A handful of the big players use two. For example Amazon, TripAdvisor and IMDB use Dyn and UltraDNS. LinkedIn, AOL and eBay use their own nameservers in combination with those of a third party DNS provider.

Here at TurboBytes we use NSONE and AWS Route53 in our Multi-CDN platform, and our friends at MaxCDN recently started using these providers as well. We had two reasons for using two DNS providers:

improve the reliability of our service
improve DNS lookup times

The combined network map of NSONE and AWS Route53 is impressive (view) and most resolvers have a built-in mechanism for quick failover and response time optimization (SRTT; more on this later).

In this article we'll show you - using our RUM for DNS performance data - that using two DNS providers indeed results in lower response times and higher reliability, for example in Brazil, as you can see in our Authoritative DNS Performance Reports.

NSONE and AWS Route53 in Brazil

We'll take a look at two possible situations:

Performance of both providers is 'as usual'
Performance of one provider is significantly worse than normal

Case 1: Performance of both providers is ‘as usual’

Both providers have a POP in Sao Paulo and AWS Route53 also has a POP in Rio de Janeiro. From that alone you’d expect Route53 to outperform NSONE in terms of response time. And that is indeed the case:

The days where both Route53 and NSONE have 'a normal day' (not: March 25 and 26), you can see in the chart that

Route53 was faster than the combo
the combo's response time was almost on par with Route53: the difference is small

This can be explained by the SRTT (Smoothed Round Trip Time) mechanism that most resolvers have: the resolver figures out which nameserver gives best performance and sends most - but not all - queries to that nameserver.

One may expect that the SRTT mechanism also results in a low Fail Ratio. And it does: in this 14 day time frame, the Fail Ratio of the combo was clearly better than that of a single provider.

Our RUM beacons a Fail when the authoritative was too slow, down or sent a bad response. More info.

In the first chart you saw that on March 25 and March 26, Route53 response time increased by ~ 30% and became worse than NSONE's response time. Let's look a bit more closely at dual provider DNS performance in case the performance of one of the providers goes bad.

Case 2: one provider has degraded performance

So far we’ve been taking a country level view. Let’s zoom in on a few networks to have better insight in what happened in Brazil in the past 14 days and especially on March 25 and 26.

AS28573 (NET Servios de Comunicao S.A.)

About 31% of our performance tests in BR were initiated by end users connected to AS28573 and most of them used resolvers on the same network. Route53 response time was not very consistent and most days better than NSONE. On March 25 and 26, Route53’s response time jumped and so did the response time of the combo, but not so much. Do the resolvers here do SRTT? One can argue it’s not crystal clear. Let’s zoom in on those two days in March:

And now we know: resolvers on AS28573 do SRTT.

AS18881 (Global Village Telecom)

AS18881 was the #2 network in Brazil, with 17% of our performance tests initiated by end users connected to that network. Again, most of them used resolvers on the same network. Normally, Route53 and NSONE median response times are not too far apart (5 - 15 ms difference), and it’s clear the resolvers here do SRTT and favored Route53. If the resolvers did not do SRTT, on March 21 the blue line would have peaked too just like NSONE did. Thank you SRTT!

AS7738 (Telemar Norte Leste S.A.)

The #3 network in Brazil for us at that time and - together with AS28573 - the network that made a difference on March 25 and 26: Route53 response times jumped from ~ 130 ms to ~ 215 ms daily median, SRTT kicked in and most queries flowed to NSONE.

Query Share

So far we’ve made statements about resolvers doing SRTT based on performance data only, but we can do better.
In our RUM for DNS, we have a way to track the ‘Query Share’ of each provider in the combo: we know what % of responses were served by NSONE and what % by Route53. See the table below for the Query Share per provider for the 3 aforementioned ASNs for March 31.

ASN	NSONE query share	Route53 query share	Fail %
AS28573	58.51%	41.38%	0.11%
AS18881	20.36%	79.59%	0.05%
AS7738	23.15%	76.77%	0.08%

Conclusions & Summary

Assume all DNS providers have performance degradations

We’ve shown data here for just two providers, NSONE and AWS Route53, but you can be assured the other providers have performance hiccups too (hint: more blog posts coming soon).

SRTT is awesome

Queries from your end users to your authoritative nameservers go through resolvers, and many resolvers have a built-in mechanism for failover and RTT/latency optimization. Isn’t that just great? We think it is.
FYI, not all resolvers do SRTT (hint: more blog posts coming soon).

Performance matters, but it’s not everything

We acknowledge there are several valid reasons not to use more than one DNS provider, but if you care a lot about DNS performance and the reliability of your sites and apps, you should consider using two DNS providers.

We always welcome your thoughts, ideas and feedback. Please share below in the comments section and don't forget to check out our Authoritative DNS Performance Reports.

Introducing RUM for DNS

Tue, 31 Mar 2015 15:00:00 +0000

How good is the response time and availability of your authoritative DNS, really?

To answer that question, it's important to query authoritative nameservers through the resolvers that people use at home, in the office, on mobile and at the local Starbucks. TurboBytes does just that, monitoring the real-world performance of authoritative DNS providers from across the globe, 24/7, by running tests in the browsers of millions of people that are connected to thousands of networks.
We're excited to announce our RUM for DNS !

In this article you'll read about why we built RUM for DNS, our test methodology and the benefits of RUM (Real User Monitoring) versus synthetic monitoring. But maybe you want to skip all that and take a look at some of the data? View the past 14 days performance of CloudFlare, AWS Route53, Dyn and others in our Authoritative DNS Performance Reports.

Why we developed RUM for DNS

TurboBytes runs a global Multi-CDN platform: it closely monitors CDN performance (with RUM) and makes sure traffic is always routed to the best performing CDN. Our platform constantly switches CDNs by changing low TTL CNAME records. Needless to say, our DNS needs to be awesome, with excellent performance and all the features we need.

We had been using Dyn’s DNS platform since 2012 and never had issues with performance, but we did run into must-have functional requirements that Dyn could not meet. Last year in Q2 we started looking into alternatives to Dyn and obviously performance was a key evaluation criterium. We had to be sure the performance of our new DNS provider(s) was good across the globe and we wanted to have a real-world view on authoritative DNS performance, and not benchmark performance based on a handful of tests from a handful of datacenters. We needed ‘RUM for DNS’, so we built it.

How we measure real-world authoritative DNS performance

All TurboBytes Multi-CDN customers add our non-blocking JavaScript snippet to their webpages, which executes after page load and then silently in the background runs tests to measure performance of a few CDN and DNS providers.

We want to give you some insight in what our JS code does for the DNS performance tests and the big challenge we ran into, but we’ll start with laying out our requirements.

Our requirements for RUM for DNS

can measure response time ánd fail ratio (availability)
timing data is accurate
works with all resolvers, including resolvers that do NXDOMAIN hijacking
our JS has no negative impact on the user experience
works at least in Chrome and Chrome for Mobile
is scalable and future-proof

We’re happy to say our solution meets all these requirements.

The challenge

The key challenge we quickly ran into was this: there is no way with JavaScript to instruct the browser to ‘do just a DNS lookup and let me know how long that took’. We first played a bit with dynamically inserting a dns-prefetch link element into the DOM but that was a dead end, simply because the browser does not expose how long the DNS lookup took. It did not take long to decide the only way forward was to use the Resource Timing API. This API exposes timing information for webpage resources. The API is implemented in IE10+, FF35+, Chrome, Opera and the default browser in Android 4.4+. We tested the behaviour of the API in all those browsers and found out that it had serious issues in IE and Firefox (DNS lookup data is missing, wrong or unreliable), and so we implemented a check in our JS to only run our RUM for DNS tests in Chrome and Opera for now.

The good thing about the using the Resource Timing API is that in Chrome and Opera the data is reliable and accurate: we always get the real DNS lookup time. But we also want to reliably detect the Authoritative DNS was too slow/unreachable, down or sent a bad response, so we can track the Fail Ratio/Availability too and not just response time. Read the next section to find out how we accomplished this.

Our solution

TurboBytes’ RUM for DNS test methodology in a nutshell:

fetch a very small object from a TurboBytes webserver - going through resolver only - and get the DNS Lookup Time from the Resource Timing API
if successful, do the same but now going through the authoritative DNS

Resolver HIT test

Before we run tests that go to authoritative, we always first do a test hitting a FQDN with a 24 hrs TTL A record: the resolver hardly ever goes to authoritative because it has the response in cache. We have developed a way in JavaScript to force the browser/OS to go to the resolver, and not use the DNS response from its local cache (magic!). This test must complete within 5000 ms. If not, it’s likely our web server is not in good shape and we then don’t run any performance tests hitting authoritative. If the resolver HIT test does complete within 5000 ms, we know two things:

the time it takes to get a response from resolver (nice to have)
our web server is reachable and responding well

We’re good to go and run tests hitting the authoritative DNS.

Resolver MISS test

Unlike the Resolver HIT test, the Resolver MISS tests don’t have a time limit. We just let it run. Browsers and resolvers do retries and have timeout limits and we just let it run. If the authoritative can’t be reached, was very slow or sent a bad response (not a NOERROR), then at some point in time the browser will receive the SERVFAIL response from the resolver and our JS will then beacon a Fail for the authoritative. The test can’t have failed because of our web server because just a few seconds ago we ran the Resolver HIT test and from that we know our server is reachable and responding just fine. After monitoring performance of several DNS providers for a few months, spotting jumps in Fail Ratio and talking to DNS providers about this, we know for a fact that our Fail Ratio metric is solid.

Benefits of RUM versus synthetic monitoring

There are three important benefits of our RUM for DNS compared to the synthetic monitoring done by for example dnsperf.com, SolveDNS and CloudHarmony:

Relevance: our tests run in the browser of millions of Internet users and go through real-world resolvers to authoritative
Reach: our tests run on many networks and through many different resolvers
Test frequency: our tests run often, not just once per 5 or 15 minutes

In the spotlight: The Netherlands

To give you a feel for our relevance, reach and test frequency, here are some numbers for March 29 2015 based on beacons received from clients in The Netherlands for a single DNS provider:

Metric	Value
Beacons	98134
Unique Client IPs	84981
Unique Client networks (ASNs)	146
Unique Resolver IPs	920
Unique Resolver networks (ASNs)	237

2.2% of all tests went through Google Public DNS (AS15169) and 0.5% of tests hit the authoritative via OpenDNS (AS36692).

Future

More DNS providers

VeriSign, UltraDNS, DNS Made Easy. Those are just some of DNS providers we want to add to our RUM for DNS tracking. Who would you like to see added? Let us know on Twitter!

Increase reach and test frequency

In some countries/on some networks we want to run tests more often. Over the course of the next months we’ll increase count there.

Blog posts about DNS performance

We want to regularly publish blog posts about findings from our RUM for DNS data and things related to (authoritative) DNS performance. In the next article we will probably put the spotlight on the NSONE-Route53 combo.

We always welcome your thoughts, ideas and feedback. Please share below in the comments section and don’t forget to check out our Authoritative DNS Performance Reports.

TurboBytes' Multi-CDN prevents 60 minutes of downtime

Tue, 16 Dec 2014 14:00:34 +0000

No CDN is excellent all the time everywhere. They all have their bad days.

Last week we showed how several CDNs struggle to deliver great performance in France and on Nov 27 CloudFront had a major global outage lasting ~ 90 minutes, allegedly caused by DNS issues. Today we write about another CDN's performance that recently went bad as a result of DNS issues (Highwinds) and we show how our Multi-CDN platform provided great value by reducing downtime by 60 minutes.

DNS breaks: Failratio spikes

Here at TurboBytes, we monitor performance of CDNs with RUM (Real User Monitoring) all the time from across the globe, and we use the data to power our Multi-CDN service. Our non-blocking JS executes after page load and then silently in the background fetches a 15 KB object from a few CDNs, and beacons the load time details to our servers. If the 15 KB object failed to load within 5 seconds, we beacon a Fail.

On Sunday Dec 14 2014 around 14:20 UTC, the Failratio of Highwinds jumped globally and it was not until well over an hour later that the Failratio returned to a normal level. Let’s take a look at what happened with Highwinds and see how TurboBytes’ Multi-CDN platform performed. We zoom in on France because in that country the TurboBytes Multi-CDN platform was routing traffic to Highwinds when the problem started.

Multi-CDN triumphs over single CDN

In the charts below, a vertical blue line was drawn for every test that passed (browser fetched 15 KB object from CDN within 5000 ms) and a vertical red line was drawn for every test that failed to finish within 5000 ms. Before the problem started, it’s clear there is a lot more blue than red, and during the outage, red has the upperhand.

You can see the Failratio increase starting around 14:22 and after 10 minutes it’s almost all red, but some ‘ok’ beacons do keep coming in (our guess is some resolvers will serve a stale response if they can’t get to the authoritative DNS). Around 15:25 - about an hour after the issue started - we started receving a lot more ‘ok’ beacons and the Failratio declined. All in all the downtime lasted about 70 minutes.

How did TurboBytes’ Multi-CDN service perform? Did we switch away from Highwinds quickly? As it turned out, we switched to Highwinds right before the issue started at 14:22. In the next few minutes, more and more traffic started flowing to Highwinds but most traffic was still going to the CDN we previously mapped to. Apparently most DNS resolvers in France were still handing out the CNAME to that previous CDN (we use a DNS TTL of 300 seconds). At 14:25, our platform decided to switch away from Highwinds and it automatically updated our authoritative DNS. It took about 5 minutes, due to the DNS TTL, for traffic to stop flowing to Highwinds alltogether. Conclusion: TurboBytes prevented ~ 60 minutes of downtime in France.

A look outside France

As mentioned, Highwinds suffered from DNS issues not just in France but globally. Below you see charts for Highwinds and TurboBytes in The Netherlands and Global.
Note: the charts for TurboBytes don’t provide great value because we were not routing traffic to Highwinds in The Netherlands at the time of the issue and a global comparison makes little sense because our platform doesn’t make routing decisions at a global level, but I thought it’s best to show them here anyway.

Some final words

How we will improve our Multi-CDN service

We’ve closely analyzed our data and what happened in our platform before, during and after the issue with Highwinds, and from that analysis we have defined a few ways to optimize our service, with the object of switching away from a bad CDN more quickly. One thing we can do is lower the DNS TTL. That is easy and this was already on our To Do list. Another way to improve has to do with how our platform processes the incoming beacons and makes switching decisions: process the data more quickly and make an equally good decision with less data.

Highwinds proactively informed customers: +1

We can’t know from our RUM data why Highwinds was failing, and we did not run any other tests during the time of the incident. We know it was DNS because Highwinds told us. Highwinds informed customers about the incident not long after it started and proactively kept them informed until it was resolved. Highwinds deserves credit for this behavior. Unfortunately, it’s not common for CDN providers to inform customers about content delivery performance degradations.

Not all Highwinds endpoints were broken

We don’t know how many endpoints of Highwinds were unavailable due to the DNS issue. TurboBytes has two endpoints with Highwinds: one for HTTP-only traffic and one for SSL-enabled traffic. Our HTTP-only endpoint was impacted but the other was just fine.

Was your Highwinds endpoint broken on Dec 14? If so, how did you spot it and what action did you take to mitigate the problem? We always welcome your thoughts, ideas and feedback. Please share below in the comments section.

CDNs struggle to deliver great performance in all of France

Tue, 09 Dec 2014 09:21:00 +0000

About two weeks ago, CloudFront had a major global outage lasting ~ 90 minutes. That was a exceptionally widespread outage and those don't happen too often, but regional CDN performance degradations are by no means exceptional. In the spotlight today: France.

France is a country where CDNs struggle to deliver solid performance consistently to everybody. The performance problems vary from an occasional hiccup to frequent slowdowns and elevated failratio. Let's look at a recent timeframe of 5 days, Nov 29 - Dec 03 2014.

Five days, five CDNs struggling

The chart shows the Failratio in France for CDNetworks, Akamai, Limelight, Tata Communications (formerly: Bitgravity) and Fastly, based on a sample of our RUM data. Our RUM sends a Fail beacon if the browser could not completely fetch a highly cacheable 15 KB static object from CDN within 5 seconds.

The data shows elevations in the Failratio for Akamai, CDNetworks, Limelight and Tata Communications on multiple days for part(s) of the day. Fastly is a difference case: Failratio was elevated all day, every day. Let's zoom in on Akamai, CDNetworks and Fastly, to show the problems occur not in all of France, but on one or a few networks only.

Akamai on AS8228 and AS15557

Akamai's performance on two networks of SFR is much worse than on the other networks in France. The charts below give a clear picture. A vertical blue line was drawn for every test that passed (browser fetched 15 KB object CDN within 5000 ms) and a vertical red line was drawn for every test that failed to finish within 5000 ms. The top chart is for all of France, with the two SFR networks excluded. There is a lot more blue than red and this is what you expect from a CDN. For AS8228 and AS15557 however, the charts look very different. Our RUM took more measurements on AS15557 than on AS8228, so the 'shades' of red and blue are different, but we see about the same on both networks: between ~18:15 and 21:15, there is more red than blue.

CDNetworks on AS3215

Orange is a large ISP in France and on Dec 3 2014, we received the highest number of beacons per ASN from clients on their AS3215 network. The first chart below shows CDNetworks' Failratio for France, excluding AS3215. That looks normal. On AS3215, the CDN's performance is normal for the biggest part of the day, but between~ 19:45 and 21:00 there is a lot more red. Not good.

Fastly on AS12322

And then there is Fastly. Fastly's Failratio is fine on all the major networks in France, except on AS12322 of ISP Free. Two charts again, the top showing France excl. AS12322 and below that is a chart for AS12322 only. The Failratio on that network is a stunning 50% throughout the day. What is most striking: Fastly's performance on that network has been that bad for a long time. The chart below - for all of France, not just AS12322 - shows Fastly's Failratio having been about the same at least since Nov 9.

Other CDNs have their bad moments too

As the last chart shows, it's not just Fastly, Akamai and CDNetworks. We already mentioned Limelight and Tata Communications earlier in this article, and you can see EdgeCast is not fully in the clear either, with Failratio elevations on Nov 10 and Nov 13. Let's conclude with this statement: "no CDN delivers excellent performance everywhere all the time". Also true: "All CDNs suffer from significant performance degradations every now and then".

Do you have solid insight in the real world performance of your CDN(s)? Do you use one CDN or multiple? Speak up, we love feedback on our blog and interaction with our readers. Please share below in the comments section.

Global outage of AWS CloudFront CDN on Nov 26 2014

Thu, 27 Nov 2014 15:30:34 +0000

The CDN of Amazon Web Services, CloudFront, experienced a major global outage yesterday due to DNS issues. The CloudFront outage lasted circa 90 minutes, starting at ~ 00:15 UTC (04:15 PM PST). On Twitter and other social media channels (incl. Hacker News) people started talking about it, and the news was picked up by The Next Web, Forbes and other media. The AWS status page did not mark Cloudfront as being in trouble until 45 minutes after the problems started, and surprisingly it was marked as 'informational' and not as 'big problem', or something like that.

The CloudFront outage had a big impact around the globe. Thousands of websites delivered a poor user experience and undoubtedly suffered from a drop in conversion, clicks, sales etc. But the impact does not stop there: banner ads, analytics trackers and widgets did not load, as quite a few of these 'third party content' providers use CloudFront CDN.

Real world performance of CloudFront

Here at TurboBytes, we monitor performance of CDNs with RUM (Real User Monitoring) all the time from all across the globe, and use the data to power our Multi-CDN service. Our customers add our non-blocking JS to their site, which executes after page load. It then silently in the background fetches a 15 KB object from a few CDNs and beacons the load time details to our servers. If the 15 KB object failed to load within 5 seconds, we beacon a Fail.

Our RUM clearly shows how big the CloudFront outage was. The Failratio went sky high, but CloudFront did not reach 1 (=fail all the time). Why not? Well, our RUM data does not tell us exactly what was going on, but from all the info we gathered online, it seems the authoritative DNS of cloudfront.net was not responding *most of the time*. Resolvers often do retries, and apparently, sometimes, one of the authoritative DNS servers would send a good response and the browser would then be able to connect to CloudFront. If indeed the DNS lookup was successful, it was on average much slower than normally:

While looking at the data, we created a visualization that perhaps makes even more clear how things went really bad: In this chart, a vertical blue line was drawn for every test that passed (browser fetched 15 KB object from CloudFront within 5000 ms) and a vertical red line was drawn for every test that failed to finish within 5000 ms. Before the problems started, it's clear there is a lot more blue than red, and during the outage, red has the upperhand.

Was your business impacted by this CloudFront outage? How will you prepare for a similar fail of your CDN in the future? We welcome your thoughts, ideas and feedback. Please share below in the comments section.

Meet us at Velocity Barcelona

Tue, 14 Oct 2014 10:41:24 +0000

Want to learn how to make your site faster and stronger? The O’Reilly Velocity Conference, happening 17 -19 November in Barcelona, Spain, is the place where DevOps, web operations, and performance professionals (from Fortune 500 companies to exciting startups) gather for a legendary learning and networking experience that explores why a faster, stronger web is no longer an option, but a necessity. Hear from the best speakers in the industry, who'll delve into topics ranging from hardcore math and statistics to monitoring, clustering, analytics, and organizational culture.

The TurboBytes team will be present on the two conference days (Tue and Wed) and we'd love to talk to you about CDN and improving web performance.

Velocity is great for meeting new people and learning from the pros.

Fantastic lineup of speakers, incl. Ilya Grigorik, Andrew Betts, Tammy Everts, and many more
Lots of networking opportunities, parties, meeting new people, recruiting team members ...
20% discount with our special discount code TURBO

We hope to see you in Barcelona. Register for Velocity Barcelona 2014 with 20% discount (discount code: TURBO).

Until October 20, you have a chance to win a free 2-day Velocity Barcelona conference pass!

SuperTag goes global, powered by TurboBytes

Wed, 11 Jun 2014 13:36:52 +0000

SuperTag is the leading tag management platform in Australia and rapidly expanding its business on other continents. SuperTag needed a strong content delivery solution and was already convinced multi-CDN is the only way forward. After testing with various solution providers, SuperTag selected TurboBytes.

"We were using a multi-CDN solution but wanted to have better: easy to use, excellent performance globally including in South-America and we wanted the solution to be including the CDNs: one stop shop. We engaged with TurboBytes and found both the team and service to meet all our needs. ", says SuperTag lead engineer Jeremie Leca.

SuperTag is a great customer for TurboBytes. They truly care about performance and content delivery speed & reliability. We feel honored they have chosen to work with us.

SuperTag gives customers insight in CDN usage

The team at SuperTag is always busy making their service better for their customers. In a recent release, SuperTag implemented a tracker that allows the customer to see how much of the SuperTag CDN their account is using. TurboBytes provides SuperTag the raw access logs from the CDNs and SuperTag present the data in an easy to use interface.

About SuperTag

SuperTag was created by Datalicious approximately 4 years ago, currently employs 15 people (and they’re hiring!). SuperTag features include enterprise level workflow and security, support for synchronous and asynchronous tags and the ability to implement complex web tagging solutions such as media attribution and cross domain and conversion tracking. Learn more about SuperTag on supertag.datalicious.com.

SoundCloud executes multi-CDN strategy with TurboBytes

Tue, 24 Sep 2013 14:22:41 +0000

SoundCloud, the world's leading social sound platform, has selected TurboBytes to help them improve the speed and reliability of their content delivery globally. TurboBytes' multi-CDN platform constantly measures the real world performance of SoundCloud's CDNs and makes sure traffic is automatically routed to the best performing CDN.

"SoundCloud has a global audience of millions of users on desktop and mobile and they expect our content to load fast, always. We have researched using multiple CDNs for some time and recently decided to actually do it with TurboBytes, starting with our mobile site. Their platform works smoothly and effectively and the team is great", says SoundCloud VP of Engineering Alexander Grosse.

We are very excited SoundCloud has selected TurboBytes to help execute their multi-CDN strategy. We have met the SoundCloud people regularly at conferences and it's always a pleasure to talk about technology, CDN and performance. We feel honored they have chosen to work with us.

About SoundCloud

SoundCloud is the world’s leading social sound platform where anyone can create sounds and share them everywhere. Recording and uploading sounds to SoundCloud lets people easily share them privately with their friends or publicly to blogs, sites and social networks. SoundCloud is headquartered in Berlin, Germany and has offices in San Francisco, London and Sofia. Learn more about SoundCloud on soundcloud.com.