I Crawled Top 25K Sites For Speed Analysis.

Table of Contents

I decided to scan the top 25K websites on the internet The purpose was to see who all were still implementing the meta keyword tag. As I started crawling these websites the experiment became bigger and bigger. Below I present some of my findings.

I used the top 25K list from Quantcast. Approximately 1120 sites had a hidden profile so I could scan only 23880 websites.

Response times

Approximately 1352 websites failed either a DNS lookup or the connection timed out (slower than 20 seconds).

Here is a table for the response times of these sites.

1 9170 38.4
2 7380 30.9
3 3393 14.21
4 1218 5.1
5 818 3.43
6 196 0.82
7 65 0.27
8 44 0.18
9 187 0.78
10 31 0.13

Approx 92% of the sites opened up in the first 5 seconds.

  • less than 1000 websites were above 5 seconds.

Response codes

  • 5.6% (1352 took longer than expected or had no response.
  • 63.85% had 200 status ok. (2xx)
  • 28.57% used a redirect. ( 3xx )
  • 1.68% (402) websites had a not found error (4xx)
  • 0.23% had a server error (5xx)

Bonus Analysis on analytics software.

I also crawled their web page to find out how many were using Google tag manager. I also checked for others such as

  • Google Tag Manager
  • Google Analytics
  • Doubleclick
  • Marketo
  • Adobe Analytics

This is a relative percentage markup as not each site was tracked or followed. Approx 13K sites provided some sort of a js code to detect what system they were using.

The following is the table breakdown.

Contains: googletagmanager 1243 8.17%
Contains: google-analytics 8553 56.19%
Contains: doubleclick 3218 21.14%
Contains: marketo 126 0.83%
Contains: adobe 986 6.48%



SEO Playbook
Only free for a short period of time.
This field is for validation purposes and should be left unchanged.

Related Posts

E-commerce SEO

Over the past decade, e-commerce has been growing at an incredible rate. This is thanks to many things. Such as

19 Channels of Traction

You’ve created a brand and are ready to start selling your products or services. The next step is to find