What does the entire registered internet actually look like?
We're measuring all of it. Every CDN, certificate authority, nameserver, DNS record, and robots.txt across the world's registered domains, published as open, citable statistics. The state of the whole internet, in one place.
378,500,000 domains · 1,593 TLDs · one measurement
Measurement in progress
The first full pass over the internet is running right now across 378.5 million domains, several weeks of continuous measurement. When it completes, this page becomes the index of a growing statistics library: who runs the web's infrastructure, how the modern stack is actually adopted, and which crawlers the internet welcomes or blocks. Until then, you can read what the crawler does and why.
What's coming
publishing after cycle 01CDN & hosting market share
Which CDNs, hosting providers, and nameservers actually run the internet, broken down by country and TLD.
Who blocks which bots
The robots.txt census: most-blocked crawlers, AI bot policies for GPTBot, ClaudeBot & co., and how the web talks to crawlers.
DNSSEC, DMARC & TLS reality
Real-world adoption of DNSSEC, DMARC policies, SPF, CAA, TLS versions and certificate authorities, per TLD.
IPv6, HTTP/3 & the modern web
How far the modern stack has actually spread: IPv6 reachability, HTTP versions, ECH, and HTTPS adoption.
The SaaS census
What verification records and MX data reveal: email provider share, Google vs. Bing webmaster tooling, SaaS adoption by region.
The internet over time
Every measurement cycle adds a data point. Watch adoption curves move and see which technologies are winning and which are quietly dying.
How it works
three requests, once per cycleOur crawler, SmetBot, visits each domain once per measurement cycle and fetches exactly three resources: robots.txt, the declared sitemap, and the homepage. That's enough to describe the technology of the whole internet without being a burden to any single site on it. Results are aggregate only. This is a census, not a directory.