Crawler identification
You've met SmetBot.
If you found this page through your access logs — welcome. SmetBot is the crawler behind Smet.cz, a non-commercial project building open statistics about the world's registered domains.
Opting out is honored automatically: we fetch robots.txt first, and if SmetBot is disallowed, your homepage is never requested — not in this cycle, not in future ones. That's how robots.txt works for any well-behaved crawler, and we're one of them.
What SmetBot does on your site
Exactly three HTTP requests per measurement cycle, reusing a single connection where possible:
| Request | Purpose |
|---|---|
GET /robots.txt | Crawler policy census — which bots the web allows and blocks, sitemap declarations, crawl-delay usage |
GET sitemap | The sitemap declared in robots.txt (or /sitemap.xml), read capped at 512 KB — URL counts and freshness only, never crawled further |
GET / | Homepage only — HTTP status, headers, TLS parameters, and detectable technologies (CDN, server, CMS) |
That's the entire footprint. No deep crawling, no link following, no form submissions, no repeated visits within a cycle. A cycle covers the whole internet and runs over several weeks, so a typical site sees SmetBot for a few seconds, a few times per year.
Why we're doing this
Smet.cz is measuring 378.5 million domains across 1,593 top-level domains — effectively every registered, reachable domain there is. The result is a public dataset answering questions that currently have no citable answer: CDN market share per country, real DNSSEC and DMARC adoption, which AI crawlers the web blocks most, how large and how fresh the web's sitemaps really are, certificate authority distribution per TLD, and more.
Everything we publish is aggregate statistics — counts, shares, and trends, measured on real production websites only: parked, for-sale, and placeholder domains are identified and excluded so the numbers describe the internet people actually use. The project is non-commercial: no data resale, no lead generation, no per-site profiles. Just the census the internet doesn't have.
Crawling principles
- Identified — every request carries the UA above, linking to this page.
- robots.txt first — a disallow means your site is skipped, permanently and automatically.
- Rate-limited per server — strict per-IP concurrency caps, so shared hosting never sees load from us even when it hosts thousands of measured domains.
- Minimal — three small requests per cycle. We typically transfer less data from your site than this page you're reading.
- Reachable — questions or problems: [email protected]. We answer.
For the curious webmaster
Since you're the kind of person who reads user-agent strings — the data this crawl produces is made for you. Once the first cycle completes, smet.cz will publish open stats on exactly the things you check in your own configs: who blocks which bots, TLS and HTTP version adoption, security header usage, and what the rest of the internet's robots.txt files look like. Free to read, free to cite.