Featuresrobots.txt & sitemaps

robots.txt and sitemaps. One line can block a whole section.

Every check of a URL also fetches the robots.txt of its host. Search Sentinel stores it, shows what changed line by line, and works out whether Googlebot is still allowed to crawl that URL.

All plansOn by default

01

Every edit, as a line diff

The full file and its HTTP status are stored with every check. When the content changes, or the file disappears, you see the previous and current version next to each other, and the alert lists the key lines that were added or removed.

  • Disallow, Allow, Sitemap, User-agent and Crawl-delay lines are called out in the alert e-mail.
  • History with a date filter and an "only changes" view.
  • Per host: a subdomain's robots.txt is fetched for URLs on that subdomain.
www.example-shop.com/robots.txt · illustrative
User-agent: *
Disallow: /checkout/
Disallow: /search
+ Disallow: /products/
- Sitemap: https://www.example-shop.com/sitemap.xml
+ Sitemap: https://www.example-shop.com/sitemap_index.xml

# 2 lines added, 1 removed · status 200

02

Is Googlebot still allowed?

A Google-compatible parser checks each monitored URL against the rules: user-agent groups, Allow and Disallow, * and $ wildcards, longest match wins. Blocked URLs get a red badge in the URL list.

  • The rule that matched, its type and its line number.
  • The last 20 results per URL, allowed or blocked.
Googlebot crawl permission
URLruleresult
/products/running-shoes/Disallow: /products/ · line 4blocked
/category/sale/no matchallowed
/checkout/Disallow: /checkout/ · line 2blocked
/blog/no matchallowed

03

The sitemaps it lists

Sitemaps named in robots.txt are checked with every check of the URL, and their XML is validated once a day. You hear about it when a sitemap is added, removed, returns another status, or is blocked.

  • Validation: well-formed urlset or sitemapindex, namespace, at most 50,000 URLs and 50 MB, absolute <loc> values, W3C <lastmod> dates.
  • Counts of URLs, or of child sitemaps in an index.
  • Blocked by robots.txt, or served with an X-Robots-Tag.
sitemap status
/sitemap_index.xmlindex: 4 sitemaps · 18 KB · validated 06:12valid200
/sitemap-products.xml12,480 URLs · 2 warnings: lastmod formatvalid200
/sitemap.xmlremoved from robots.txtremoved404

Questions about robots.txt.

How often is robots.txt checked?

With every check of every URL on that host, so as often as your most frequent URL, at most hourly.

Can I turn it off?

Yes, with the project's "robots.txt" tracking setting. It is on by default.

Does it check other crawlers than Googlebot?

The crawl permission check evaluates Googlebot. The file itself, and every change to it, is stored whatever the user-agent groups say.

Are gzipped sitemaps supported?

Not yet; validation reads plain XML sitemaps and does not follow the children of a sitemap index.

Know when robots.txt changes. Before Googlebot does.

Add one URL per host and its robots.txt is watched from the first check.

7 days free, no credit card