DocsMonitoringrobots.txt & sitemaps

robots.txt and sitemaps.

One edit to robots.txt can block a whole section. Search Sentinel fetches it with every check, tells you what changed and whether Googlebot may still crawl each URL.

On this page

robots.txt changes

When Track Robots.txt Changes is on for a project (it is by default), every check of a URL also fetches /robots.txt from that URL's host. Search Sentinel stores the full file and its HTTP status, and logs a change when the content changes or the file becomes unavailable. Subdomains have their own robots.txt and are watched separately.

Because it is fetched with each check, robots.txt is checked as often as the most frequently checked URL on that host.

The Robots & XML tab

Open a URL and choose Robots & XML to see the robots.txt history with a date filter and Only Changes, the HTTP status of each version, and a line-by-line comparison of the previous and current file.

The robots.txt alert lists how many lines were added and removed, and up to three important added and removed directives (Disallow, Allow, Sitemap, User-agent, Crawl-delay).

Googlebot crawl permission

On every check, Search Sentinel tests whether Googlebot may crawl the URL according to the current robots.txt. It follows Google's rules: user-agent groups, Allow and Disallow, the * and $ wildcards, and the longest matching rule wins, with Allow winning a tie.

The result shows as Allowed or Blocked with the matched user agent, the rule type, the pattern and its line number. The last 20 results are listed under "Googlebot URL Crawl Permission Status", and blocked URLs get a red Blocked label in the URL list.

Sitemaps

Search Sentinel finds sitemaps through the Sitemap: lines in robots.txt. On every check it requests each sitemap and records its HTTP status, its X-Robots-Tag, and whether robots.txt blocks it for Googlebot. These events are logged and can trigger alerts:

  • Sitemap Added and Sitemap Removed: a line appeared in or disappeared from robots.txt.
  • Sitemap HTTP Status: the sitemap's status changed.
  • Sitemap Blocking: robots.txt or a nofollow header now blocks the sitemap.

XML validation

Sitemaps are also validated as XML about once a day and after URL checks. The validation checks that the file loads, is at most 50 MB, has a urlset or sitemapindex root, uses the sitemaps.org 0.9 namespace, has at most 50,000 URLs, that every entry has an absolute <loc>, and that <lastmod> uses the W3C date format. The Sitemap Status panel shows the result per sitemap: Valid or Invalid, the number of URLs, the size, and the errors and warnings.

Limits

Gzipped sitemaps are not supported, child sitemaps of a sitemap index are not followed, and validation errors are shown in the panel but do not send an alert. Sitemaps are not used to add URLs to a project.