DocsReferenceThe Search Sentinel crawler

The crawler. What a check does.

Every check is a plain HTTP request for the page's HTML from Search Sentinel's servers, with a user agent that is unique to your account.

On this page

User agent

Requests use the user agent SearchSentinel.app <crawler ID>, for example SearchSentinel.app k3x9q2m7a1bz. The crawler ID is a random 12-character code assigned to your account at sign-up. Because it is unique, you can allow your own checks without allowing anyone else's.

Finding your crawler ID

When a page returns HTTP 403, the URL detail page shows "Crawler Blocked" with the full user agent and a copy button. Copy it from there and give it to whoever manages the firewall or CDN.

Allowlisting

If a firewall, bot protection or CDN blocks the checks, add a rule that allows requests whose user agent contains your full user agent string. Most tools (Cloudflare, Akamai, AWS WAF, Wordfence and the like) support a custom user-agent allow rule. Then click Recheck Now on the URL to confirm.

Note

Allow the user agent, not IP addresses: checks run on cloud infrastructure and the addresses can change.

How a check works

  1. The URL is requested without following redirects. The status code of this first response is the stored HTTP status. For a 301, 302, 307 or 308, the Location header is stored as the redirect URL.
  2. Redirects are then followed, up to 5 hops, to read the final page.
  3. The HTML is read as the server sends it. JavaScript is not executed and there is no browser rendering or screenshot.
  4. /robots.txt of the host is fetched, and the sitemaps it lists are requested. See robots.txt & sitemaps.
  5. All elements are extracted and compared with the last successful check.

Limits

  • Only http and https on ports 80 and 443. Private, local and internal addresses cannot be monitored.
  • The page request times out after 15 seconds; robots.txt and sitemap requests after 10 seconds.
  • One location. There is no choice of device or user agent.

Effect on your site and analytics

A check is one request for the HTML plus robots.txt and the listed sitemaps, at the interval you set (at most hourly). Because JavaScript is not run, analytics tools that work in the browser, like Google Analytics, do not count checks. Server log analysis will show them with the user agent above.