User agent
Requests use the user agent SearchSentinel.app <crawler ID>, for example SearchSentinel.app k3x9q2m7a1bz. The crawler ID is a random 12-character code assigned to your account at sign-up. Because it is unique, you can allow your own checks without allowing anyone else's.
Finding your crawler ID
When a page returns HTTP 403, the URL detail page shows "Crawler Blocked" with the full user agent and a copy button. Copy it from there and give it to whoever manages the firewall or CDN.
Allowlisting
If a firewall, bot protection or CDN blocks the checks, add a rule that allows requests whose user agent contains your full user agent string. Most tools (Cloudflare, Akamai, AWS WAF, Wordfence and the like) support a custom user-agent allow rule. Then click Recheck Now on the URL to confirm.
Note
Allow the user agent, not IP addresses: checks run on cloud infrastructure and the addresses can change.
How a check works
- The URL is requested without following redirects. The status code of this first response is the stored HTTP status. For a 301, 302, 307 or 308, the
Locationheader is stored as the redirect URL. - Redirects are then followed, up to 5 hops, to read the final page.
- The HTML is read as the server sends it. JavaScript is not executed and there is no browser rendering or screenshot.
/robots.txtof the host is fetched, and the sitemaps it lists are requested. See robots.txt & sitemaps.- All elements are extracted and compared with the last successful check.
Limits
- Only
httpandhttpson ports 80 and 443. Private, local and internal addresses cannot be monitored. - The page request times out after 15 seconds; robots.txt and sitemap requests after 10 seconds.
- One location. There is no choice of device or user agent.
Effect on your site and analytics
A check is one request for the HTML plus robots.txt and the listed sitemaps, at the interval you set (at most hourly). Because JavaScript is not run, analytics tools that work in the browser, like Google Analytics, do not count checks. Server log analysis will show them with the user agent above.