AergosBot

What is AergosBot?

AergosBot is the web crawler operated by Aergos (Aergos.ai), an SEO platform used by digital agencies, consultants, and businesses to audit their own websites and the websites of clients they manage.

AergosBot fetches pages on behalf of Aergos customers so the platform can run technical SEO audits, surface broken links and indexation issues, analyze on-page content and metadata, and monitor changes over time.

Which sites does AergosBot crawl?

AergosBot only crawls sites that an Aergos customer has explicitly added to their account. It does not perform open or speculative crawling of the web.

In practice this means one of the following is true for any site AergosBot visits:

• The site owner is a direct Aergos customer who added it themselves.
• The site owner has hired an agency or consultant who is an Aergos customer and added the site on their behalf.

If you are seeing AergosBot traffic on your site and you are not aware of an arrangement that authorizes it, please contact us at [email protected] and we will investigate immediately.

Identification

User-Agent

All AergosBot requests carry a User-Agent header that identifies the crawler and links back to this page:

Mozilla/5.0 (compatible; AergosBot/1.0; +https://www.aergos.ai/bot)

A Mobile suffix is appended when crawling with a mobile device profile.

IP addresses

AergosBot makes requests from the following IPv4 address:

147.135.31.174

If this list changes we will update this page. Site owners and security teams who need to allow-list AergosBot can rely on this address. We will provide at least 30 days notice on this page before adding additional egress IPs.

Reverse DNS

The IP above resolves via reverse DNS to crawler.aergos.ai, and that hostname forward-resolves back to the same IP. You can use this to verify that traffic claiming to be AergosBot actually originated from our infrastructure:

$ host 147.135.31.174
174.31.135.147.in-addr.arpa domain name pointer crawler.aergos.ai.

$ host crawler.aergos.ai
crawler.aergos.ai has address 147.135.31.174

Behavior and rate

• robots.txt — AergosBot fetches and honors /robots.txt on every site it crawls. Both User-agent: AergosBot and User-agent: * rules are respected. Disallowed paths are not fetched.
• Crawl-Delay — If your robots.txt sets a Crawl-Delay directive, AergosBot honors it. Values above 5 seconds also reduce the crawler's effective concurrency.
• Default rate — In the absence of a Crawl-Delay directive, AergosBot crawls with a default concurrency of 5 and a 200ms delay between requests per host (≈25 requests/second peak per site, sustained lower).
• Per-audit cap — A single audit fetches a bounded set of URLs (default 500, configurable per customer). AergosBot does not perform continuous or open-ended crawling.
• Audit cadence — Audits are triggered on demand by Aergos customers or on a schedule they configure. Most sites see audits run weekly or monthly.
• JavaScript rendering — When required for accurate audits, pages may be rendered with a headless Chromium browser. Rendered requests use the same User-Agent and IP and are subject to the same rate limits.

Controlling AergosBot

Slow it down

Add a Crawl-Delay directive to your robots.txt:

User-agent: AergosBot
Crawl-Delay: 10

Limit which paths it can fetch

Use standard robots.txt Disallow rules:

User-agent: AergosBot
Disallow: /private/
Disallow: /search

Block AergosBot entirely

If you do not want AergosBot to crawl your site at all, add this to your robots.txt:

User-agent: AergosBot
Disallow: /

AergosBot will stop crawling within 24 hours (the maximum cache window for robots.txt). If you need an immediate stop, contact us at [email protected].

Reporting issues

If you believe AergosBot is misbehaving on your site — crawling too aggressively, ignoring robots.txt, or causing load issues — please reach out and we will respond promptly.

• Email: [email protected]
• Include: the affected hostname, an approximate time window, and (if possible) a sample of access-log lines showing the AergosBot User-Agent so we can correlate with our audit records.

Operator

AergosBot is operated by Aergos. The Aergos platform and this crawler are products of MJW Consulting / Aergos. For company information see our About page; for terms of use see Terms and Acceptable Use.