Write to us

In development. Sign-up is not open yet.

The BabahSEO-Audit crawler

You may have found this user agent in your server logs. It is the crawler that runs site audits for BabahSEO, a product developed by PT Babah Algo Digital in Indonesia.

BabahSEO is still in development. This page sets out the rules the crawler is built to follow.

What you see in your logs

User agentBabahSEO-Audit/<version> (+https://babahseo.com/bot)

Requests
GET and HEAD only
Rate
1 per second per host
At the same time
2 at most
Time limit
15 s per request
Response size
5 MiB at most
Pages per audit
500 by default, 2,000 at most
robots.txt
Followed, Crawl-delay up to 10 s
Default limits. Your robots.txt can make the crawler slower or keep it to fewer pages.

When it visits

Only after the owner of a site adds it to BabahSEO and proves they control it. The crawler does not roam the web.

Proof of control is one of these:

  • A DNS TXT record at _babahseo-verify.<your domain>
  • A babahseo-verification meta tag on the home page
  • A file at /.well-known/babahseo-verification.txt

The proof is checked again every week. If it is gone, audits pause after a 72-hour grace period.

You did not add your site?

Then someone with access to your DNS, home page or files placed the proof. Remove it to pause audits, or write to us.

How it behaves

It reads public pages of the verified site and nothing else.

What it does

  • Sends only GET and HEAD requests
  • Stays on the host that was verified
  • Follows the robots.txt rules for its user agent and honors Crawl-delay up to 10 seconds
  • Makes 1 request per second per host, with no more than 2 at the same time
  • Stops waiting after 15 seconds and reads no more than 5 MiB of a response
  • Fetches up to 500 URLs per audit by default, and never more than 2,000
  • Skips sensitive paths such as /admin, /wp-admin, /login, /cart, /checkout and /api/, and links with parameters such as token, session or password

What it never does

  • Send cookies or log in
  • Submit forms
  • Follow links or redirects to other hosts, including subdomains such as app. or admin.
  • Run the scripts on your pages
  • Try to get past bot protection

Limit or block it

Add a group for BabahSEO-Audit to the robots.txt file at the root of your site. Pages you disallow are not fetched, and the audit records why they were skipped.

Keep it out of one folder

User-agent: BabahSEO-Audit
Disallow: /private/

Block it from the whole site

User-agent: BabahSEO-Audit
Disallow: /

Slow it down

User-agent: BabahSEO-Audit
Crawl-delay: 10

If you use bot protection

The crawler never tries to get past a challenge page or CAPTCHA. It does not switch user agents, use a headless browser or retry aggressively. It skips the blocked page, marks the audit as partial and tells the site owner how to let it through. To allow audits, allow the BabahSEO-Audit user agent in your bot protection settings.

Questions about the crawler

Write to hello@babahseo.com.

We have not published a list of the crawler's IP addresses yet. Anyone can copy a user agent, so if a request looks wrong, send us the time, the URL and the full log line, and we will look into it.