The KamataBot crawler 🤖

Hello! This page explains what our crawler does and how to control it.

What is KamataBot?

KamataBot is Kamata’s web crawler. It reads the public pages a business publishes about itself so we can describe companies accurately for B2B research: what a company does, its size and location, its open roles, and the team it lists publicly.

How does it identify itself?

User-Agent: Mozilla/5.0 (compatible; KamataBot/1.0; +https://api.kamata.ai/bot)

Every request is also signed with HTTP Message Signatures (“Web Bot Auth”), so you can verify it really comes from us. Our public keys are at https://api.kamata.ai/.well-known/http-message-signatures-directory.

What do we crawl?

Only public pages about the business, typically:

• https://YOURDOMAIN/robots.txt and your sitemap
• your homepage
• a few pages such as about, team, careers, products, customers and contact

We never log in, fill in or submit forms, read anything behind a password, solve CAPTCHAs, or try to get past anti-bot challenges. We do not crawl your whole site.

How often do we crawl?

• A few pages per site per visit, one request at a time, with pauses between requests (we honour Crawl-delay).
• We come back when pages are likely to have changed, usually every few weeks; pages that rarely change are visited less and less often.
• We slow down when your site answers slowly, and back off on 429 or 5xx answers.

Opting out

We respect robots.txt. To stop us fetching anything from your site:

User-agent: KamataBot
Disallow: /

To keep us out of part of your site only, disallow those paths instead (for example Disallow: /private/). Blocking all bots with User-agent: * and Disallow: / also blocks us. Changes are picked up within a day.

Contact

If you notice a problem or have a question, write to contact@kamata.ai with your domain and we will look into it.