Kavra LabWhat makes a bot good
"Good" is not about the technology. A good bot and a bad bot can run the same headless browser. The difference is honesty and purpose: a good bot declares its operator, stays within the operator's published infrastructure, respects your rules about what it may access, and does work that is useful to you or to your users.
- Declared: it names itself in the user agent and links to documentation about what it does.
- Verifiable: its requests come from IP ranges the operator publishes, or carry a cryptographic signature you can check.
- Respectful: it follows robots.txt, keeps a sensible request rate and backs off when asked.
- Useful: search indexing, uptime monitoring, link previews, accessibility checks, or an AI agent acting for a real customer.
Why verification matters
Naming a famous crawler in a user agent costs nothing, so impostors do it all the time. A scraper calling itself a search crawler hopes you have an allow rule for that name. That is why a claimed identity is only the start. Search engines publish their crawler IP ranges and support reverse DNS checks, and a growing number of AI agent operators sign their requests so a site can confirm they are real.
A declared bot that comes from outside its operator's ranges, or fails its signature check, is not a good bot. It is a bad bot wearing a good bot's name.
Examples of good bots
Most sites benefit from some automation. The question is which operators, on which pages.
| Good bot | What it does for you | How to verify it |
|---|---|---|
| Search engine crawler | Gets your pages into search results | Published IP ranges and reverse DNS |
| AI assistant or agent | Answers questions about your products or completes a purchase for a user | Request signatures and published IP ranges |
| Uptime and performance monitor | Alerts you when a page breaks or slows down | Your own vendor's documented addresses |
| Link preview bot | Builds the card shown when your link is shared in chat or social apps | Operator documentation and IP ranges |
| Accessibility and SEO checker | Audits your pages on a schedule you set | Your own account with the vendor |
Good bot vs bad bot
Good bot
- Names its operator honestly
- Comes from published IP ranges or signs its requests
- Follows robots.txt and rate limits
- Indexes, monitors or acts for a real user
- Easy to allow on purpose
Bad bot
- Pretends to be a person or a famous crawler
- Hides behind proxies and spoofed devices
- Ignores robots.txt, rotates to dodge limits
- Scrapes, stuffs credentials, hoards stock or farms bonuses
- Built to pass one check at a time
How to let good bots in safely
Decide per bot and per page. You may want search crawlers everywhere, AI shopping agents on product pages and checkout, and no automation at all on signup or bonus claims. Kavra recognizes verified search crawlers and AI agents by their cryptographic signatures and their operators' published IP ranges, flags declared bots that fail those checks as unverified, and lets you choose to allow, check or block each one. See AI agent detection for how verified agents and impostors are told apart.