# Block commercial SEO scrapers that provide no value User-Agent: Bytespider Disallow: / # 2026-11-14: Exa AI search crawler - owner decided to block User-Agent: ExaSearchBot Disallow: / # 2026-11-14: Keenable AI search + ElevenLabs crawlers - owner decided to block User-Agent: KeenableBot Disallow: / User-Agent: ElevenlabsCrawler Disallow: / User-Agent: AhrefsBot Disallow: / User-Agent: MJ12bot Disallow: / User-Agent: SemrushBot Disallow: / User-Agent: DotBot Disallow: / User-Agent: BLEXBot Disallow: / User-Agent: PetalBot Disallow: / User-Agent: serpstatbot Disallow: / User-Agent: DataForSeoBot Disallow: / # Default — allow all pages, no crawl delay: 46k+ URLs want a fast, healthy # crawl, not a self-imposed brake. Google ignores Crawl-delay anyway; Bing and # others would obey it and crawl slower for no benefit. User-Agent: * Disallow: /edit_motorcycle.php Disallow: /add_spec.php Disallow: /contact.php Disallow: /image_review.php Disallow: /tidy.php Disallow: /motorcycles.php Disallow: /admin.php Disallow: /login_action.php Disallow: /signup-check.php Disallow: /imgapprove.php Disallow: /imgdelete.php Disallow: /do.php Disallow: /garage/ # 2026-11-13: query-form compare URLs only redirect to /compare/// (curated pairs are in the sitemap); bots were walking endless id1/id2 combinations Disallow: /compare/? Disallow: /compare_ajax.php Disallow: /mot_check.php Disallow: /garage_api.php Sitemap: https://www.motorcycledb.com/sitemap.xml # AI crawlers are deliberately NOT given their own groups. # # A crawler obeys only its most-specific matching group, so the previous # per-bot "User-Agent: GPTBot / Allow: /" blocks meant those bots never saw # the Disallow list under "User-Agent: *" - they were being invited into # /admin.php, /do.php, /login_action.php and the rest. Allowing is also the # default, so those blocks bought nothing and cost the protections. # # Removing them leaves exactly one policy: the "User-Agent: *" group above, # which every crawler including the AI ones now follows. Public content stays # open to them, admin and action endpoints do not. # # If a specific AI crawler ever needs to be BLOCKED, add a group for it here # that repeats the Disallow list plus "Disallow: /" - do not add an Allow # group, or it will opt that bot out of every rule above.