# robots.txt — see Terms of Service Section 4(d) (Automated Access). # AI/LLM crawlers are denied. Recognized public search engines may crawl # and index public pages only. # Last updated: July 30, 2026 # OpenAI (training crawler) User-agent: GPTBot Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # OpenAI (user-initiated fetch) User-agent: ChatGPT-User Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # OpenAI (search index) User-agent: OAI-SearchBot Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Anthropic (training crawler) User-agent: ClaudeBot Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Anthropic (user-initiated fetch) User-agent: Claude-User Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Anthropic (search index) User-agent: Claude-SearchBot Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Anthropic (legacy UA) User-agent: Claude-Web Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Anthropic (legacy UA) User-agent: Claude Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Anthropic (legacy UA) User-agent: Anthropic-AI Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Common Crawl (feeds many AI training sets) User-agent: CCBot Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Google Gemini training User-agent: Google-Extended Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Apple Intelligence training User-agent: Applebot-Extended Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # ByteDance / TikTok User-agent: Bytespider Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Perplexity (crawler) User-agent: PerplexityBot Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Perplexity (user-initiated fetch) User-agent: Perplexity-User Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Meta AI training User-agent: meta-externalagent Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Meta (on-demand fetch) User-agent: Meta-ExternalFetcher Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Amazon User-agent: Amazonbot Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Cohere User-agent: cohere-ai Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # You.com User-agent: YouBot Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Scale AI User-agent: ScaleAI Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Omega AI User-agent: OmegaBot Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Diffbot User-agent: Diffbot Allow: /.well-known/ Allow: /api/mcp_advisor/mcp Disallow: / # Recognized public search engines — public pages only. Bingbot is # deliberately unblocked (the pre-2026-07-30 file blocked it as "Bing's # AI crawler"): Bing search indexing is the ToS 4(d) exception at work. # Known trade-off: Bing's index also grounds Copilot answers, and # Microsoft's only AI opt-out is per-page noarchive/nocache meta tags. User-agent: Googlebot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /admin/* Disallow: /notifications Disallow: /profile/edit Disallow: /subscription Disallow: /sign-up Disallow: /sign-in Disallow: /verify/email Disallow: /jobs/* Disallow: /c/* Disallow: /h/*/contributors/ Disallow: /h/*/s$ Disallow: /h/*/contributors Disallow: /h/*/a$ Disallow: /h/*/a/* Disallow: /h/*/search User-agent: Bingbot Allow: / Disallow: /admin Disallow: /admin/ Disallow: /admin/* Disallow: /notifications Disallow: /profile/edit Disallow: /subscription Disallow: /sign-up Disallow: /sign-in Disallow: /verify/email Disallow: /jobs/* Disallow: /c/* Disallow: /h/*/contributors/ Disallow: /h/*/s$ Disallow: /h/*/contributors Disallow: /h/*/a$ Disallow: /h/*/a/* Disallow: /h/*/search # AdsBot ignores `User-agent: *` unless named explicitly (documented # Google quirk) — bind it to the same public-page rules. User-agent: AdsBot-Google Allow: / Disallow: /admin Disallow: /admin/ Disallow: /admin/* Disallow: /notifications Disallow: /profile/edit Disallow: /subscription Disallow: /sign-up Disallow: /sign-in Disallow: /verify/email Disallow: /jobs/* Disallow: /c/* Disallow: /h/*/contributors/ Disallow: /h/*/s$ Disallow: /h/*/contributors Disallow: /h/*/a$ Disallow: /h/*/a/* Disallow: /h/*/search User-agent: AdsBot-Google-Mobile Allow: / Disallow: /admin Disallow: /admin/ Disallow: /admin/* Disallow: /notifications Disallow: /profile/edit Disallow: /subscription Disallow: /sign-up Disallow: /sign-in Disallow: /verify/email Disallow: /jobs/* Disallow: /c/* Disallow: /h/*/contributors/ Disallow: /h/*/s$ Disallow: /h/*/contributors Disallow: /h/*/a$ Disallow: /h/*/a/* Disallow: /h/*/search # Everyone else: public pages only. User-agent: * Allow: / Disallow: /admin Disallow: /admin/ Disallow: /admin/* Disallow: /notifications Disallow: /profile/edit Disallow: /subscription Disallow: /sign-up Disallow: /sign-in Disallow: /verify/email Disallow: /jobs/* Disallow: /c/* Disallow: /h/*/contributors/ Disallow: /h/*/s$ Disallow: /h/*/contributors Disallow: /h/*/a$ Disallow: /h/*/a/* Disallow: /h/*/search Sitemap: https://sharebird.com/sitemap.xml.gz