diff options
| author |   Sam James <[email protected]> | 2025-09-22 18:30:42 +0100 |
|---|---|---|
| committer | Sam James <[email protected]> | 2025-09-22 18:30:42 +0100 |
| commit | f4610178b5993d5d3e12031d97d0dd6b2ace7ef5 (patch) | |
| tree | bb996f6b25c9bb429ce95b4356ae9efe2aaea8f8 | |
| parent | _data/iso3166-sort-of.yaml: add Colombia (CO) (diff) | |
| download | www-f461017.tar.gz www-f461017.tar.bz2 www-f461017.zip | |
robots.txt: update
https://raw.githubusercontent.com/TecharoHQ/anubis/237a6a98e268df70f2655809010c2b87ce3fca53/web/static/robots.txt
Signed-off-by: Sam James <[email protected]>
| -rw-r--r-- | robots.txt | 21 |
1 files changed, 17 insertions, 4 deletions
@@ -1,6 +1,7 @@ User-agent: AhrefsBot Disallow: / +User-agent: AddSearchBot User-agent: AI2Bot User-agent: Ai2Bot-Dolma User-agent: aiHitBot @@ -9,29 +10,38 @@ User-agent: Andibot User-agent: anthropic-ai User-agent: Applebot User-agent: Applebot-Extended +User-agent: Awario User-agent: bedrockbot +User-agent: bigsur.ai User-agent: Brightbot 1.0 User-agent: Bytespider User-agent: CCBot +User-agent: ChatGPT Agent User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Claude-Web User-agent: ClaudeBot +User-agent: CloudVertexBot User-agent: cohere-ai User-agent: cohere-training-data-crawler User-agent: Cotoyogi User-agent: Crawlspace +User-agent: Datenbank Crawler +User-agent: Devin User-agent: Diffbot User-agent: DuckAssistBot +User-agent: Echobot Bot User-agent: EchoboxBot User-agent: FacebookBot User-agent: facebookexternalhit User-agent: Factset_spyderbot User-agent: FirecrawlAgent User-agent: FriendlyCrawler +User-agent: Gemini-Deep-Research User-agent: Google-CloudVertexBot User-agent: Google-Extended +User-agent: GoogleAgent-Mariner User-agent: GoogleOther User-agent: GoogleOther-Image User-agent: GoogleOther-Video @@ -42,16 +52,20 @@ User-agent: ImagesiftBot User-agent: img2dataset User-agent: ISSCyberRiskCrawler User-agent: Kangaroo Bot +User-agent: LinerBot User-agent: meta-externalagent User-agent: Meta-ExternalAgent User-agent: meta-externalfetcher User-agent: Meta-ExternalFetcher +User-agent: MistralAI-User User-agent: MistralAI-User/1.0 User-agent: MyCentralAIScraperBot +User-agent: netEstate Imprint Crawler User-agent: NovaAct User-agent: OAI-SearchBot User-agent: omgili User-agent: omgilibot +User-agent: OpenAI User-agent: Operator User-agent: PanguBot User-agent: Panscient @@ -66,18 +80,17 @@ User-agent: QuillBot User-agent: quillbot.com User-agent: SBIntuitionsBot User-agent: Scrapy -User-agent: SemrushBot -User-agent: SemrushBot-BA -User-agent: SemrushBot-CT User-agent: SemrushBot-OCOB -User-agent: SemrushBot-SI User-agent: SemrushBot-SWA User-agent: Sidetrade indexer bot +User-agent: Thinkbot User-agent: TikTokSpider User-agent: Timpibot User-agent: VelenPublicWebCrawler +User-agent: WARDBot User-agent: Webzio-Extended User-agent: wpbot +User-agent: YaK User-agent: YandexAdditional User-agent: YandexAdditionalBot User-agent: YouBot |