summary | log | graphiclog1 | graphiclog2 | commit | commitdiff | tree | refs | edit | fork
raw | patch | inline | side by side (parent: 445f3f7)
serverconfig: update exclusionsrorcz
authorThe repo.or.cz admin team <[email protected]>
Tue, 11 Feb 2025 01:05:40 +0000 (11 02:05 +0100)
committerThe repo.or.cz admin team <[email protected]>
Tue, 11 Feb 2025 01:05:40 +0000 (11 02:05 +0100)
Signed-off-by: The repo.or.cz admin team <[email protected]>
apache.conf.in patch | blob | blame | history
robots.txt patch | blob | blame | history

diff --git a/apache.conf.in b/apache.conf.in
index 58cf1dc..f363491 100644 (file)
--- a/apache.conf.in
+++ b/apache.conf.in
@@ -10,6 +10,9 @@
                RewriteCond %{SERVER_NAME} !=@@httpdnsname@@
                RewriteRule ^ http://@@httpdnsname@@%{REQUEST_URI} [L,NE,R=301]
        <IfDefine @@TLSHost@@>
+               # bots to port 80 are always rejected rather than redirected to port 443
+               RewriteCond %{HTTP_USER_AGENT} (Slurp|Bot|Spider|Riddler|ltx71|Crawl) [NC]
+               RewriteRule ^ - [F,L]
                RewriteCond %{REQUEST_URI} !^/\.well-known/acme-challenge/
                RewriteCond %{SERVER_NAME} =@@httpdnsname@@
                RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [L,NE,R=301]
diff --git a/robots.txt b/robots.txt
index 535f649..90d9575 100644 (file)
--- a/robots.txt
+++ b/robots.txt
@@ -1,4 +1,8 @@
 User-agent: ahrefsbot
+User-agent: amazonbot
+User-agent: bytespider
+User-agent: claudebot
+User-agent: gptbot
 User-agent: ltx71
 User-agent: riddler
 Disallow: /
@@ -23,3 +27,4 @@ Disallow: /*/project_list
 Disallow: /*/atom
 Disallow: /*/rss
 Allow: /
+Crawl-delay: 30
Girocco README