From 05f4d5071fdca02123bd1ff4c0935b847c7f9963 Mon Sep 17 00:00:00 2001 From: Andre Klapper Date: Fri, 17 Nov 2023 08:44:19 +0100 Subject: [PATCH] Disallow webcrawlers to index Diffusion commits Summary: Phorge already sets `Disallow: /diffusion/` and `Disallow: /source/`. Thus consequently also disallow accessing specific commits via `/r*`. See https://secure.phabricator.com/T4610 for previous discussions. Closes T15670 Test Plan: Go to `/robots.txt` in the web browser. Cross fingers that more webcrawlers abide by RFC 9309. Reviewers: O1 Blessed Committers, speck Reviewed By: O1 Blessed Committers, speck Subscribers: tobiaswiese, valerio.bozzolan, Matthew, Cigaryno Maniphest Tasks: T15670 Differential Revision: https://we.phorge.it/D25474 --- .../controller/robots/PhabricatorRobotsPlatformController.php | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/applications/system/controller/robots/PhabricatorRobotsPlatformController.php b/src/applications/system/controller/robots/PhabricatorRobotsPlatformController.php index 82028918d7..7151f2e4aa 100644 --- a/src/applications/system/controller/robots/PhabricatorRobotsPlatformController.php +++ b/src/applications/system/controller/robots/PhabricatorRobotsPlatformController.php @@ -18,6 +18,8 @@ final class PhabricatorRobotsPlatformController $out[] = 'User-Agent: *'; $out[] = 'Disallow: /diffusion/'; $out[] = 'Disallow: /source/'; + // See T15670. Also prevent directly accessing commits in Diffusion. + $out[] = 'Disallow: /r*'; // See T15662. Prevent indexing line anchor links in Pastes. Per RFC 9309 // section 2.2.3, percentage-encode "$" to avoid interpretation as end of