revi-archive/phorge-phorge
1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-21 22:32:41 +01:00
Code Issues Releases Wiki Activity

Disallow webcrawlers to index Diffusion commits

Browse source 
Summary:
Phorge already sets `Disallow: /diffusion/` and `Disallow: /source/`.
Thus consequently also disallow accessing specific commits via `/r*`.
See https://secure.phabricator.com/T4610 for previous discussions.

Closes T15670

Test Plan:
Go to `/robots.txt` in the web browser.
Cross fingers that more webcrawlers abide by RFC 9309.

Reviewers: O1 Blessed Committers, speck

Reviewed By: O1 Blessed Committers, speck

Subscribers: tobiaswiese, valerio.bozzolan, Matthew, Cigaryno

Maniphest Tasks: T15670

Differential Revision: https://we.phorge.it/D25474
This commit is contained in:
Andre Klapper 2023-11-17 08:44:19 +01:00
parent 76ed0c7ff7
commit 05f4d5071f
1 changed files with 2 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
src/applications/system/controller/robots/PhabricatorRobotsPlatformController.php
View file

@ -18,6 +18,8 @@ final class PhabricatorRobotsPlatformController
$out[] = 'User-Agent: *'; $out[] = 'User-Agent: *';
$out[] = 'Disallow: /diffusion/'; $out[] = 'Disallow: /diffusion/';
$out[] = 'Disallow: /source/'; $out[] = 'Disallow: /source/';
// See T15670. Also prevent directly accessing commits in Diffusion.
$out[] = 'Disallow: /r*';
// See T15662. Prevent indexing line anchor links in Pastes. Per RFC 9309 // See T15662. Prevent indexing line anchor links in Pastes. Per RFC 9309
// section 2.2.3, percentage-encode "$" to avoid interpretation as end of // section 2.2.3, percentage-encode "$" to avoid interpretation as end of