Disallow webcrawlers to index Diffusion commits

Summary:
Phorge already sets `Disallow: /diffusion/` and `Disallow: /source/`.
Thus consequently also disallow accessing specific commits via `/r*`.
See https://secure.phabricator.com/T4610 for previous discussions.

Closes T15670

Test Plan:
Go to `/robots.txt` in the web browser.
Cross fingers that more webcrawlers abide by RFC 9309.

Reviewers: O1 Blessed Committers, speck

Reviewed By: O1 Blessed Committers, speck

Subscribers: tobiaswiese, valerio.bozzolan, Matthew, Cigaryno

Maniphest Tasks: T15670

Differential Revision: https://we.phorge.it/D25474

src/applications/system/controller/robots/PhabricatorRobotsPlatformController.php

@@ -18,6 +18,8 @@ final class PhabricatorRobotsPlatformController
     $out[] = 'User-Agent: *';
     $out[] = 'Disallow: /diffusion/';
     $out[] = 'Disallow: /source/';
+    // See T15670. Also prevent directly accessing commits in Diffusion.
+    $out[] = 'Disallow: /r*';
 
     // See T15662. Prevent indexing line anchor links in Pastes. Per RFC 9309
     // section 2.2.3, percentage-encode "$" to avoid interpretation as end of