1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-23 14:00:56 +01:00

Disallow /source/ in robots.txt

Summary: Ref T4245. We disallow `/diffusion/` in robots.txt already because indexers tend to get lost blaming every line of every file throughout history, but didn't update the list for the `/source/` alias. Update it.

Test Plan: Visited `/robots.txt` locally.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D17637
This commit is contained in:
epriestley 2017-04-06 15:51:17 -07:00
parent 3d816e94df
commit 9856802ba2

View file

@ -20,6 +20,7 @@ final class PhabricatorRobotsController extends PhabricatorController {
$out[] = 'User-Agent: *';
$out[] = 'Disallow: /diffusion/';
$out[] = 'Disallow: /source/';
// Add a small crawl delay (number of seconds between requests) for spiders
// which respect it. The intent here is to prevent spiders from affecting