mirror of
https://we.phorge.it/source/phorge.git
synced 2024-11-22 23:02:42 +01:00
Disallow /source/ in robots.txt
Summary: Ref T4245. We disallow `/diffusion/` in robots.txt already because indexers tend to get lost blaming every line of every file throughout history, but didn't update the list for the `/source/` alias. Update it. Test Plan: Visited `/robots.txt` locally. Reviewers: chad Reviewed By: chad Maniphest Tasks: T4245 Differential Revision: https://secure.phabricator.com/D17637
This commit is contained in:
parent
3d816e94df
commit
9856802ba2
1 changed files with 1 additions and 0 deletions
|
@ -20,6 +20,7 @@ final class PhabricatorRobotsController extends PhabricatorController {
|
|||
|
||||
$out[] = 'User-Agent: *';
|
||||
$out[] = 'Disallow: /diffusion/';
|
||||
$out[] = 'Disallow: /source/';
|
||||
|
||||
// Add a small crawl delay (number of seconds between requests) for spiders
|
||||
// which respect it. The intent here is to prevent spiders from affecting
|
||||
|
|
Loading…
Reference in a new issue