Commit graph

20 commits

Author SHA1 Message Date
revi 8084acebfa
Phorge(robots.txt): add applebot-extended to disallow
Summary:
Ref: https://support.apple.com/en-us/119829#datausage

Signed-off-by: Yongmin Hong <revi@omglol.email>

Test Plan: Verify the user-agent matches.

Reviewers: O1 revi & automations, revi

Reviewed By: O1 revi & automations, revi

Differential Revision: https://issuetracker.revi.xyz/D508
2024-07-04 00:12:13 +09:00
revi 387440faf8
Phorge(robots.txt): add trailing slashes to phorge lines
Summary:
Entries with(out) slashes.

Signed-off-by: Yongmin Hong <revi@omglol.email>

Test Plan: Deploy it.

Reviewers: O1 revi & automations, revi

Reviewed By: O1 revi & automations, revi

Differential Revision: https://issuetracker.revi.xyz/D385
2024-06-21 16:09:12 +09:00
revi a1117fa7c1
phorge(robots.txt): add more domains
Summary:
Almost complete copy of rWEBXYZ. cookie unset didn't work as hoped,
comment it out.

Signed-off-by: Yongmin Hong <revi@omglol.email>

Test Plan: deploy it.

Reviewers: O1 revi & automations, revi

Reviewed By: O1 revi & automations, revi

Differential Revision: https://issuetracker.revi.xyz/D370
2024-06-19 15:17:40 +09:00
revi 68f2133e9c
PhabExt(robots.txt): add more rules, ...
Summary:
setClientIDCookie none to return Response, more rules, todo:

Signed-off-by: Yongmin Hong <revi@omglol.email>

Test Plan: deploy?

Reviewers: O1 revi & automations, revi

Reviewed By: O1 revi & automations, revi

Differential Revision: https://issuetracker.revi.xyz/D338
2024-06-17 12:46:53 +09:00
revi bf5c8a9281
Robots.txt: f off PetalBot
Summary:
Is on .xyz/robots.txt blacklist. add it here as well.

Signed-off-by: Yongmin Hong <revi@omglol.email>

Test Plan: Deploy

Reviewers: O1 revi & automations, revi

Reviewed By: O1 revi & automations, revi

Differential Revision: https://issuetracker.revi.xyz/D125
2024-05-23 04:20:04 +09:00
revi 467eb9b2d1
robotstxt: just use *
Summary:
Fixes T138. Seems like Google doesn't care.

Signed-off-by: Yongmin Hong <revi@omglol.email>

Test Plan: Try http://go/google/search w/ updated robots.txt entries

Reviewers: O1 revi & automations, revi

Reviewed By: O1 revi & automations, revi

Maniphest Tasks: T138

Differential Revision: https://issuetracker.revi.xyz/D75
2024-05-11 17:29:11 +09:00
revi a257d3ccb4
robots.txt: shorten some links
Summary:
Less size /shrug

Signed-off-by: Yongmin Hong <revi@omglol.email>

Test Plan: try visiting URLs to test redirection

Reviewers: O1 revi & automations, revi

Reviewed By: O1 revi & automations, revi

Differential Revision: https://issuetracker.revi.xyz/D66
2024-05-10 18:19:11 +09:00
revi c628542223
Revert "RobotsTxt: specify more verbose classes"
Summary:
This reverts commit 88946a7fe2.

PHP Fatal error:  Cannot override final method PhabricatorRobotsController::processRequest() in /var/www/phorge/phorge/src/extensions/PhabricatorCustomRobotsTxtController.php on line 13

Fatal error: Cannot override final method PhabricatorRobotsController::processRequest() in /var/www/phorge/phorge/src/extensions/PhabricatorCustomRobotsTxtController.php on line 13

Signed-off-by: Yongmin Hong <revi@omglol.email>

Reviewers: #acl_sudoers, #blessed_reviewers, revi

Reviewed By: #acl_sudoers, #blessed_reviewers, revi

Differential Revision: https://issuetracker.revi.xyz/D10
2024-05-02 19:56:13 +09:00
revi 88946a7fe2
RobotsTxt: specify more verbose classes
Summary:
Using PhabricatorRobotsController overwrites robots.txt for blogs, URL shortener, and CDN domains. Using [PhabricatorRobotsPlatformController](https://we.phorge.it/book/dev/class/PhabricatorRobotsPlatformController/) should fix it.

Signed-off-by: Yongmin Hong <revi@omglol.email>

Test Plan: Reload personal-phorge.revicdn.net/robots.txt and it is not verbatim copy of issuetracker.revi.xyz/robots.txt

Reviewers: #acl_sudoers, #blessed_reviewers, revi

Reviewed By: #acl_sudoers, #blessed_reviewers, revi

Differential Revision: https://issuetracker.revi.xyz/D9
2024-05-02 19:48:19 +09:00
revi 0e1656da0c
meta: various changes
Summary:
Submitting three changes at once for lazyness:

meta: config cleanups

  Add editorconfig, gitignore, and conform to them.

RobotsTxt: add differential rule and conform to the editorconfig

time.php: add metadata, editorconfig conformance

Signed-off-by: Yongmin Hong <revi@omglol.email>

Test Plan: `code .` and edit files. for php files, deploy.

Reviewers: #acl_sudoers, #blessed_reviewers, revi

Reviewed By: #acl_sudoers, #blessed_reviewers, revi

Differential Revision: https://issuetracker.revi.xyz/D8
2024-05-02 19:30:12 +09:00
revi 09470b6a92
RobotsTxt: update lastModified
Bug: N/A
Signed-off-by: Yongmin Hong <revi@omglol.email>
2024-04-30 03:27:53 +09:00
revi 6b530af898
RobotsTxt: remove duplicate
Turns out /file is already listed...

Bug: N/A
Signed-off-by: Yongmin Hong <revi@omglol.email>
2024-04-30 03:24:41 +09:00
revi bc577f0632
phabExt: add lastModified time and few more rules
Should figure out how to disable set-cookie to enable Cloudflare
cache...

Bug: N/A
Signed-off-by: Yongmin Hong <revi@omglol.email>
2024-04-29 18:02:07 +09:00
revi 59fb26f299
PhabExt: add more domains
More LLM domains
Source: https://darkvisitors.com/docs/set-up-a-robots-txt

Bug: N/A
Signed-off-by: Yongmin Hong <revi@omglol.email>
2024-04-21 04:36:07 +09:00
revi 683a11ca27
RobotsTxt: fix another typo
It's probably high time that I go to bed.

Bug: N/A
Signed-off-by: Yongmin Hong <revi@omglol.email>
2024-04-17 01:50:42 +09:00
revi 1df92aac36
RobotsTxt: fix typo
And remove redundant rule.

Bug: N/A
Signed-off-by: Yongmin Hong <revi@omglol.email>
2024-04-17 01:42:05 +09:00
revi cb20e4d795
RobotsTxt: add more rules
Interest of nobody.

Bug: N/A
Signed-off-by: Yongmin Hong <revi@omglol.email>
2024-04-17 01:35:10 +09:00
revi c4eb4aba4e
RobotsTxt: fix comment error
Whoopsie.

Bug: Ref T43
Signed-off-by: Yongmin Hong <revi@omglol.email>
2024-04-16 21:17:47 +09:00
revi 70cbb7ccd8
phab-ext: test
Signed-off-by: Yongmin Hong <revi@omglol.email>
2024-04-16 21:04:16 +09:00
revi 11f02e587f
initial commit
Yet another sandbox.

Bug: N/A
Signed-off-by: Yongmin Hong <revi@omglol.email>
2024-04-15 19:14:13 +09:00