1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-11 08:06:13 +01:00
Commit graph

13360 commits

Author SHA1 Message Date
epriestley
ad880491e8 Remove product literal strings in "pht()", part 5
Summary: Ref T13658

Test Plan:
This test plan is non-exhaustive.

  - Looked at some of the config.
  - Looked at guides.

Maniphest Tasks: T13658

Differential Revision: https://secure.phabricator.com/D21769
2022-04-25 12:22:27 -07:00
epriestley
d69da878d2 Remove product literal strings in "pht()", part 4
Summary: Ref T13658.

Test Plan:
This is non-exhaustive.

  - Looked at some (most?) of the config values in Config.

Maniphest Tasks: T13658

Differential Revision: https://secure.phabricator.com/D21768
2022-04-25 12:22:27 -07:00
epriestley
b1b3fe0ff0 Remove product literal strings in "pht()", part 3
Summary: Ref T13658.

Test Plan:
This is non-exhaustive.

  - Ran all setup checks.

Maniphest Tasks: T13658

Differential Revision: https://secure.phabricator.com/D21767
2022-04-25 12:22:26 -07:00
epriestley
b8b392481f Remove product literal strings in "pht()", part 2
Summary: Ref T13658.

Test Plan:
This test plan is non-exhaustive.

  - Used Auth to start an "another copy of Phabricator" OAuth server workflow.

Maniphest Tasks: T13658

Differential Revision: https://secure.phabricator.com/D21766
2022-04-25 12:22:26 -07:00
epriestley
3e82ab5adb Remove product literal strings in "pht()", part 1
Summary: Ref T13658. I used the linter in D21763 to identify these and `split` them into arbitrary groups of 10 files.

Test Plan:
This test plan is non-exhaustive, because some of these strings are difficult to reach.

  - Looked at "Create Service" in Almanac.
  - Used "bin/auth" to go through a one-time auth workflow (not all related strings can be hit on a single workflow).
  - Started the "Generate Keypair" worfklow in "SSH Public Keys".

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13658

Differential Revision: https://secure.phabricator.com/D21765
2022-04-25 12:22:25 -07:00
epriestley
de980cc54e Fix an issue where Herald may fail to extract content from an empty commit
Summary:
Ref T13667. The Herald "content added" rule (and other similar rules) do not correctly extract content from empty commits.

When we load an empty raw diff, return an empty changed content map.

Ref T13588. Also fix some PHP8.1 null/string stuff

Test Plan:
  - Ran "bin/repository reparse --publish <commit>", with an empty commit hash and a nonempty commit hash.
  - Reviewed Herald transcripts for general sanity.

Maniphest Tasks: T13667, T13588

Differential Revision: https://secure.phabricator.com/D21761
2022-04-20 13:04:00 -07:00
epriestley
52df4ff515 Fix an issue where "git" may be unable to read a temporary file in Diffusion
Summary:
Ref T13673. After the changes in that task, we may execute "git config -l ..." as a user other than the user we used to write this temporary file.

Use "--file -" to pass the data instead, avoiding use of temporary files. This makes us agnostic to filesystem permissions.

Test Plan: Viewed a Git repository with submodules in Diffusion with "ssh.user" configured as a user relatively isolated from the webserver user.

Maniphest Tasks: T13673

Differential Revision: https://secure.phabricator.com/D21759
2022-04-20 13:03:59 -07:00
epriestley
21b1a304b6 Improve some PHP 8.1 behavior in "bin/auth recover"
Summary: Ref T13588. Fix a couple of argument parsing issues here.

Test Plan: Ran "bin/auth recover" under PHP 8.1.

Maniphest Tasks: T13588

Differential Revision: https://secure.phabricator.com/D21757
2022-04-19 14:55:19 -07:00
epriestley
3125d7a5f4 In Git, always "sudo" to the daemon user if a daemon user is configured
Summary:
See T13673. Recent versions of Git (and older versions with backported security patches) now refuse to run Git commands if the top-level repository directory is not owned by the user running the command.

Currently, we "sudo" to that user only when performing writes, so upgrading Git can aggressively break a Phabricator system by knocking out essentially all Diffusion/Conduit read pathways.

As an immediate mitigation, just "sudo" in all cases where a daemon user is available. This fixes the problem, and seems like the least-bad approach. The downside is that the web user may theoretically have fewer privileges than the daemon user and this could reduce the number of layers an attacker armed with some other Git vulnerability might have to get through to do something dangerous (e.g., perform a write on a pathway where only reads are expected), but any separation between the web and daemon accounts is essentially theoretical and has never been enforced.

Test Plan: Applied patch to impacted Phacility shard, saw Diffusion work properly again.

Differential Revision: https://secure.phabricator.com/D21756
2022-04-13 11:23:55 -07:00
epriestley
4dae3e7e1f Improve some UI/language for Phame posts when viewer doesn't have CAN_INTERACT
Summary: Ref T13661. Fix up some UI and language so it's more clear that this is about disabling blog comments.

Test Plan: Viewed UIs, saw some more usable strings.

Maniphest Tasks: T13661

Differential Revision: https://secure.phabricator.com/D21755
2022-04-01 12:52:58 -07:00
epriestley
aae23f0204 Give Phame blog posts configurable interact policies, with a default policy of "Same as Blog"
Summary: Ref T13661. This allows posts to have comments disabled (or restricted) on a per-post basis, and makes them inherit the containing blog policy by default.

Test Plan: Locked a post by editing its policy explicitly; locked a post by editing the containing blog policy.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13661

Differential Revision: https://secure.phabricator.com/D21754
2022-04-01 12:52:58 -07:00
epriestley
f555fbcb50 Remove unused "MARKUP_FIELD_SUMMARY" for Phame posts
Summary: Ref T13661. This ancient "RemarkupInterface" constant has no references.

Test Plan: Grepped for references, found none. Viewed a blog, viewed a post.

Maniphest Tasks: T13661

Differential Revision: https://secure.phabricator.com/D21753
2022-04-01 12:52:57 -07:00
epriestley
a9822a37aa Fix a PHP 8.1 unit test failure in Projects
Summary: Ref T13588. This field may be "null" (and is probably never the empty string, but that's a more ambitious fix).

Test Plan: Ran unit tests, got a pass.

Maniphest Tasks: T13588

Differential Revision: https://secure.phabricator.com/D21752
2022-04-01 12:52:57 -07:00
epriestley
c25595417f Give Phame blogs mutable interact policies
Summary:
Ref T13661.

I'm fairly sure these policies don't actually do anything (you can't "interact" with a blog) but the primarily support a Phame Post object policy of "Same as Parent Blog", which is the "natural" interact policy for a post.

Most of this is infrastructure support for mutable interact policies: today, only Maniphest has interact mutability and only via indirect effects (locking tasks), not through a directly mutable "Can Interact" policy.

Test Plan:
Ran storage upgrade, edited interact policy of a blog, saw appropriate persistence and transactions.

Created and edited a task to make sure there's no weird fallout from increasing what can be done with interact policies.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13661

Differential Revision: https://secure.phabricator.com/D21751
2022-04-01 12:52:56 -07:00
epriestley
5ce7f803f9 Fix double-bordered breadcrumbs in Phame blogs
Summary: Ref T13661. The Phame blog main page has a double breadcrumb border in non-live mode.

Test Plan: Viewed a blog page in internal and live modes, saw single border in both cases (previously: double border for internal mode).

Maniphest Tasks: T13661

Differential Revision: https://secure.phabricator.com/D21750
2022-04-01 12:52:56 -07:00
epriestley
6c2c80a373 Remove ancient Remarkup constants from Phame and Maniphest
Summary: Ref T13661. These constants are no longer used.

Test Plan:
For a blog with a nonempty description, viewed the "Manage" page and the home page; saw properly rendered description.

Grepped for removed constants, no hits.

Maniphest Tasks: T13661

Differential Revision: https://secure.phabricator.com/D21749
2022-04-01 12:52:55 -07:00
epriestley
28b12fb3cd Make Phame blog policies non-nullable
Summary: Ref T6203. Ref T13661. These policies are incorrectly nullable, although it's likely that no pathway exists in the application to write NULL into them. Fix the schema.

Test Plan: Ran `bin/storage upgrade`.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13661, T6203

Differential Revision: https://secure.phabricator.com/D21748
2022-04-01 12:52:55 -07:00
Matthew Bowker
7d4357683a Hide the blurb of a user when that user is disabled
Summary: T15074

Test Plan: Disabled a user, then made sure their blurb disappeared.

Reviewers: O1 Blessed Committers, avivey, speck

Reviewed By: O1 Blessed Committers, avivey, speck

Subscribers: 20after4, avivey, speck, tobiaswiese, valerio.bozzolan

Maniphest Tasks: T15074

Differential Revision: https://we.phorge.it/D25035
2022-03-30 09:17:29 -06:00
epriestley
6136964093 Fix a PHP 8.1 strlen() issue with "mysql.pass" configuration
Summary:
Ref T13588. This configuration value may not be set.

Also fix an issue in `bin/storage` and whatever else I hit between now and this diff actually uploading.

Also fix a MySQLi report mode difference, beginning in PHP 8.1.

Also update a bunch of "static" property usage in Lisk.

Test Plan: Ran `bin/files ...` locally under PHP 8.1.

Maniphest Tasks: T13588

Differential Revision: https://secure.phabricator.com/D21744
2021-12-16 15:24:21 -08:00
epriestley
dc705cea7f Document Mailgun as discouraged, and update Postmark remote address blocks
Summary:
Ref T13669. See that task for discussion. Based on a disheartening interaction in June 2020, discourage use of Mailgun.

Also update the default Postmark inbound address list, since they've added a new address.

Test Plan: Read documentation, grepped for "mailgun".

Maniphest Tasks: T13669

Differential Revision: https://secure.phabricator.com/D21738
2021-12-02 14:56:36 -08:00
epriestley
c7b04170fe Provide missing "AlmanacInterfaceTransactionQuery"
Summary:
Ref PHI2149. This provides the trivial Query class required for the daemons to publish Almanac Interface transactions.

(Publishing these doesn't do anything interesting, but currently leaves an error in the daemon logs.)

Test Plan:
  - Stopped the daemons.
  - Edited the port of an Interface in Almanac.
  - Ran `bin/worker execute --active --class PhabricatorApplicationTransactionPublishWorker` to publish the transaction.
    - Before: fatal on missing class, "Unable to load query for transaction object...".
    - After: transaction publishes cleanly.

Differential Revision: https://secure.phabricator.com/D21726
2021-11-16 11:48:25 -08:00
epriestley
b9ba0418ed Correct a parameter order swap in "diffusion.historyquery" for Mercurial
Summary: Ref T13666. See T13666#256253. The order of these parameters is incorrect (introduced in D18817, which was trying to turn the revset "A or B" into "B or A"), but when "commit" is the single head of "branch" (which is common), the revset is functionally equivalent.

Test Plan:
An easy reproduction case is to make a "diffusion.historyquery" call manually against a Mercurial repository, with a valid "branch" name and some non-head "commit".

  - Before patch, queried "head^" (by commit hash, not by symbol) of "default" and got "head" too (wrong).
  - Applied patch to `secure001.phacility.net` (my local `hg` web environment is iffy right now).
  - Re-ran the same query, saw "head^" as the first result rather than "head" (all hashes rather than symbols, as above), which is desired.

Reviewers: cspeckmim

Reviewed By: cspeckmim

Maniphest Tasks: T13666

Differential Revision: https://secure.phabricator.com/D21722
2021-09-04 18:16:32 -07:00
Christopher Speck
3b2868e155 Fix searching legalpad documents by contributors
Summary:
Found in a change submitted to Phorge (https://we.phorge.it/D25018#inline-231), this fixes a typo in populating search the `contributorPHIDs` constraint in the Legalpad search engine.

Currently when trying to search legalpad documents by contributor an error is encountered:
```lang=console
Array for %Ls conversion is empty. Query: contributor.dst IN (%Ls)
```

Test Plan: I searched for legalpad documents based on a contributor and got back correct results.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D21719
2021-09-04 20:37:29 -04:00
Christopher Speck
09c3c7d879 Add support to marking commits as UNREACHABLE for Mercurial
Summary:
When previously known commits have been destroyed in a Mercurial repository, Phabricator does not end up marking the commits as unreachable. This results in daemon tasks which continuously fail and retry.

This updates `PhabricatorRepositoryDiscoveryEngine` and `PhabricatorManagementRepositoryMarkReachableWorkflow` to include support of marking commits as unreachable for Mercurial repositories.

The `PhabricatorMercurialGraphStream` also needed updated to support a stream with no starting commit.

Refs T13634

Test Plan:
1. I set up a hosted Mercurial repository.
2. I removed the head commit from the on-disk repository state.
3. I attempted to load the repository page and saw an exception due to a missing commit.
4. I went to `/manage` for the repository and scheduled an update of the repository.
5. After an updated performed, I went to the repository main page and saw there was no exception and the history view properly did not have the commit I had removed.
6. I checked the phd logs and verified there were no exceptions related to the repository.
7. I ran the `./bin/repository mark-reachable` command on the Mercurial repository and it reported that it marked the commit as unreachable.
8. I pushed the same commit back upstream and verified that the commit was found and displayed in the history view of the repository page and `mark-unreachable` did not identify it as being unreachable.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T13634

Differential Revision: https://secure.phabricator.com/D21715
2021-09-04 19:05:08 -04:00
Christopher Speck
458ad4a861 Remove history query from DiffusionRepositoryController as it is unused
Summary:
The history query for the repository page isn't actually used to display any content. It looks like it was previously used to display the last user which modified a file however this looks to be removed in D21404. This removes the history query from happening as well as updates `DiffusionBrowseTableView` to remove the parameters for passing this information in, resulting in also updating `DiffusionBrowseController` to no longer need to put this information together.

Refs T13666

Test Plan:
1. I removed commits from a repository on the local state.
2. I navigated to the repository's landing page and saw that the landing page attempted to render content and only failed to load the browse files section.
3. I navigated to the history tab and verified that it showed an exception about failing to query commit information.
4. I restored the repository working state to function properly.
5. I navigated to a repository's landing page and verified it loaded properly, including showing the last modified date for each file.
6. I navigated to the Code, Branches, Tags, and History tabs to verify each tab page loaded properly.
7. I verified on the Code tab that the last modified date for each file displayed properly.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T13666

Differential Revision: https://secure.phabricator.com/D21717
2021-09-04 18:50:46 -04:00
epriestley
b757e5c302 Use "MethodCallFuture" to move Diffusion Conduit exceptions to resolution time
Summary:
Depends on D21720. Ref T13666. See D21720 for additional discussion.

Use "MethodCallFuture", introduced in D21720, so that exceptions raised in "execute()" are thrown when the future is resolved, not when the future is created.

This makes exception behavior for clustered and non-clustered setups consistent, and chooses the intended (clustered) behavior in both cases, which currently deals with errors better.

Test Plan:
  - Applied both parts of the patch in T13666 (break history queries, force immediate futures) to reproduce the issue in T13666.
    - Loaded a Diffusion landing page, reproduced the error described in that task.
  - Applied this patch, verified landing page works again.
  - Removed the "break history queries" change, verified landing page works in forced-immediate mode.
  - Removed the "force immediate" change, verified landing page works in "actual future" mode.

Reviewers: cspeckmim

Reviewed By: cspeckmim

Maniphest Tasks: T13666

Differential Revision: https://secure.phabricator.com/D21721
2021-09-04 14:36:41 -07:00
James Brown
a4948ec800 Add conduit endpoints for querying legalpad
Summary:
This diff adds conduit methods for searching for legalpad documents and signatures. This is very helpful for auditing who's actually signed a document. It also fixes the "contributorPHIDs" constraint in the existing search engine.

In order to expose legalpad signatures through Conduit, this adds a `phid` column to the `legalpad_documentsignature` table. It includes a migration (in the style of many previous phid-adding migrations) to actually populate the column.

Test Plan: We run this on my company's internal fork and it seems to work okay. I don't think any other conduit methods anywhere have tests (???), but if you can point me at one I'm glad to write a unit test!

Reviewers: O1 Blessed Committers, speck

Reviewed By: O1 Blessed Committers, speck

Subscribers: 20after4, speck, tobiaswiese

Differential Revision: https://we.phorge.it/D25018
2021-09-04 13:53:13 -04:00
Dmitri Iouchtchenko
1965b78b34 Fix subtype extension support check
Summary: Fixes T13663. `supportsSubtypes` tries to create an editable object, but this isn't always valid for `PhabricatorCalendarImport`. Use `instanceof` instead.

Test Plan:
  - Edited calendar import, tasks (2 different subtypes), and projects (2 different subtypes).
  - Changed task subtypes using {nav Change Subtype} action and batch editor.
  - Changed task and project subtypes using Conduit.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Maniphest Tasks: T13663

Differential Revision: https://secure.phabricator.com/D21714
2021-08-20 18:49:35 -04:00
epriestley
8daaf5ef21 Pass a real context object to Phriction previews, fixing mentions
Summary:
Fixes T13662. Phriction currently passes a map as a "context object", but this code is ancient and predates the modern meaning of a "context object". In modern code, context objects should be real objects.

Provide a real object as a context object. We do this by either loading the actual document or constructing a synthetic version of it.

Test Plan:
  - Edited an existing document, observing the preview:
    - Used a mention rule, saw a preview.
    - Used `[[ a ]]` and `[[ ./a ]]` absolute and relative reference rules, saw accurate previews.
  - Edited a new document, observing the preview:
    - Used a mention rule, saw a preview.
    - Used absolute/relative references, saw accurate previews.
  - Grepped for other references to the removed properties (`phriction.isPreview` and `phriction.slug`), found none remaining.

Reviewers: 0

Reviewed By: 0

Maniphest Tasks: T13662

Differential Revision: https://secure.phabricator.com/D21709
2021-08-02 08:36:15 -07:00
epriestley
c7550dbee9 Fix Phriction document previews for the root document ("/") with Apache option "MergeSlashes On"
Summary:
Ref T13662. I ran into this while trying to reproduce the mention issue discussed there.

Currently, the root document (with slug "/") attempts to preview using the URI `/phriction/preview//` (with two `//` at the end). This is collapsed into "/phriction/preview/" by Apache if "MergeSlashes On" is configured, which is the default behavior. The route then 404s.

Instead, just use "/phriction/preview/?slug=/" so this endpoint functions properly regardless of the "MergeSlashes" configuration.

Test Plan:
  - Configured Apache with "MergeSlashes On" (which is the default behavior).
  - Tried to preview a content edit of the root document in Phriction, which didn't work and generated 404s for "/phriction/preview//" in the console log.
  - Applied patch.
  - Previwed content in Phriction (which now worked properly).
  - Accessed `/a//b///c////` and similar with "MergeSlashes On" and "MergeSlashes Off", confirmed that this option controls whether PHP receives a URI with or without merged slashes in "__path__" after rewriting.

Reviewers: 0

Reviewed By: 0

Maniphest Tasks: T13662

Differential Revision: https://secure.phabricator.com/D21708
2021-08-02 08:36:15 -07:00
epriestley
cc8cbed029 Make "DifferentialDiff->properties" a proper "attachable" property
Summary:
See PHI498. This should be initialized to "self::ATTACHABLE" like other attachable properties, but is currently initialized to "array()".

Initialize it the normal way and try to catch all code paths which may have accessed it without actually loading and attaching it.

Also, remove UI for the very old "excuse" property, which "arc" has not written for well more than a year.

Test Plan: Grepped for affected symbols, loaded various revision pages. Somewhat tricky to be 100% sure that every pathway is caught, but it should be obvious if I missed anything once someone hits the code path.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D21710
2021-07-27 13:39:12 -07:00
epriestley
387d3b4983 Resolve deleted packages properly as having no mailable members
Summary:
Fixes T13648. If a package recipient has been destroyed, this query currently fails to return an expanded recipient value.

Instead, make sure all input PHIDs get an output. For destroyed packages, the output will just be an empty list.

Test Plan:
  - Added a package to a revision as a reviewer.
  - Destroyed the package.
  - Commented on the revision.
  - Processed the publishing worker with `bin/worker execute`.
  - Before: fatal after expanding the destroyed package.
  - After: clean publish.

Maniphest Tasks: T13648

Differential Revision: https://secure.phabricator.com/D21707
2021-07-21 14:50:07 -07:00
epriestley
4cffaa600b Rename "HarbormasterRestartException" to "HarbormasterMessageException"
Summary: Ref T13072. This exception is now raised by all of the message-sending code. Pretty straight find/replace.

Test Plan: Grepped for old class name, no hits.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21699
2021-07-21 14:17:00 -07:00
epriestley
4c4123f98b Allow "harbormaster.sendmessage" to send control command (pause, restart, abort, resume) to Builds/Buildables
Summary: Ref T13072. Expand the role of "harbormaster.sendmessage" and allow it to send control messages to Builds and Buildables.

Test Plan: Read documentation, sent commands to Builds and Buildables, hit a bunch of error cases, will deploy to catch full-lifecycle Build Target use cases.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21698
2021-07-21 14:17:00 -07:00
epriestley
3df1e17527 Add a side nav to Conduit API method console pages
Summary: Ref T13072. Make large Conduit doc pages a bit more navigable. This prepares for updating "harbormaster.sendmessage" to support sending messages to builds.

Test Plan: Viewed various Conduit API documentation pages, clicked links.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21696
2021-07-21 14:16:59 -07:00
epriestley
2ff1d4b3b0 Add stub "harbormaster.build.edit" and "harbormaster.buildable.edit" API methods
Summary: Ref T13072. These don't do anything useful yet, but get the skeletons in.

Test Plan: Loaded documentation pages without fataling.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21695
2021-07-21 14:16:59 -07:00
epriestley
346ffc51e1 Modularize "HarbormasterBuildableTransaction"
Summary: Ref T13072. Trivially convert this into a modular transaction type.

Test Plan: Issued commands to a buildable.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21694
2021-07-21 14:16:58 -07:00
epriestley
706b91adf6 Remove "HarbormasterBuildableTransaction::TYPE_CREATE"
Summary: Ref T13072. This transaction type has no writers and is mooted by EditEngine.

Test Plan: Grepped for transaction constant.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21693
2021-07-21 14:16:57 -07:00
epriestley
1a1395579f Remove "HarbormasterBuildCommand"
Summary: Ref T13072. Update the last few constant references to this class and remove it.

Test Plan: Grepped for "HarbormasterBuildCommand", got no hits.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21692
2021-07-21 14:16:57 -07:00
epriestley
1212dc5fbe Modularize almost all Harbormaster build message workflows and UI/UX
Summary: Ref T13072. Push nearly all Harbormaster build message logic into the new per-message transaction classes.

Test Plan:
  - Issued every message to Buildables.
  - Issued every message to Builds.
  - Looked at a big pile of error messages, couldn't find any typos.
  - Grepped for affected symbols, etc.
  - Ran `bin/harbormaster restart ...`.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21691
2021-07-21 14:16:56 -07:00
epriestley
8bbee92139 Modularize individual Harbormaster build messages
Summary:
Ref T13072. Further modularize build messages by applying each one in a separate transaction type.

This makes it easier to add new types of messages (although I have no particular plans to do this, offhand) and reduces the amount of switch-boilerplate.

This will probably also simplify validating "harbormaster.sendmessage".

Test Plan:
  - Applied all commands.
  - Ran migration, saw transactions render properly

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21690
2021-07-21 14:16:56 -07:00
epriestley
6dfea0adad Modularize HarbormasterBuildTransaction
Summary: Ref T13072. Update "HarbormasterBuild" to use modern modular transactions.

Test Plan:
  - Aborted, restarted, paused, and resumed a build.
  - Used `bin/harbormaster restart`.
  - Grepped for use of old "::TYPE_COMMAND" constant, didn't find any hits.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21689
2021-07-21 14:16:55 -07:00
epriestley
b84c02959a Remove "HarbormasterBuildTransaction::TYPE_CREATE"
Summary: Ref T13072. No callers currently generate these transactions, and they probably never have. Remove them.

Test Plan: Grepped for "HarbormasterBuildTransaction::TYPE_CREATE" and "self::TYPE_CREATE" in the class, found no hits.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21688
2021-07-21 14:16:55 -07:00
epriestley
bf83fffca1 Correct the flow of edit authority when sending messages to HarbormasterBuild objects
Summary:
Ref T13072. Currently, Harbormaster builds react to messages by applying a transaction inline (which can race) that has no real effect.

Later, the BuildEngine picks up the mesasge and applies a real effect, but this isn't transactional.

This is backwards, and makes it more difficult to transition to ModularTransaction and EditEngine. The desired workflow is:

  - sending a message //just// writes to the message table (and queues a worker to process the message);
  - the BuildEngine processes the message and applies effects in a transactional way.

Force this into at least roughly the right sequence of behaviors. This paves the way for porting to ModularTransaction, which should allow further cleanup.

Test Plan: Paused, resumed, aborted, and restarted a build. Ran BuildWorkers to process the commands, saw builds update appropriately.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21687
2021-07-21 14:16:54 -07:00
epriestley
012af00731 Improve formality of "HarbormasterBuild" states
Summary:
Ref T13072. Currently, Builds have basic states (like "passed" and "failed") and pending states where a command has been issued but not yet executed (pausing, resuming, restarting, and aborting).

These are handled in a bit of an ad-hoc way, and not everything treats them the same way. In particular, the build page can concurrently report a build as "Aborting" and "Pausing", with different icons and colors.

Make everything use the same logic so that a Build can only be in exactly zero or one pending state, and use the same icons and colors.

Also tighten up which transitions are allowed: for example, it doesn't make sense to pause an aborting build.

The tighter rules don't all produce great UX right now (like "You can't pause this build.", when it would be better as "You can't pause a build which is already aborting." or similar), but just leave that alone for now.

Test Plan: Viewed builds, applied various state changes, ran BuildWorker to effect the state changes, grepped for affected methods, tried to issue various out-of-sequence build commands.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21685
2021-07-21 14:16:54 -07:00
epriestley
b48d4fabaf Merge the "HarbormasterBuildCommand" table into "HarbormasterBuildMessage"
Summary:
Ref T13072. These two similar tables don't make sense to keep separate. Instead, make Build a valid receiver for BuildMessage objects.

These tables are practically the same, so this is straightforward: just copy the rows in and then drop the old table.

(This table was trivial and ephemeral anyway, so I'm not bothering to do the usual "keep it around for a couple years just in case".)

Test Plan:
  - Populated BuildCommand table, ran migration, saw Builds end up in the proper transitional state (e.g., pausing, aborting, restarting) with appropriate queued messages.
  - Queued new messages by clicking UI buttons.
  - Ran BuildWorkers, saw them process messages and mark them as consumed.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21684
2021-07-21 14:16:53 -07:00
epriestley
0f93d1ffe4 Rename "HarbormasterBuild" methods to prepare for use of the "BuildMessages" table
Summary:
Ref T13072. Rename various "command" properties to "message" properties, to prepare for merging "HarbormasterCommand" into "HarbormasterMessage".

This change only renames variables and methods and should not affect program behavior.

Test Plan: Grepped for affected symbols, found no unmodified hits.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13072

Differential Revision: https://secure.phabricator.com/D21683
2021-07-21 14:16:52 -07:00
epriestley
af6cc0c934 Use "resolve()", not "execute()", for PhutilExecPassthru callsites in Phabricator
Summary: Ref T13660. Clean up callsites to "PhutilExecPassthru->execute()" to prepare to deprecate it.

Test Plan:
  - Grepped for "PhutilExecPassthru" and looked for callsites.
  - Ran `GIT_SSH=.../ssh-connect git ls-remote origin` to execute the "ssh-connect" code.
  - The two passthru future methods have no callers and could possibly be removed, but I'm just letting sleeping dogs lie for now.

Reviewers: cspeckmim

Reviewed By: cspeckmim

Maniphest Tasks: T13660

Differential Revision: https://secure.phabricator.com/D21703
2021-07-21 10:21:06 -07:00
Christopher Speck
e5de7f0843 Update templates used with mercurial to remove '--debug'
Summary:
Refer to discussion on D21677#275541

Refs D21681 (arcanist changes)

Phabricator has several uses of the `--debug` flag being used with Mercurial. Use of this flag causes additional output to be added which Phabricator needs, however the behavior of `--debug` is not guaranteed to be stable, and in newer versions of Mercurial there have been additional output that has caused Phabricator to choke on parsing the output. This change removes several uses of `--debug` in favor of using `--template` with the `hg log` or `hg annotate` commands in combination with the `{p1.node}` or `{p2.node}` template format.

The use of `{p1node}` format in templates was added in [[ https://www.mercurial-scm.org/wiki/WhatsNew/Archive#Mercurial_2.4_.282012-11-01.29 | Mercurial 2.4 (2012) ]]. This format was deprecated in [[ https://www.mercurial-scm.org/wiki/WhatsNew#Mercurial_4.9_.282019-02-01.29 | Mercurial 4.9 (2019) ]] in favor of using `{p1.node}` format which is unclear when this new format was added (presumably earlier than Mercurial 4.9).

The use of `--template` with `hg annotate` is only officially supported in [[ https://www.mercurial-scm.org/wiki/Release4.6 | Mercurial 4.6 (2018) ]], though does appear to work in 4.5 but is not documented.

Since the `{p1node}` format was introduced in 2.4 this bumps the required version of `hg` to 2.4 (from 1.9). Since the `annotate --template` feature wasn't added until 4.6 (which is still fairly recent), the use of it is gated on a capability test, but still preferred for use where possible to avoid extraneous output from `--debug` flag.

Test Plan:
I verified I could do the following in a mercurial repository, while having mercurial 5.8 installed:
1. Navigate and view files in Diffusion under e.g. `/source/test-repo/`.
2. While viewing a file in Diffusion verified that I could view the blame of the file and the history/annotations looked accurate for the files I was browsing.
3. From the blame sidebar, select to view a commit which loaded and displayed changes properly.
4. View the history of the repository under e.g. `/source/test-repo/history/default/`. I verified the history looked correct and the tree-like structure showing relationship of commits also looked accurate.

I setup mercurial to run version 4.4, created a new repository, added some commits, and verified all the above behavior still works properly.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D21679
2021-07-09 12:35:20 -04:00
epriestley
5521f76fe4 Remove some "Phacility" and "epriestley" references
Summary:
Ref T13658. This just scrubs some of the simple references from the codebase.

Most of what's left is in documentation which won't be relevant for a fork and/or which I need to separately revise (or more-or-less delete) at some point anyway.

I removed the "install RHEL" and "install Ubuntu" scripts outright since I don't have any reasonable way to test them and don't plan to maintain them.

Test Plan: Grepped for "phacility", "epriestley"; ran unit tests.

Reviewers: cspeckmim

Reviewed By: cspeckmim

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13658

Differential Revision: https://secure.phabricator.com/D21678
2021-07-08 10:46:17 -07:00
Christopher Speck
14c516b7a1 Updating the filtering of Mercurial debug output
Summary:
With newer versions of Mercurial come newer debug messages which need filtered out.

1. In the scenario of Phabricator observing a hosted Mercurial repository which exists on a server in a multi-user environment it's possible that a repository computes branch cache at a tip revision which is not present. When this happens Mercurial will include in the debug output this information. This message indicates that the cache is going to be re-computed. See https://www.mercurial-scm.org/pipermail/mercurial/2014-June/047239.html.
2. Likely in some version with added or improved support for `pager` the debug info seems to indicate when a pager is being invoked for a command. This seems to print out regularly despite piping the stdout.
3. If the repository on Phabricator ever had the `largefiles` extension enabled then some additional details about "updated patterns" will print out.

Test Plan:
I verified an observed repository's history could be browsed, specifically the history of files which previously resulted in "Undefined offset: 1".

Added a unit test to check the results of `filterMercurialDebugOutput()`.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D21677
2021-07-06 17:27:57 -04:00
Christopher Speck
9637420861 Update a few random typos
Summary: Found a few typos which could be updated.

Test Plan:
I tested the Configuration page change by navigating to `/config` and verifying the page title set in the browser as well as the page title text on the page
|Before|After|
|---|---|
|{F9013208}|{F9013210}|
|{F9013300}|{F9013301}|

I verified the Conduit error message by navigating to `/auth/start/?__conduit__=1`
{F9013289}

The CircleCI error message was not verified due to the involvement of testing with CircleCI however the change is very minor and has very little risk of impacting any functionality.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D21675
2021-06-27 17:38:34 -04:00
epriestley
a641ec82a3 Add an "Authority" control to Packages to support "Watcher" packages
Summary: See T13657. An install has "watcher" packages which should not allow owners to "Force Accept" other packages.

Test Plan:
  - Created package A, which I own, on "/", with "Weak" authority.
  - Created package B, which I do not own, on "/src".
  - Created a revision which touches "/src" and added package B as a reviewer.
  - Attempted to accept the revision...
    - Before patch: permitted to "Force Accept" for package B.
    - After patch: not allowed to "Force Accept" for package B.
  - Verified that setting package "A" back to "Strong" authority allows a force-accept for package B.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D21674
2021-06-25 13:48:46 -07:00
Amaury Séchet
e7740c8669 Add HarbormasterHookController as an entry point for all Harbormaster hooks
Summary: This makes the set of hooks easily extensible, as a first step toward integrating more 3rd party CI in phorge.

Test Plan: Send requests to `/harbormaster/hook/circleci/` and `/harbormaster/hook/buildkite/` and check they run the proper handler.

Reviewers: O1 Blessed Committers, Matthew

Reviewed By: O1 Blessed Committers, Matthew

Subscribers: Matthew, speck, tobiaswiese

Maniphest Tasks: T15018

Differential Revision: https://we.phorge.it/D25005
2021-06-21 10:48:30 +00:00
epriestley
51cb7a3db9 Provide an ad-hoc maintenance lock for clustered repositories
Summary: Ref T13614. Provide "bin/repository lock" to temporarily lock repositories for manual maintenance.

Test Plan:
  - Read instructions.
  - Used `bin/repository lock` according to the instructions.
  - Saw Storage tab in Diffusion report lock held during maintenance, released after it completes.
  - Saw "maintenance" push log generated and repository version bump.
  - Tried to lock some invalid repositories.

Maniphest Tasks: T13614

Differential Revision: https://secure.phabricator.com/D21671
2021-06-01 08:29:54 -07:00
epriestley
12a5eb4062 Allow maintenance scripts to write synthetic events to the push log that act as repository updates
Summary:
Ref T13614. When a script holds the write lock but modifies the repository directly (rather than by pushing), the repository version won't change when the script releases the write lock. Thus, the writes may not propagate to other nodes (it depends which node lucks out and accepts the next write).

To guarantee that writes propagate, allow these scripts to pretend they pushed the repository. These are bare-bones valid events flagged as "Maintenance".

Test Plan:
  - Wrote a script to hold the write lock, wait (or pretend to do something), then release the write lock.
  - Applied patches, modified script to use new APIs ("newMaintenanceEvent()").
  - Ran script, saw repository verison bump and relevant push logs:

{F8814923}

Maniphest Tasks: T13614

Differential Revision: https://secure.phabricator.com/D21670
2021-06-01 08:29:54 -07:00
epriestley
bdda7eed07 Improve display behavior for write locks held by omnipotent users
Summary:
Ref T13614. When an omnipotent user calls "synchronizeWorkingCopyBeforeWrite()", we record a WorkingCopyVersion record with a null "userPHID". The UI then renders this as "Unknown Object (????)".

Improve this behavior:

  - When no PHID is available, just render nothing in the UI (this doesn't seem meaningfully different from no version existing at all).
  - Allow callers to provide an acting user PHID, similar to Editor.

There's currently no way to perform this kind of write legitimately in the upstream, but T13614 is providing one.

Test Plan:
  - Wrote a script that calls "synchronizeWorkingCopyBeforeWrite()" as the omnipotent user.
  - Ran script, saw "Unknown Object (????)" in the UI.
  - Applied UI fix, saw empty UI.
  - Applied "acting as" fix, modified script to act as the Diffusion application, ran script, saw "Diffusion" attribution in UI.

{F8814806}

Maniphest Tasks: T13614

Differential Revision: https://secure.phabricator.com/D21669
2021-06-01 08:29:53 -07:00
epriestley
a8f429cdbf Correct an issue where "dashboard.panel.edit" fatals in the API console
Summary:
Ref T13650. Currently, viewing the API console help page for this method fatals because it constructs a generic, untyped panel.

As a step toward improving this, generate a concrete panel type instead. This isn't the best possible fix, see T13650 for discussion.

Test Plan: Viewed "dashboard.panel.edit" API page, now saw a usable page.

Maniphest Tasks: T13650

Differential Revision: https://secure.phabricator.com/D21668
2021-05-31 23:49:14 -07:00
epriestley
95662ae8f1 Don't attempt to test capabilities on incomplete handles
Summary:
As backstory: I accidentally added the subscriber `PHID-USER-abcd` to `T1` on this install by calling `maniphest.edit`. I intended to edit `T1` on my local install.

This edit is permitted for messy technical reasons, described in T13429. It's not valid, but it's hard to prevent.

The state we reach is also possible even if the edit is rejected (i.e., someone can go manually update the database).

Regardless of how we get into this state, the state (a non-user subscriber) breaks the UI on the task page when it attempts to test if the subscriber can see the task.

To prevent this, only claim that a Handle can have capabilities if the handle is complete. If the handle is incomplete (an invalid or restricted object), it either can't be meaningfully tested for capabilities or the viewer isn't allowed to know them.

Test Plan:
Viewed `T1` on this install, saw a fatal. Applied the same edit to `T1` locally, got the same fatal. Applied patch, no more fatal. Now saw "Unknown Object (User)" in subscriber curtain.

Specifically, the fatal is:

> Attempting to test capability "view" for handle of type "USER", but this capability has not been attached.

Differential Revision: https://secure.phabricator.com/D21662
2021-04-07 14:56:26 -07:00
epriestley
428fff2e58 Fix an issue when undoing mutiple inline comment deletions
Summary:
Ref T13559. If you create comments A and B, then delete comments A and B, then undo the deletion of A, the UI undoes the deletion of B instead.

This is becasue the undo rows are shipped down with a static scalar metadata reference. When copied multiple times to create multiple undo rows, they reference the same data object.

Preventing this in the general case is a problem with greater scope. For now, just avoid rendering these rows with any metadata so they don't alias a single data object.

Test Plan:
  - Created comments A, B.
  - Deleted comments A, B.
  - Clicked "Undo" on A.
    - Before: Deletion of "B" undone.
    - After: Deletion of "A" undone.

Maniphest Tasks: T13559

Differential Revision: https://secure.phabricator.com/D21650
2021-03-29 09:00:26 -07:00
epriestley
aa70b008f3 Skip "git for-each-ref" when identifying deleted commits
Summary:
Ref T13647. The ref discovery process prunes commits that no longer exist in the repository before executing "git log <new heads> --not <old heads>" to identify newly published commits.

If we don't do this, the "git log" command will fail if any old head has been pruned from the repository.

Currently, this test for missing commits starts with a call to "git for-each-ref" to attempt to resolve symbols as tag or branch names, but:

  - this is painfully slow in repositories with many refs; and
  - this is incorrect (not consistent with "git" behavior) for 40-character hex strings, which Git will never resolve as symbolic names.

Instead, when a symbol is a 40-character hex string, skip "git for-each-ref" and jump directly to "git cat-file --batch-check".

Test Plan:
  - Ran `bin/repository update` in a repository with 65K refs and extra debugging info.
    - Before: took ~30s, three calls to `git for-each-ref`.
    - After: took ~20s, two calls to `git for-each-ref`. Same resolution result on queries.

Maniphest Tasks: T13647

Differential Revision: https://secure.phabricator.com/D21658
2021-03-28 11:04:29 -07:00
epriestley
5b8b5f2141 Correct issue with "bindings" conduit attachment
Summary: Ref T13641. These conditions are swapped, and "activeBindings" loads more data than necessary while "bindings" doesn't load enough.

Test Plan: Called method with each attachment, got good results instead of an exception.

Maniphest Tasks: T13641

Differential Revision: https://secure.phabricator.com/D21657
2021-03-27 09:26:24 -07:00
epriestley
61272e7ac3 Correct "getActiveBindings()" method name
Summary: This method was incorrectly renamed by D21628. See <https://discourse.phabricator-community.org/t/unable-to-lease-host/4696>.

Test Plan: Looked at it, will deploy etc.

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D21644
2021-03-24 10:15:17 -07:00
epriestley
db9191f9a8 Correct minor "jump to symbol" behavior in Differential
Summary:
Ref T13644. Ref T13638.

  - Double-encode the symbol that is used as a path component, similar to Diffusion.
  - Fix an outdated reference to ".path", which provided context for symbol lookup.
  - Prevent command-clicking headers from looking up the path as a symbol.

Test Plan:
  - Command-clicked headers, no longer got a symbol.
  - Command-clicked stuff with "/", saw it double-encoded and decoded properly.
  - Command-clicked normal symbols, saw "path" populate correctly.

Maniphest Tasks: T13644, T13638

Differential Revision: https://secure.phabricator.com/D21641
2021-03-17 15:32:07 -07:00
epriestley
d6ed9392d4 Replace Differential "lint stars" with icons
Summary:
Ref T9764. These stars are inconsistent, not accessible, and generally weird. They predate icons.

Update them to use icons instead.

Test Plan:
{F8545721}

{F8545722}

{F8545723}

Maniphest Tasks: T9764

Differential Revision: https://secure.phabricator.com/D21640
2021-03-17 13:45:52 -07:00
epriestley
527dd3ce50 Replace Differential "unit stars" with icons
Summary:
Ref T9764. These "star" icons are unclear, inconsistent, and not friendly to colorblind users.

They date from a time long ago when the product didn't have icons.

Modernize them and make them more consistent with the similar statuses in Harbormaster.

Test Plan:
{F8545690}

{F8545691}

{F8545692}

Maniphest Tasks: T9764

Differential Revision: https://secure.phabricator.com/D21639
2021-03-17 13:45:52 -07:00
epriestley
1d1003af78 When using "Update Diff" from the web UI, prefill "Repository" properly
Summary: Ref T9499. When using the manual "Update Diff" workflow on the web, the "Repository" field isn't pre-filled properly. This can lead to revisions losing their repository after a manual update.

Test Plan: Did a manual update of a revision with a repository, saw it stick.

Maniphest Tasks: T9499

Differential Revision: https://secure.phabricator.com/D21638
2021-03-17 12:23:24 -07:00
epriestley
ff0a4a2c6f Use a less-misleading example for Conduit custom constraints
Summary:
See PHI2032. The Conduit UI shows some "Example Custom Constraints" that are intended to be generic, but use of "statuses" is misleading since many objects have a status and most of them don't support these specific values.

Make it more clear that these are generic values.

Test Plan: Read new text.

Differential Revision: https://secure.phabricator.com/D21637
2021-03-17 11:47:41 -07:00
epriestley
e6a23274fe Default the Almanac Devices query to "Active Devices"
Summary: Ref T13641. Now that we can provide an "Active Devices" query, provide it and make it the default.

Test Plan: Viewed Almanac devices, got a list of active devices. Clicked "All Devices" to get all devices.

Maniphest Tasks: T13641

Differential Revision: https://secure.phabricator.com/D21636
2021-03-16 15:51:51 -07:00
epriestley
12341e4bc8 Forbid disabled devices from authenticating via SSH or HTTP
Summary:
Ref T13641. Phabricator sometimes makes intracluster requests that authenticate as a device.

Forbid these requests from authenticating as a disabled device.

Test Plan:
  - Ran `bin/ssh-exec --phabricator-ssh-device ...` as an enabled/disabled device (worked; sensible error).
  - Made Conduit calls as an enable/disabled device (worked; sensible error).

Maniphest Tasks: T13641

Differential Revision: https://secure.phabricator.com/D21635
2021-03-16 15:51:51 -07:00
epriestley
3267859aee Modernize "mailKey" on Fund initiatives
Summary: Ref T13065. Migrate "mailKey" and drop the column.

Test Plan: Ran "bin/storage upgrade", got a clean upgrade, saw migrated values in mail properties table.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13065

Differential Revision: https://secure.phabricator.com/D21634
2021-03-16 15:51:50 -07:00
epriestley
eeb009b4fa Modernize "mailKey" for Calendar Event
Summary: Ref T13065. Migrate "mailKey" and drop the column.

Test Plan: Ran "bin/storage upgrade", got a clean report, saw migrated data in mail properties table.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13065

Differential Revision: https://secure.phabricator.com/D21633
2021-03-16 15:51:50 -07:00
epriestley
86669ab54f Modernize "mailKey" for Almanac Networks
Summary: Ref T13065. Ref T13641. Migrate "mailKey" and drop the column.

Test Plan: Ran "bin/storage upgrade", got a clean report, saw keys migrate in mail properties table.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13641, T13065

Differential Revision: https://secure.phabricator.com/D21632
2021-03-16 15:51:49 -07:00
epriestley
8226b3c880 Modernize "mailKey" on Almanac Namespaces
Summary: Ref T13065. Ref T13641. Migrate "mailKey" and drop the column.

Test Plan: Ran "bin/storage upgrade", got a clean report, saw mail keys migrated to mail properties table.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13641, T13065

Differential Revision: https://secure.phabricator.com/D21631
2021-03-16 15:51:49 -07:00
epriestley
7c0e33c34d Modernize "mailKey" for Almanac Bindings
Summary: Ref T13065. Ref T13641. Migrate "mailKey" and drop the column.

Test Plan: Ran "bin/storage upgrade", got a clean report and saw binding mail keys in the mail properties table.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13641, T13065

Differential Revision: https://secure.phabricator.com/D21630
2021-03-16 15:51:48 -07:00
epriestley
fadb2bd52b Modernize "mailKey" on AlamnacService
Summary: Ref T13641. Ref T13065. Migrate and drop the onboard "mailKey" column for Almanac Services.

Test Plan: Ran storage migration, got clean report, saw migrated value in mail properties table.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13641, T13065

Differential Revision: https://secure.phabricator.com/D21629
2021-03-16 15:51:48 -07:00
epriestley
15c0c895a5 Make upstream callers respect "active bindings" when querying Almanac
Summary: Ref T13641. Make "active bindings" a real query and make callers that only care about active bindings only query for active bindings.

Test Plan:
  - Queried for "bindings" and "activeBindings" via Conduit.
  - Disabled/enabled devices, saw binding status update in UI.
  - Loaded Diffusion cluster layout.
  - Grepped for `needBindings()`, `getActiveBindings()`, etc.

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13641

Differential Revision: https://secure.phabricator.com/D21628
2021-03-16 15:51:47 -07:00
epriestley
5d64fb1815 Add a "status" property to Almanac devices
Summary:
Ref T13641. Add a "status" property with most of the relevant support code.

This currently has no impact on use of the device or bindings by Diffusion or Drydock: they ignore the status of devices bound to services.

Test Plan:
  - Created a new device.
  - Changed the status of a device via web and API.
  - Queried for devices via API.
  - Searched for active and disabled devices.
  - Viewed UI in list view, detail view.
  - Used typeahead to add a new binding to an interface on a disabled device, got disabled hint in typeahead UI.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13641

Differential Revision: https://secure.phabricator.com/D21627
2021-03-16 15:51:47 -07:00
epriestley
9003a45369 Make minor Almanac device modernization updates
Summary: Ref T13641. Provide minor modernizations before adding a "Disabled" state.

Test Plan: Browsed devices, created a new device.

Maniphest Tasks: T13641

Differential Revision: https://secure.phabricator.com/D21626
2021-03-16 15:51:46 -07:00
epriestley
c3e6db6f0b Migrate Almanac Device "mailKey" to modern storage
Summary: Ref T13065. See similar changes attached to that task.

Test Plan: Ran migration, got a clean database state, saw mail keys populate in mail property table.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13065, T13641

Differential Revision: https://secure.phabricator.com/D21625
2021-03-16 15:51:46 -07:00
epriestley
42c0c0e3d2 Remove or correct various "phabricator/" references to "libphutil"
Summary:
Ref T13395. "libphutil/" was stripped for parts, but some documentation still references it. This is mostly minor corrections, but:

  - Removes "Javelin at Facebook", long obsolete.
  - Removes "php FPM warmup", which was always a prototype and is obsoleted by PHP preloading in recent PHP.

Test Plan: `grep` / reading

Maniphest Tasks: T13395

Differential Revision: https://secure.phabricator.com/D21624
2021-03-16 10:28:07 -07:00
epriestley
30d58de1bc Expose "affectedPaths" to "differential.revision.search" Conduit API method
Summary: Ref T13639. Support querying by "affectedPaths" in the API.

Test Plan: {F8539347}

Maniphest Tasks: T13639

Differential Revision: https://secure.phabricator.com/D21621
2021-03-15 16:16:11 -07:00
epriestley
6d33ba7dc4 Move Diffusion to "withPaths()" for "Recent Open Revisions", and remove "withPath()" from DifferentialRevisionQuery
Summary: Ref T13639. Move Diffusion to use the new API and get rid of the old API now that it no longer has any callers.

Test Plan:
Grepped for remaining callers.

{F8539335}

Maniphest Tasks: T13639

Differential Revision: https://secure.phabricator.com/D21620
2021-03-15 16:16:11 -07:00
epriestley
925c9a71e7 Support a "withPaths()" API in DifferentialRevisionQuery, and use it on the revision view
Summary: Ref T13639. Move away from "withPath(..., ...)" to "withPaths(...)".

Test Plan: {F8539323}

Maniphest Tasks: T13639

Differential Revision: https://secure.phabricator.com/D21619
2021-03-15 16:16:11 -07:00
epriestley
26c68942bd Update "AffectedPath" table when a revision's repository changes
Summary:
Ref T13639. There's currently a hard-to-hit bug where editing the "Repository" of a revision doesn't update this index.

Instead: update the index on repository change, not just diff update.

Test Plan:
  - Updated a revision, used debug view to see index update.
  - Changed repository on a revision, used debug view to see index update.

Maniphest Tasks: T13639

Differential Revision: https://secure.phabricator.com/D21618
2021-03-15 16:16:10 -07:00
epriestley
01ea84029d Update table schema for "AffectedPath" table
Summary:
Ref T13639. Make schema changes:

  - Make repositoryID nullable, for revisions with no repository.
  - Remove "epoch", which has no readers and no clear use.
  - Change the ordering of the key, since "pathID" has more unique values and no queries ever issue without it.

Test Plan:
  - Ran `bin/storage upgrade`, got a clean schema.
  - Reindexed all revisions with an external script.
  - Reviewed index via debug UI, saw appropriate index for non-repositoy revisions.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13639

Differential Revision: https://secure.phabricator.com/D21617
2021-03-15 16:16:10 -07:00
epriestley
38ef910da8 Move "Affected Path" index updates to a separate class
Summary:
Ref T13639. Move operations related to updating the "AffectedPath" index to a dedicated class.

This change has no functional effect and only moves code.

Test Plan: Used an external script to rebuild every revision index; destroyed a revision with `bin/remove destroy`.

Maniphest Tasks: T13639

Differential Revision: https://secure.phabricator.com/D21616
2021-03-15 16:16:09 -07:00
epriestley
e919b4c35a Add a debug view of the "Affected Path" index to Differential
Summary:
Ref T13639. The "Affected Path" table is currently hard to inspect: there's no UI, and using MySQL just gives you a bunch of IDs.

Add a simple UI and a debug-mode link to it.

Test Plan:
{F8539098}

{F8539099}

Maniphest Tasks: T13639

Differential Revision: https://secure.phabricator.com/D21615
2021-03-15 16:16:09 -07:00
epriestley
c317d16bdd Lift peculiar side effect of path indexing out of indexer
Summary:
Ref T13639. Updating the affected path table has a peculiar side effect from D19426, which is a simplification of a peculiar side effect from earlier.

Don't condition Owners behavior on path index behavior.

Test Plan: Created a revision.

Maniphest Tasks: T13639

Differential Revision: https://secure.phabricator.com/D21614
2021-03-15 16:16:08 -07:00
epriestley
bcd592cf7e Remove support for "paths" parameter in "differential.query"
Summary: See T13639. This change simplifies providing a more modern approach to querying this data via "differential.revision.search".

Test Plan: Called "differential.query" with paths (got an error) and without paths (got a valid query result).

Differential Revision: https://secure.phabricator.com/D21613
2021-03-15 16:16:08 -07:00
epriestley
e730f55e88 Retitle "Recently Open Revisions" panel to "Recent Open Revisions"
Summary:
Ref T13639. In D17754, this:

> OPEN REVISIONS
> Recently updated open revisions affecting this file.

...was simplified into:

> RECENTLY OPEN REVISIONS

This is a bit misleading, since the panel doesn't contain "recently open" results. Use "Recent Open" instead, which is a bit more consistent with other product text. This is still slightly misleading, but probably close enough.

Test Plan: Read text.

Maniphest Tasks: T13639

Differential Revision: https://secure.phabricator.com/D21612
2021-03-15 16:16:07 -07:00
epriestley
b11c6fcacd Clarify the behavior of "audit.can-author-close-audit"
Summary:
Ref T13631. This option has a behavior other than the behavior implied by the name and documented.

Document the correct behavior, at least. This can likely be removed after T10574.

Test Plan: Read config option help in Config.

Maniphest Tasks: T13631

Differential Revision: https://secure.phabricator.com/D21610
2021-03-12 09:20:16 -08:00
epriestley
4b529e6009 Fix a followup notification paging error with partial objects
Summary: Ref T13623. In D21603, I made the "partial object" this query returns a raw row, which paging keys can no longer be extracted from properly.

Test Plan: Paged notifications to page 2, no longer saw an error.

Maniphest Tasks: T13623

Differential Revision: https://secure.phabricator.com/D21609
2021-03-12 09:09:45 -08:00
epriestley
31c9d4094f Improve routing of "/robots.txt", "/favicon.ico", and "/status/" on Short and Blog sites
Summary: Ref T13636. Add routing for "/robots.txt", "favicon.ico", and "/status/" on the ShortSite and BlogSite.

Test Plan: Visted all resources (and 404 pages) on Short and Blog sites, and Platform site.

Maniphest Tasks: T13636

Differential Revision: https://secure.phabricator.com/D21607
2021-03-11 14:05:39 -08:00
epriestley
36c6eb9663 Improve routing of "/robots.txt", "/favicon.ico", "/status/", and 404 on custom Sites
Summary:
Fixes T12919. Fixes T13636. Prior to this change, some well-known resource paths don't route on sites like ResourceSite.

- `/robots.txt`: Make it route on ResourceSite and just deny the whole site.
- `/favicon.ico`: Make it route on ResourceSite.
- `/status/`: Make it route on ResourceSite.
- 404: Make it render a 404 on ResourceSite.

Test Plan:
  - Visited all URIs on ResourceSite, got sensible responses.
  - Visited all URIs on main site.
  - Visited 404 while logged out, got login page.

Maniphest Tasks: T13636, T12919

Differential Revision: https://secure.phabricator.com/D21606
2021-03-11 14:05:39 -08:00
epriestley
4484946cfd In JSON DocumentEngine, preserve the distinction between "{}" and "[]"
Summary:
Ref T13635. Currently, the JSON DocumentEngine uses "phutil_json_decode()", but this can confuse "{}" and "[]".

Be more careful about how the JSON value is decoded, to preserve the distinction.

Test Plan: {F8520479}

Maniphest Tasks: T13635

Differential Revision: https://secure.phabricator.com/D21605
2021-03-11 12:49:56 -08:00
epriestley
0815891e42 Fix an error when users receive notifications about objects they can no longer see
Summary:
Ref T13623. The change in D21577 could lead to a case where we try to access stories the user can't see.

Move the story-loading piece to "willFilterPage()" to make our way thorugh this.

Test Plan:
  - Made FeedStory return nothing to simulate invisible notifications, loaded page.
    - Before: index access fatal.
    - After: clean "no notifications".
  - Loaded notifications normally, saw normal notifications.

Maniphest Tasks: T13623

Differential Revision: https://secure.phabricator.com/D21603
2021-03-11 10:44:42 -08:00
epriestley
4cff4dc68b Interpret search tokens in the for "_..." as substring search
Summary: Ref T13632. Users searching for `__FILE__`, etc., almost certainly mean to perform a substring search.

Test Plan: Added tests and made them pass. Searched for various tokens, saw compiler interpretation in UI.

Maniphest Tasks: T13632

Differential Revision: https://secure.phabricator.com/D21602
2021-03-10 12:00:01 -08:00
epriestley
afdef332fb Allow "transaction.search" to be called on an object type
Summary:
Ref T13631. This supports a more robust version of "poll for updates by using dateModified window queries" that uses transactions as a logical clock.

This is particularly relevant for commits, since they don't have a "dateModified" at time of writing.

Test Plan:
  - Queried for transactions by type and object.
  - Issued various invalid transaction queries, got appropriate errors.

Maniphest Tasks: T13631

Differential Revision: https://secure.phabricator.com/D21601
2021-03-10 10:20:03 -08:00
epriestley
404b55ce57 Give audit statuses API constants that match their UI strings
Summary: Ref T13631. These strings were a little inconsistent; make them more consistent.

Test Plan: Called `diffusion.commit.search` with the appropriate attachment, saw slightly more consistent statuses.

Maniphest Tasks: T13631

Differential Revision: https://secure.phabricator.com/D21600
2021-03-10 10:20:03 -08:00
epriestley
ac2f5a1046 Modernize and clean up "PhabricatorAuditStatusConstants"
Summary:
Ref T13631. Move "PhabricatorAuditStatusConstants" to a more modern object ("PhabricatorAuditRequestStatus").

Expose the status value via Conduit.

Test Plan:
  - Ran `bin/audit delete`.
  - Viewed a commit with auditors in the web UI.
  - Grepped for affected symbols.
  - Called Conduit with the "auditors" attachment, saw auditor statuses.

Maniphest Tasks: T13631

Differential Revision: https://secure.phabricator.com/D21599
2021-03-10 09:21:55 -08:00
epriestley
2636d84d0c Remove very old Audit status constants and AuditRequest data
Summary:
Ref T13631. See that task for discussion.

  - "NONE": Probably never used?
  - "CC": Obsoleted by subscribers.
  - "AUDIT_NOT_REQUIRED": For Owners packages, obsoleted by edges.
  - "CLOSED": For "Close Audit", obsoleted by "Request Verification".

Test Plan:
  - Grepped for constants, browsed Diffusion.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13631

Differential Revision: https://secure.phabricator.com/D21598
2021-03-10 09:21:54 -08:00
epriestley
55532b3f74 Add a very basic "auditors" attachment to "differential.commit.search"
Summary: Ref T13631. For now, this only shows the auditor PHID. The current status constants could use some cleanup before they're exposed.

Test Plan: Queried with "auditors" attachment, saw basic auditor information.

Maniphest Tasks: T13631

Differential Revision: https://secure.phabricator.com/D21597
2021-03-10 09:21:54 -08:00
epriestley
bfe7cdc5a2 Provide default image alt text in more contexts and support custom alt text
Summary:
Ref T13629.

  - Allow files to have custom alt text.
  - If a file doesn't have alt text, try to generate a plausible default alt text with the information we have.

Test Plan:
  - Viewed image files in DocumentEngine diffs, files, `{Fxxx}` embeds, and lightboxes.
  - Saw default alt text in all cases, or custom alt text if provided.
  - Set, modified, and removed file alt text. Viewed timeline and feed.
  - Pulled alt text with "conduit.search".

Maniphest Tasks: T13629

Differential Revision: https://secure.phabricator.com/D21596
2021-03-04 16:51:23 -08:00
epriestley
fceb9a3f9e Hide the "Author" field in Differential on the web create workflow
Summary:
Ref T13628. When creating a revision, hide the recently introduced "Author" field.

It seems exceedingly unlikely that anyone would ever want to adjust this field when creating a revision, and this workflow already has a lot of fields and complexity, so this likely adds more noise than signal on the balance.

Test Plan:
  - Created revisions via copy/paste on the web workflow, no longer saw "Author".
  - Used "Edit Revision" to adjust the author normally.

Maniphest Tasks: T13628

Differential Revision: https://secure.phabricator.com/D21595
2021-03-04 15:50:15 -08:00
epriestley
a9704428ff In Audit, use repository identities to prevent author-auditors
Summary:
See PHI2015. Diffusion attempts to prevent a commit's author from being made an auditor, but currently uses an out-of-date method for identifying the author.

Use the modern ("Repository Identity" aware) method instead.

Test Plan:
  - Authored a commit as user "X", mapped to my account.
  - Pushed/imported/discovered it.
  - Changed the identity mapping for "X" from my account to a different account.
  - Tried to add myself as an auditor.
    - Before: error, "author can't be an auditor".
    - After: succeeds.
  - Tried to add the newly mapped user as an auditor. This correctly fails with the "author can't be an auditor" error.

It's possible to put commits into a wonky state by remapping the author identity to a user who is already an auditor, but I think that isn't important and we can't do much about it, realistically.

Differential Revision: https://secure.phabricator.com/D21594
2021-03-04 09:33:49 -08:00
epriestley
9b6a030292 Use the same icon for "Resigned" in Differential and Diffusion
Summary:
See PHI2015. Currently, "resigned" reviewers/auditors get different icons in Differential and Diffusion.

The Diffusion icon is exceptionally poor and confusing: it does not communicate "resigned" and it is similar to other icons.

For clarity and consistency, use the Differential icon (a grey "X") in both applications.

Test Plan: {F8492303}

Differential Revision: https://secure.phabricator.com/D21593
2021-03-04 09:33:48 -08:00
epriestley
f3caf7e9fc Improve rendering behavior for "foist upon" with an empty tokenizer
Summary:
Ref T13628. If you "Foist Upon", then delete the value in the tokenizer, then scroll down to the preview, you currently get an ugly "Unknown Object (???)" rendering.

This is technically correct in some vague sense and the transaction won't apply, but provide a more aesthetic rendering instead.

Test Plan: {F8487050}

Maniphest Tasks: T13628

Differential Revision: https://secure.phabricator.com/D21592
2021-03-03 16:08:26 -08:00
epriestley
dec1413684 Provide a more general "Author" transaction for Differential
Summary:
Ref T13628. Currently, Differential has a "Commandeer" action, but no way to edit the author otherwise.

This is largely archaic: there is no reason to prevent editing the author, and this makes it difficult to undo mistakes if you commandeer by accident.

Instead, provide a normal "Author" field and a "Foist Upon" action, similar to the "Owner" and "Claim/Assign" fields in Maniphest.

Test Plan:
  - Applied author transactions as the old author ("foisted"), the new author ("commandeerd"), and an arbitrary third party ("changed author").
  - Tried to unassign author, etc.
  - Viewed stories in feed and transaction timeline.
  - Saw sensible automatic reviewer changes.
  - Used existing "Commandeer" action, which is unchanged.
  - Called "transaction.search" and saw reasonable transaction values.

Maniphest Tasks: T13628

Differential Revision: https://secure.phabricator.com/D21591
2021-03-03 15:49:22 -08:00
Matthew Bowker
c0ac5be8a8 Provide new tool "bin/user approve" to approve an account.
Summary: T13578

Test Plan:
This method uses the existing transaction.  As such, most of the testing focused on the integration between the workflow and transaction.  The only change made to the transaction was to allow an omnipotent user to make the change in addition to an admin.

Other than that, I removed the "approved" flag from the user, then ran the command-line utilty until the user was successfully approved.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T13578

Differential Revision: https://secure.phabricator.com/D21587
2021-03-03 04:35:50 +00:00
epriestley
466013f11a Prevent external connections from being mutated on held locks
Summary: Ref T13627. This makes the API harder to misuse: setting an external connection on a held lock isn't a meaningful operation. Prevent it.

Test Plan: Added a failing test, made it pass.

Maniphest Tasks: T13627

Differential Revision: https://secure.phabricator.com/D21584
2021-03-02 13:44:16 -08:00
epriestley
55f4a258d2 When updating revisions in responset to commits, use the omnipotent viewer to pull diffs
Summary:
Ref T13625. See that task for discussion.

Currently, the Viewer when performing revision updates in response to commits may be an arbitrary low-privilege user (an Application, a disabled User, a bot, a mailing list, etc).

Today, this leads to an exception when trying to make API calls.

Ideally, we probably would not perform the update in these cases. However, performing the update isn't a policy violation and is generally less surprising than not performing it, so continue performing it for now: just use the omnipotent user to interact with the API.

Test Plan:
  - Authored a commit as a bot user without permission to view the repository or revision.
  - Commented out a couple of caches, and used `bin/repository reparse --publish ...` to republish the commit.
    - Before: exception when trying to interact with the API.
    - After: clean publish.

Maniphest Tasks: T13625

Differential Revision: https://secure.phabricator.com/D21582
2021-03-01 11:11:34 -08:00
epriestley
0a3093ef9c Fix an issue where paginating notifications could fail a GROUP BY test
Summary:
Ref T13623. When paginating notifications, we may currently construct a query which:

  - loads from non-unique rows; and
  - returns multiple results.

In particular, `chronologicalKey` isn't unique across the whole table (only for a given viewer). We can get away with this because no user-facing view of notifications is truly "every notification for every viewer" today.

One fix would be to implicitly force the paging query to include `withUserPHIDs(viewerPHID)`, but puruse a slightly more general fix:

  - Load only unique stories.
  - Explictly limit the pagination subquery to one result.

Test Plan:
  - Set page size to 1, inserted duplicate notifications of all stories for another user, clicked "Next", got the GROUP BY error.
  - Applied the "only load unique stories" part of the change, got a "expected one row" error instead.
  - Applied the "limit 1" part of the change, got a second page of notifications.

Maniphest Tasks: T13623

Differential Revision: https://secure.phabricator.com/D21577
2021-02-26 15:11:25 -08:00
epriestley
10162ad43b Support an SSH error log
Summary: Ref T13624. Depends on D21578. In "sshd" subprocess contexts, use "PhutilErrorLog" to direct errors to both stderr and, if configured, a logfile on disk.

Test Plan:
  - Confiugured an error log.
  - Forced `ssh-auth` to fatal.
  - Saw errors on stderr and in log.

Maniphest Tasks: T13624

Differential Revision: https://secure.phabricator.com/D21579
2021-02-26 14:54:54 -08:00
epriestley
f970b350ea Correct behavior of "writable" Almanac service binding for repository services
Summary: Ref T13611. This property worked correctly when implemented in D19357. The behavior was broken by D20775, which tested node-level routing but did not specifically re-test the "writable" property. This was difficult to spot because ref query outcomes weren't observable in the UI, and the ref itself had the correct property value.

Test Plan:
See D21575. After this change, the UI shows the correct state, rather than showing a read-only service ref as writable:

{F8465865}

Maniphest Tasks: T13611

Differential Revision: https://secure.phabricator.com/D21576
2021-02-25 12:29:17 -08:00
epriestley
39077be746 Add an internal service ref panel to repository "Storage" information
Summary: Ref T13611. Currently, the "writable" property on service bindings has no effect because of a trivial bug. Provide more information in the UI to make this kind of problem observable.

Test Plan:
Viewed "Storage" section of management UI, saw a more-obvious problem with ref management (a non-writable ref is listed as writable).

{F8465851}

Maniphest Tasks: T13611

Differential Revision: https://secure.phabricator.com/D21575
2021-02-25 12:29:17 -08:00
epriestley
e9804bb7e5 Provide hovercards for generic edge stories, and include more message information in commit hovercards
Summary:
Ref T13620.

  - Make generic edge stories render links with hovercards. Other story types (like subscriptions) already do this so I'm fairly certain this is just old code from before hovercards.
  - Include a longer commit message snippet in hovercards.

Test Plan: {F8465645}

Maniphest Tasks: T13620

Differential Revision: https://secure.phabricator.com/D21574
2021-02-25 10:29:58 -08:00
epriestley
6bfa990254 When mapping phantom comments across changes, correct an off-by-one issue
Summary:
Ref T13617. When an inline comment is added inside a block of added lines, it currently ends up off-by-one when porting forward.

This is a disagreement between the mapping engine and the display engine about what "offset" means. Choose the simpler of the two interpretations.

Test Plan:
  - Created a revision with the diff in T13617.
  - Added an inline in the middle of the added block.
  - Updated the revision with the same diff.
    - Before: inline incorrectly moves up by one line.
    - After: inline maps correctly.

Maniphest Tasks: T13617

Differential Revision: https://secure.phabricator.com/D21572
2021-02-23 11:08:11 -08:00
epriestley
5bfd6bda77 Provide a more structured result log for Herald rules
Summary: Ref T13586. In the footsteps of D21563, make Herald rule results more formal and structured to support meaningful exception reporting.

Test Plan:
Ran various Herald rules and viewed transcripts, including rules with recursive dependencies and condition exceptions.

{F8447894}

Maniphest Tasks: T13586

Differential Revision: https://secure.phabricator.com/D21565
2021-02-19 11:16:22 -08:00
epriestley
b047653e53 Lift core of "HeraldConditionResult" to "HeraldTranscriptResult"
Summary: Ref T13586. Lift the behavioral core of "HeraldConditionResult" into a new abstract base "HeraldTranscriptResult", with the intent to introduce a "HeraldRuleResult".

Test Plan:
  - Ran Herald rules, reviewed transcripts.
  - This change should have no behavioral effect.

Maniphest Tasks: T13586

Differential Revision: https://secure.phabricator.com/D21564
2021-02-19 11:16:21 -08:00
epriestley
e77ae13d5c Provide a more structured result log for Herald conditions
Summary:
Ref T13586. Currently, Herald condition logs encode "pass" or "fail" robustly, "forbidden" through a sort of awkward side channel, and can not properly encode "invalid" or "exception" outcomes.

Structure the condition log so results are represented unambiguously and all possible outcomes (pass, fail, forbidden, invalid, exception) are clearly encoded.

Test Plan:
{F8446102}

{F8446103}

Maniphest Tasks: T13586

Differential Revision: https://secure.phabricator.com/D21563
2021-02-19 11:16:21 -08:00
epriestley
5408429466 Separate Herald transcripts into several different views
Summary: Ref T13586. The Herald transcript page has become more and more complicated over time, and recently added "Transactions" and "Profiler" sections. Split these across separate navigation tabs to limit the maximum complexity of any single view and make it easier to navigate to particular sections, like the profiler section.

Test Plan: Viewed various transcripts, saw nice digestible sections.

Maniphest Tasks: T13586

Differential Revision: https://secure.phabricator.com/D21493
2021-02-19 11:16:20 -08:00
epriestley
be0bb68f65 Remove Facebook OAuth dependency on "security_settings" property
Summary: Ref T13615. This property was removed from the Facebook API at some point, perhaps November 2020. Stop relying no it.

Test Plan: Created a local Facebook OAuth app, registered a new account locally.

Maniphest Tasks: T13615

Differential Revision: https://secure.phabricator.com/D21571
2021-02-19 10:27:42 -08:00
epriestley
7c44657ca5 Add more useful PHIDs to Harbormaster build variables
Summary: Ref T13609. Add the Object PHID (object being built), Container PHID (container of the object being built), Build PHID, and Buildable PHID to Harbormaster build variables.

Test Plan:
{F8448191}

{F8448192}

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13609

Differential Revision: https://secure.phabricator.com/D21569
2021-02-18 13:37:54 -08:00
epriestley
6703fec3e2 When documents are indexed, record the indexer version (versus the object version) and index epoch
Summary:
Ref T13587. D21495 has significant changes to the ngram indexer, which might possibly contain bugs.

Make it easier to reindex a subset of documents (based on the date when the index was built, and/or the software version which generated the index).

This is in addition to the existing versioning, which is focused on object versions.

Test Plan: Ran `bin/search index` with various old and new arguments. Spot-checked the `IndexVersion` table.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13587

Differential Revision: https://secure.phabricator.com/D21560
2021-02-16 16:09:31 -08:00
epriestley
4f647fb6be When updating a Ferret search index document, reuse existing rows where possible
Summary:
Ref T13587. Currently, when a document is reindexed by Ferret, the old document is completely discarded and a new version is inserted to replace it.

This approach is simple to implement, but can lead to exhaustion of the ngram AUTO_INCREMENT id column in reasonable circumstances.

Conceptually, this approach "should" be fine and this exhaustion is an awkard implementation detail. However, since it's easy to be less wasteful when performing document updates and all the other approaches are awkward or leaky in other ways that are probably worse, use a more complex implementation to avoid executing unnecessary INSERT statements.

Test Plan:
  - Created and indexed a new document, searched for it.
  - Updated a document, indexed it with `bin/search index ... --force --trace`, saw only modifications updated in the index.
  - Searched for newly added terms (got hits) and removed terms (no longer got hits) to verify add/delete index behavior.

Maniphest Tasks: T13587

Differential Revision: https://secure.phabricator.com/D21495
2021-02-16 16:09:31 -08:00
epriestley
5d6dddc5eb Add more constraints to "harbormaster.target.search"
Summary: Ref T13607. Add some time-oriented constraints to this API method to support compiling build statistics.

Test Plan:
  - Called "harbormaster.target.search" with all new constraints.
  - Viewed documentation in API console.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13607

Differential Revision: https://secure.phabricator.com/D21559
2021-02-16 12:22:31 -08:00
epriestley
9feb7343e6 Provide a "differential.changeset.search" Conduit API method
Summary: Ref T13605. Support selecting a diff's changesets (to get a list of affected file paths) via the API.

Test Plan: Called API with no arguments, diffPHIDs, PHIDs, IDs. Got sensible output.

Maniphest Tasks: T13605

Differential Revision: https://secure.phabricator.com/D21558
2021-02-15 11:11:13 -08:00
epriestley
ec5476a01f Add a PHID to Changesets
Summary:
Ref T13605. Changesets currently have no PHID, which limits their ability to use standard API infrastructure.

Give them a PHID, since there's no reason they don't have one other than their age.

Test Plan:
  - Ran migrations, saw PHIDs populated.
  - Created new changesets, saw PHIDs.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13605

Differential Revision: https://secure.phabricator.com/D21557
2021-02-15 11:11:12 -08:00
epriestley
42c26821ef When a revision has only human reviewers but none can view it, show a warning banner
Summary: Ref T13602. Warn when a reivison has at least one human reviewer, no non-human reviewers, and no human reviewers can view it.

Test Plan: {F8430683}

Maniphest Tasks: T13602

Differential Revision: https://secure.phabricator.com/D21556
2021-02-13 13:37:38 -08:00
epriestley
2f33dedc8b When a reviewer can't see a revision, show it clearly in the reviewer list
Summary: Ref T13602. Similar to subscriber and mention treatments, make it clear when a user doesn't have view permission.

Test Plan: {F8430595}

Maniphest Tasks: T13602

Differential Revision: https://secure.phabricator.com/D21555
2021-02-13 13:37:37 -08:00
epriestley
90903282c7 Render user hovercards with context information about their ability to see the context object
Summary:
Ref T13602. When rendering a user hovercard, pass the object on which the reference appears. If the user can't see the object, make it clear on the hovecard.

Restyle the "nopermission" markup in mentions to make it more obvious what the style means: instead of grey text, use red with an explicit icon.

Test Plan: {F8430398}

Maniphest Tasks: T13602

Differential Revision: https://secure.phabricator.com/D21554
2021-02-13 13:37:37 -08:00
epriestley
2aac3156f7 Restructure Hovercards to support more context information
Summary:
Ref T13602. Currently, Hovercards are functions only of the object they represent (and the viewer, etc).

Recent changes to how users who can't see an object are rendered motivate making them a function of both the object they represent //and// the context in which they are being viewed. In particular, this enables a hovecard for a user to explain "This user can't see the thing you're lookign at right now.", so visual "exiled" markers can have a path forward toward discovery.

Test Plan:
  - This change isn't expected to affect any behavior.
  - Viewed hovercards, moused over/out, resized windows, viewed standalone cards, viewed debug cards, saw no behavioral changes.

Maniphest Tasks: T13602

Differential Revision: https://secure.phabricator.com/D21553
2021-02-13 13:37:36 -08:00
epriestley
58bbd6ee88 Propagate the "ContextObject" to Remarkup rendering in timelines
Summary:
Ref T13602. Currently, timeline comment rendering does not (by default) propagate the context object to the rendering layer.

This means that `@mentions` of users who can't see the object aren't rendered properly (currently: they show up as blue, but should show up as grey).

Pass the context down the stack and into the remarkup engine.

Test Plan: {F8382905}

Maniphest Tasks: T13602

Differential Revision: https://secure.phabricator.com/D21548
2021-02-13 13:37:36 -08:00
epriestley
a4cb2bb772 When a subscriber can't see an object, clearly show that they're missing the permission in the curtain UI
Summary:
Ref T13602. When a subscriber can't see an object, it's currently hard to figure it out.

Show this status clearly in the curtain UI.

Test Plan: {F8382865}

Maniphest Tasks: T13602

Differential Revision: https://secure.phabricator.com/D21547
2021-02-13 13:37:36 -08:00
epriestley
f0dc065290 Lift bulk tests for "many users against one object" capabilities into "PolicyFilterSet"
Summary:
Ref T13602. Currently, the policy framework can not execute "test if many users can see one object" particluarly efficiently. This test must be executed more broadly to implement the changes in T13602.

To avoid making this any worse than it already is, lift this block into a wrapper class that has a bulk queue + fetch API and could eventually be optimized.

Test Plan: Viewed a task with an `@mention` of a user without permission to see it in the summary, saw it rendered in a disabled style.

Maniphest Tasks: T13602

Differential Revision: https://secure.phabricator.com/D21546
2021-02-13 13:37:35 -08:00
epriestley
00cf93548b Add an "--ignore-locality" flag to "bin/repository pull"
Summary:
Ref T13600. When migrating observed repositories between cluster services, impact can be better controlled by fetching a copy of the repository on the target host before clusterizing it.

In particular, in the Phacility cluster, migrations are generally from one shared shard to one dedicated shard. It's helpful to perform these migrations synchronously without waiting for the cluster to sync in the background (helpful in the sense that there are fewer steps and fewer commands to run).

This supports an "--observe" mode to the internal "bin/services load-repository" workflow, which transfers repository data by refetching it from the remote rather than by getting it from the older host. This fetch occurs before cluster configuration is adjusted.

Test Plan: Ran locally as a sanity check, will apply in production.

Maniphest Tasks: T13600

Differential Revision: https://secure.phabricator.com/D21544
2021-02-04 11:33:14 -08:00
epriestley
a7bd58c4bb Remove "AlmanacPropertyInterface" from "AlmanacNamespace"
Summary:
See <https://discourse.phabricator-community.org/t/i-cant-create-almanac-space/4424/>.

Almanac namespaces have never really had property support, but they implemented the interface in the original implementation.

At the time, this had no effect. Later changes integrated properties into the edit flows and broke this no-op integration.

Remove the interface for now. They could be given property support later, but need a bit of support code.

This feature is very rarely used and primarily useful for Phacility instances.

Test Plan: Created new namespaces and edited namespaces, browsed namespace UI.

Differential Revision: https://secure.phabricator.com/D21543
2021-02-03 15:20:26 -08:00
epriestley
9502312b60 Remove "final" from "private" methods in Phabricator
Summary:
Ref T13588. This has never been meaningful, but a "final private" method is specifically forbidden in PHP8.

Remove meaningless "final" from these methods, per new lint checks.

Test Plan: Ran `arc lint --everything` to identify affected methods, then `... | xargs -n1 arc lint --apply-patches`.

Maniphest Tasks: T13588

Differential Revision: https://secure.phabricator.com/D21540
2021-02-03 14:13:29 -08:00
epriestley
faf3f7b787 Internally, align commit processing tasks around PHIDs, not IDs
Summary: Ref T13591. This is a minor consistency change to use PHIDs instead of IDs in the commit import processing pipeline. PHIDs are generally more powerful in more contexts and it would be unusual for a modern worker to use an ID here.

Test Plan:
  - Made the "accept either ID or PHID" part of the change only.
  - Pushed a commit, parsed and reparsed it step by step (this tests that "commitID" tasks can still process normally).
  - Made the "write PHIDs" part of the change.
  - Pushed a commit, parsed and reparsed it step by step.
  - Looked at the task row in the database, saw PHID data.

Maniphest Tasks: T13591

Differential Revision: https://secure.phabricator.com/D21533
2021-02-02 13:40:10 -08:00
epriestley
de70a4ff26 Improve consistency in use of "via", "objectPHID", and "containerPHID" parameters in repository workers
Summary:
Ref T13591. Improve how parameters are passed between commit worker tasks:

  - Always pass "via", to track where tasks came from.
  - Always provide "objectPHID" (with the commit PHID).
  - Always provide "containerPHID" (with the repository PHID).

Test Plan:
  - Pushed a new commit.
  - Ran `bin/repository pull` + `bin/repository discover`, saw commit with all parameters.
  - Ran `bin/worker execute ...`, saw a Change worker and then a Publish worker with appropriate parameters.
  - Ran `bin/repository reparse ... --background`, saw workers queue with appropriate parameters.

Maniphest Tasks: T13591

Differential Revision: https://secure.phabricator.com/D21532
2021-02-02 13:40:09 -08:00
epriestley
6d5920fa2d Improve help for "bin/policy unlock" to make it explicit that you can name objects with a PHID
Summary: See <https://discourse.phabricator-community.org/t/how-to-unlock-a-portal-to-view-edit-it-as-an-admin-similar-to-tasks/4547/>.

Test Plan: Ran `bin/policy help unlock`, saw explicit "you can use a PHID" documentation.

Differential Revision: https://secure.phabricator.com/D21530
2021-02-01 09:13:16 -08:00
epriestley
9cbbbe2a87 Execute project membership materialization as "SELECT" + "INSERT", not "INSERT ... SELECT"
Summary:
Ref T13596. See that task for discussion. Executing "INSERT ... SELECT" at default isolation levels requires more locking than executing "SELECT" + "INSERT" separately.

Decompose this "INSERT ... SELECT" into "SELECT + INSERT", and reformat it to execute a minimal set of changes instead of wiping everything out and then writing all of it back. In most cases, this means we write 1 row instead of `O(number of project members)` rows.

Test Plan:
  - Created a project. Added and removed members, looked at database and saw a consistent membership/materialization list.
  - Created a subproject. Added and removed members, looked at database and saw a consistent membership/materialization list.

I wasn't successful in reproducing the LOCK WAIT issue locally by trying various concurrent SELECT / INSERT / INSERT ... SELECT strategies. It may depend on the "DELETE + INSERT ... SELECT" structure used here, or versions/config/etc, so we'll have to see how that fares in production.

Maniphest Tasks: T13596

Differential Revision: https://secure.phabricator.com/D21527
2021-01-28 09:04:44 -08:00
epriestley
d6fd365704 Correct Diffusion browse behavior when visiting a path URI with no trailing slash
Summary:
See PHI1983. Ref T13599. Ref T13589. Currently, if you browse to a path browse URI in Diffusion without a trailing slash (`/browse/master/src`), you get a nonsensical view (the directory as a single item).

Be more precise in how "git ls-tree" arguments are constructed.

Test Plan: Visited files and directories in the browse view, with and without trailing slashes. Saw improved behavior for directories with no trailing slash and reasonable behavior in all other cases.

Maniphest Tasks: T13599, T13589

Differential Revision: https://secure.phabricator.com/D21528
2021-01-28 08:52:58 -08:00
epriestley
b4f2cef76c Prevent interruption by the PHP "set_time_limit()" mechanism while holding the durable write lock
Summary:
Ref T13590. By default, PHP kills execution after web scripts run for 30 seconds. If this occurs in the locked section of a repository write while we're holding the durable write lock, the lock will get stuck.

Use "set_time_limit(0)" to prevent this mechanism from interrupting execution while the durable lock is held.

Test Plan:
  - Added "set_time_limit(1)" before the lock and "while (1);" in the critical section of the lock.
  - Pushed, got the lock stuck.
  - Cleared the lock, applied this patch, pushed.
  - Got an infinite hang instead. (Normally, we expect the script to take more than 30 seconds to execute because there is a large push that executes in finite time, not because there's an infinte loop.)

Maniphest Tasks: T13590

Differential Revision: https://secure.phabricator.com/D21526
2021-01-26 16:14:05 -08:00
epriestley
da7d92dd0a Catch more HTTP VCS errors and convert them into VCS repsonses
Summary:
Ref T13590. Currently, errors arising from cluster locking (like the "stuck write lock" exception) are not caught and converted into VCS responses on the HTTP VCS workflow.

Catch a broader range of exceptions and convert them into appropriate responses.

Test Plan:
  - Forced a "stuck write lock" exception, pushed to a Git repository over HTTP.
  - Before: generic fatal.
  - After: VCS-specific fatal with a useful message in the "X-Phabricator-Message" response header.

Maniphest Tasks: T13590

Differential Revision: https://secure.phabricator.com/D21525
2021-01-26 16:14:04 -08:00
epriestley
32c82a53de After loading the effective Viewer during a VCS request, flag them for inline cache generation
Summary:
Ref T13590. User objects have some inline caches that don't do readthrough generation by default because it may be indicative of high-impact performance problems in code.

During a VCS request, these caches are normally unnecessary, but they may be hit on some unusual pathways (like error handling).

Flag VCS users as okay for inline generation. This does not indicate a performance problem and access to these caches is very rare, at least today.

Test Plan:
  - Executed a Git HTTP request which hit an unhandled exception (stuck write lock).
    - Before: got a second-level exception while handling the first exception, when trying to access user preferences to render a standard uncaught exception page.
    - After: no second-level exception.

Maniphest Tasks: T13590

Differential Revision: https://secure.phabricator.com/D21524
2021-01-26 16:14:04 -08:00
epriestley
3a74701555 Return Git HTTP error messages in an HTTP header
Summary:
Ref T13590. Currently, when you encounter a HTTP error in Git, there is no apparent way to make the client show any additional useful information. In particular, the response body is ignored.

We can partially get around this by putting the information in an "X-Phabricator-Message: ..." HTTP header, which is visible with "GIT_CURL_VERBOSE=1 git ...". Users won't normally know to look here, but it's still better than nothing.

Test Plan:
  - Ran "GIT_CURL_VERBOSE=1 git fetch" against a Phabricator HTTP URI that returned a HTTP/500 error.
    - Before: no clue what happened on the client.
    - After: client shows useful message in the "X-Phabricator-Message" header in debug output.

Maniphest Tasks: T13590

Differential Revision: https://secure.phabricator.com/D21523
2021-01-26 16:14:03 -08:00
epriestley
acd767c7f3 Allow "differential.createinline" to accept JSON "false" for "isNewFile"
Summary:
See <https://discourse.phabricator-community.org/t/error-creating-inline-comment-via-conduit-api/4535>. See T12678.

This API method currently does not accept a JSON "false", but reasonably should.

Test Plan:
  - Called method with "isNewFile: false".
    - Before: type error.
    - After: inline comment.

Differential Revision: https://secure.phabricator.com/D21522
2021-01-26 14:56:37 -08:00
epriestley
ed86c42b26 Improve performance of repository discovery in repositories with >65K refs
Summary:
Ref T13593. The commit cache in this Engine has a maximum fixed size (currently 65,535 entries).

If we execute discovery in a repository with more refs than this (e.g., 180K), we get fast lookups for the first 65,535 refs and slow lookups for the remaining refs.

Instead, divide the refs into chunks no larger than the cache size, and perform an explicit cache fill before each chunk is processed.

Test Plan:
  - Created a repository with 1K refs. Set cache size to 256. Ran discovery.
    - Before patch: saw one large cache fill and then ~750 single-gets.
    - After patch: saw four large cache fills.
  - Compared `bin/repository discover ... --verbose` output before and after patch for overall effect; saw no differences.

Maniphest Tasks: T13593

Differential Revision: https://secure.phabricator.com/D21521
2021-01-26 12:27:02 -08:00