1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2025-02-07 20:38:32 +01:00
Commit graph

483 commits

Author SHA1 Message Date
epriestley
7fd8263185 Remove product literal strings in "pht()", part 22
Summary: Ref T13658.

Test Plan: Static checks only.

Maniphest Tasks: T13658

Differential Revision: https://secure.phabricator.com/D21788
2022-04-25 16:46:26 -07:00
epriestley
adcc063242 Fix a PHP 8.1 string strictness issue in repositories
Summary: Ref T13588. This is a little too strict, as `$line` may be an integer.

Test Plan: Will deploy

Maniphest Tasks: T13588

Differential Revision: https://secure.phabricator.com/D21775
2022-04-25 12:30:45 -07:00
epriestley
4d3f0dc7c7 Remove product literal strings in "pht()", part 8
Summary: Ref T13658.

Test Plan:
This test plan is non-exhaustive.

  - Viewed "remarkup.process" Conduit method API page.
  - Viewed URIs in a Diffusion repository.
  - Viewed editor protocol configuration in Settings.

Maniphest Tasks: T13658

Differential Revision: https://secure.phabricator.com/D21772
2022-04-25 12:22:29 -07:00
epriestley
de980cc54e Fix an issue where Herald may fail to extract content from an empty commit
Summary:
Ref T13667. The Herald "content added" rule (and other similar rules) do not correctly extract content from empty commits.

When we load an empty raw diff, return an empty changed content map.

Ref T13588. Also fix some PHP8.1 null/string stuff

Test Plan:
  - Ran "bin/repository reparse --publish <commit>", with an empty commit hash and a nonempty commit hash.
  - Reviewed Herald transcripts for general sanity.

Maniphest Tasks: T13667, T13588

Differential Revision: https://secure.phabricator.com/D21761
2022-04-20 13:04:00 -07:00
epriestley
b757e5c302 Use "MethodCallFuture" to move Diffusion Conduit exceptions to resolution time
Summary:
Depends on D21720. Ref T13666. See D21720 for additional discussion.

Use "MethodCallFuture", introduced in D21720, so that exceptions raised in "execute()" are thrown when the future is resolved, not when the future is created.

This makes exception behavior for clustered and non-clustered setups consistent, and chooses the intended (clustered) behavior in both cases, which currently deals with errors better.

Test Plan:
  - Applied both parts of the patch in T13666 (break history queries, force immediate futures) to reproduce the issue in T13666.
    - Loaded a Diffusion landing page, reproduced the error described in that task.
  - Applied this patch, verified landing page works again.
  - Removed the "break history queries" change, verified landing page works in forced-immediate mode.
  - Removed the "force immediate" change, verified landing page works in "actual future" mode.

Reviewers: cspeckmim

Reviewed By: cspeckmim

Maniphest Tasks: T13666

Differential Revision: https://secure.phabricator.com/D21721
2021-09-04 14:36:41 -07:00
epriestley
af6cc0c934 Use "resolve()", not "execute()", for PhutilExecPassthru callsites in Phabricator
Summary: Ref T13660. Clean up callsites to "PhutilExecPassthru->execute()" to prepare to deprecate it.

Test Plan:
  - Grepped for "PhutilExecPassthru" and looked for callsites.
  - Ran `GIT_SSH=.../ssh-connect git ls-remote origin` to execute the "ssh-connect" code.
  - The two passthru future methods have no callers and could possibly be removed, but I'm just letting sleeping dogs lie for now.

Reviewers: cspeckmim

Reviewed By: cspeckmim

Maniphest Tasks: T13660

Differential Revision: https://secure.phabricator.com/D21703
2021-07-21 10:21:06 -07:00
Christopher Speck
e5de7f0843 Update templates used with mercurial to remove '--debug'
Summary:
Refer to discussion on D21677#275541

Refs D21681 (arcanist changes)

Phabricator has several uses of the `--debug` flag being used with Mercurial. Use of this flag causes additional output to be added which Phabricator needs, however the behavior of `--debug` is not guaranteed to be stable, and in newer versions of Mercurial there have been additional output that has caused Phabricator to choke on parsing the output. This change removes several uses of `--debug` in favor of using `--template` with the `hg log` or `hg annotate` commands in combination with the `{p1.node}` or `{p2.node}` template format.

The use of `{p1node}` format in templates was added in [[ https://www.mercurial-scm.org/wiki/WhatsNew/Archive#Mercurial_2.4_.282012-11-01.29 | Mercurial 2.4 (2012) ]]. This format was deprecated in [[ https://www.mercurial-scm.org/wiki/WhatsNew#Mercurial_4.9_.282019-02-01.29 | Mercurial 4.9 (2019) ]] in favor of using `{p1.node}` format which is unclear when this new format was added (presumably earlier than Mercurial 4.9).

The use of `--template` with `hg annotate` is only officially supported in [[ https://www.mercurial-scm.org/wiki/Release4.6 | Mercurial 4.6 (2018) ]], though does appear to work in 4.5 but is not documented.

Since the `{p1node}` format was introduced in 2.4 this bumps the required version of `hg` to 2.4 (from 1.9). Since the `annotate --template` feature wasn't added until 4.6 (which is still fairly recent), the use of it is gated on a capability test, but still preferred for use where possible to avoid extraneous output from `--debug` flag.

Test Plan:
I verified I could do the following in a mercurial repository, while having mercurial 5.8 installed:
1. Navigate and view files in Diffusion under e.g. `/source/test-repo/`.
2. While viewing a file in Diffusion verified that I could view the blame of the file and the history/annotations looked accurate for the files I was browsing.
3. From the blame sidebar, select to view a commit which loaded and displayed changes properly.
4. View the history of the repository under e.g. `/source/test-repo/history/default/`. I verified the history looked correct and the tree-like structure showing relationship of commits also looked accurate.

I setup mercurial to run version 4.4, created a new repository, added some commits, and verified all the above behavior still works properly.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D21679
2021-07-09 12:35:20 -04:00
Christopher Speck
9637420861 Update a few random typos
Summary: Found a few typos which could be updated.

Test Plan:
I tested the Configuration page change by navigating to `/config` and verifying the page title set in the browser as well as the page title text on the page
|Before|After|
|---|---|
|{F9013208}|{F9013210}|
|{F9013300}|{F9013301}|

I verified the Conduit error message by navigating to `/auth/start/?__conduit__=1`
{F9013289}

The CircleCI error message was not verified due to the involvement of testing with CircleCI however the change is very minor and has very little risk of impacting any functionality.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D21675
2021-06-27 17:38:34 -04:00
epriestley
12a5eb4062 Allow maintenance scripts to write synthetic events to the push log that act as repository updates
Summary:
Ref T13614. When a script holds the write lock but modifies the repository directly (rather than by pushing), the repository version won't change when the script releases the write lock. Thus, the writes may not propagate to other nodes (it depends which node lucks out and accepts the next write).

To guarantee that writes propagate, allow these scripts to pretend they pushed the repository. These are bare-bones valid events flagged as "Maintenance".

Test Plan:
  - Wrote a script to hold the write lock, wait (or pretend to do something), then release the write lock.
  - Applied patches, modified script to use new APIs ("newMaintenanceEvent()").
  - Ran script, saw repository verison bump and relevant push logs:

{F8814923}

Maniphest Tasks: T13614

Differential Revision: https://secure.phabricator.com/D21670
2021-06-01 08:29:54 -07:00
epriestley
15c0c895a5 Make upstream callers respect "active bindings" when querying Almanac
Summary: Ref T13641. Make "active bindings" a real query and make callers that only care about active bindings only query for active bindings.

Test Plan:
  - Queried for "bindings" and "activeBindings" via Conduit.
  - Disabled/enabled devices, saw binding status update in UI.
  - Loaded Diffusion cluster layout.
  - Grepped for `needBindings()`, `getActiveBindings()`, etc.

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13641

Differential Revision: https://secure.phabricator.com/D21628
2021-03-16 15:51:47 -07:00
epriestley
42c0c0e3d2 Remove or correct various "phabricator/" references to "libphutil"
Summary:
Ref T13395. "libphutil/" was stripped for parts, but some documentation still references it. This is mostly minor corrections, but:

  - Removes "Javelin at Facebook", long obsolete.
  - Removes "php FPM warmup", which was always a prototype and is obsoleted by PHP preloading in recent PHP.

Test Plan: `grep` / reading

Maniphest Tasks: T13395

Differential Revision: https://secure.phabricator.com/D21624
2021-03-16 10:28:07 -07:00
epriestley
ac2f5a1046 Modernize and clean up "PhabricatorAuditStatusConstants"
Summary:
Ref T13631. Move "PhabricatorAuditStatusConstants" to a more modern object ("PhabricatorAuditRequestStatus").

Expose the status value via Conduit.

Test Plan:
  - Ran `bin/audit delete`.
  - Viewed a commit with auditors in the web UI.
  - Grepped for affected symbols.
  - Called Conduit with the "auditors" attachment, saw auditor statuses.

Maniphest Tasks: T13631

Differential Revision: https://secure.phabricator.com/D21599
2021-03-10 09:21:55 -08:00
epriestley
2636d84d0c Remove very old Audit status constants and AuditRequest data
Summary:
Ref T13631. See that task for discussion.

  - "NONE": Probably never used?
  - "CC": Obsoleted by subscribers.
  - "AUDIT_NOT_REQUIRED": For Owners packages, obsoleted by edges.
  - "CLOSED": For "Close Audit", obsoleted by "Request Verification".

Test Plan:
  - Grepped for constants, browsed Diffusion.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13631

Differential Revision: https://secure.phabricator.com/D21598
2021-03-10 09:21:54 -08:00
epriestley
55532b3f74 Add a very basic "auditors" attachment to "differential.commit.search"
Summary: Ref T13631. For now, this only shows the auditor PHID. The current status constants could use some cleanup before they're exposed.

Test Plan: Queried with "auditors" attachment, saw basic auditor information.

Maniphest Tasks: T13631

Differential Revision: https://secure.phabricator.com/D21597
2021-03-10 09:21:54 -08:00
epriestley
f970b350ea Correct behavior of "writable" Almanac service binding for repository services
Summary: Ref T13611. This property worked correctly when implemented in D19357. The behavior was broken by D20775, which tested node-level routing but did not specifically re-test the "writable" property. This was difficult to spot because ref query outcomes weren't observable in the UI, and the ref itself had the correct property value.

Test Plan:
See D21575. After this change, the UI shows the correct state, rather than showing a read-only service ref as writable:

{F8465865}

Maniphest Tasks: T13611

Differential Revision: https://secure.phabricator.com/D21576
2021-02-25 12:29:17 -08:00
epriestley
e9804bb7e5 Provide hovercards for generic edge stories, and include more message information in commit hovercards
Summary:
Ref T13620.

  - Make generic edge stories render links with hovercards. Other story types (like subscriptions) already do this so I'm fairly certain this is just old code from before hovercards.
  - Include a longer commit message snippet in hovercards.

Test Plan: {F8465645}

Maniphest Tasks: T13620

Differential Revision: https://secure.phabricator.com/D21574
2021-02-25 10:29:58 -08:00
epriestley
6716d4f6ae Separate "shouldPublishRef()" from "isPermanentRef()" and set "IMPORTED_PERMANENT" more narrowly
Summary:
Ref T13591. Currently, the "IMPORTED_PERMANENT" flag (previously "IMPORTED_CLOSEABLE", until D21514) flag is set by using the result of "shouldPublishRef()".

This method returns the wrong value for the flag when there is a repository-level reason not to publish the ref (most commonly, because the repository is currently importing).

Although it's correct that commits should not be published in an importing repository, that's already handled in the "PublishWorker" by testing "shouldPublishCommit()". The "IMPORTED_PERMANENT" flag should only reflect whether a commit is reachable from a permanent ref or not.

  - Move the relevant logic to a new method in Publisher.
  - Fill "IMPORTED_PERMANENT" narrowly from "isPermanentRef()", rather than broadly from "shouldPublishRef()".
  - Deduplicate some logic in "PhabricatorRepositoryRefEngine" which has the same intent as the logic in the Publisher.

Test Plan:
  - Ran discovery on a new repository, saw permanent commits marked as permanent from the beginning.
  - See later changes in this patch series for additional testing.

Maniphest Tasks: T13591

Differential Revision: https://secure.phabricator.com/D21515
2021-01-22 19:51:38 -08:00
epriestley
2d0e7c37e1 Rename "IMPORTED_CLOSEABLE" to "IMPORTED_PERMANENT" to clarify the meaning of the flag
Summary:
Ref T13591. This is an old flag with an old name, and there's an import bug because the outdated concept of "closable" is confusing two different behaviors.

This flag should mean only "is this commit reachable from a permanent ref?". Rename it to "IMPORTED_PERMANENT" to make that more clear.

Rename the "Unpublished" query to "Permanent" to make that more clear, as well.

Test Plan:
  - Grepped for all affected symbols.
  - Queried for all commmits, permament commits, and impermanent commits.
  - Ran repository discovery.
  - See also further changes in this change series for more extensive tests.

Maniphest Tasks: T13591

Differential Revision: https://secure.phabricator.com/D21514
2021-01-22 19:51:38 -08:00
epriestley
cebde34425 Make "CommitData" wrap and persist a "CommitRef" record
Summary:
Ref T13552. Turn "CommitData" into an application-level layer on top of the repository-level "CommitRef" object.

For older commits which will not have a "CommitRef" record on disk, build a synthetic one at runtime. This could eventually be migrated.

Test Plan: Ran "bin/repository reparse --message", browsed Diffusion.

Maniphest Tasks: T13552

Differential Revision: https://secure.phabricator.com/D21449
2020-09-15 17:36:40 -07:00
epriestley
e454c3dafe Wrap all direct access to author/committer properties on "CommitData"
Summary: Ref T13552. Currently, various callers read raw properties off "CommitData" directly. Wrap these in accessors to support storage changes which persist "CommitRef" information instead.

Test Plan:
- Ran "diffusion.querycommits", saw the same data before and after.
- Looked at a commit, saw authorship information and date.
- Viewed tags in a repository, saw author information.
- Ran "rebuild-identities", saw no net effect.
- Grepped for callers to "getCommitDetail(...)".

Maniphest Tasks: T13552

Differential Revision: https://secure.phabricator.com/D21448
2020-09-15 17:36:39 -07:00
epriestley
3a80efa440 Build "DiffusionCommitRef" objects from "internal.commit.search", not "diffusion.querycommits", in the message parser worker
Summary: Ref T13552. Swap the call we're using to build "CommitRef" objects here to the recently-introduced "internal.commit.search" method.

Test Plan: Used "bin/repository reparse --message ..." to reparse commits, added "var_dump()" to inspect results. Saw sensible CommitRef and CommitData objects get built.

Maniphest Tasks: T13552

Differential Revision: https://secure.phabricator.com/D21446
2020-09-15 17:36:39 -07:00
epriestley
a9506097ea Add "internal.commit.search" to replace the cache bypass mode of "diffusion.querycommits"
Summary:
Ref T13552. Commit parsers currently invoke a special mode of "diffusion.querycommits", which is an older frozen method.

The replacement, "diffusion.commit.search", is not really appropriate for low-level access. This mode of having a single method which operates in "cache" or "non-cache" modes also ends up in a lot of unnecessary field shuffling.

Provide "internal.commit.search" as a modern equivalent that returns a "DiffusionCommitRef"-compatible structure.

Test Plan: Executed "internal.commit.search", got sensible low-level commit results.

Maniphest Tasks: T13552

Differential Revision: https://secure.phabricator.com/D21443
2020-09-15 17:36:38 -07:00
epriestley
a745055813 Lift Diffusion Conduit call proxying to the root level of Conduit
Summary:
Ref T13552. Some Diffusion conduit calls may only be served by a node which hosts a working copy on disk, so they're proxied if received by a different node.

This capability is currently bound tightly to "DiffusionRequest", which is a bundle of context parameters used by some Diffusion calls. However, call proxying is not fundamentally a Diffusion behavior.

I want to perform proxying on a "*.search" call which does not use the "DiffusionRequest" parameter bundle. Lift proxying to the root level of Conduit.

Test Plan: Browsed diffusion in a clusterized repsository.

Maniphest Tasks: T13552

Differential Revision: https://secure.phabricator.com/D21442
2020-09-15 17:36:37 -07:00
epriestley
c6de7c66a3 Remove the "Graph" view as a dedicated repository view
Summary:
Ref T13552. Currently, Diffusion has two effectively identical history views, the "Graph" view and the "History" view.

These arose out of product uncertainty about the importance of the graph, but I think we can just put the graph on the "object item list" view and merge these views.

Test Plan: Looked at repositories in Diffusion, no longer saw a "Graph" tab. Grepped for "graph"-related symbols.

Maniphest Tasks: T13552

Differential Revision: https://secure.phabricator.com/D21409
2020-08-12 08:59:16 -07:00
epriestley
7fd6bf26a9 Modernize "Author" and "Committer" rendering for commits
Summary:
Ref T13552. Give "Commit" objects a more modern, identity-aware way to render author and committer information.

This uses handles in a more modern way and gives us a single read callsite for raw author and committer names.

Test Plan:
  - Grepped for callers to the old methods, found none. (There are a lot of "renderAuthor()" callers in transactions, but this call takes no parameters.)
  - Viewed some commits, saw sensible lists of authors and committers.

Maniphest Tasks: T13552

Differential Revision: https://secure.phabricator.com/D21405
2020-08-12 08:58:43 -07:00
epriestley
7d496f2c6d Collapse repository URI normalization code into Arcanist
Summary: Ref T13546. Companion change to D21372. Move URI normalization code to Arcanist to we can more-often resolve remote URIs correctly.

Test Plan: Grepped for affected symbols.

Maniphest Tasks: T13546

Differential Revision: https://secure.phabricator.com/D21373
2020-06-30 15:54:33 -07:00
epriestley
a2b2c391a1 Distinguish between "Assigned" and "Effective" identity PHIDs more clearly and consistently
Summary:
Ref T13444. You can currently explicitly unassign an identity (useful if the matching algorithm is misfiring). However, this populates the "currentEffectiveUserPHID" with the "unassigned()" token, which mostly makes things more difficult.

When an identity is explicitly unassigned, convert that into an explicit `null` in the effective user PHID.

Then, realign "assigned" / "effective" language a bit. Previously, `withAssigneePHIDs(...)` actualy queried effective users, which was misleading. Finally, bulk up the list view a little bit to make testing slightly easier.

Test Plan:
  - Unassigned an identity, ran migration, saw `currentEffectiveUserPHID` become `NULL` for the identity.
  - Unassigned a fresh identity, saw NULL.
  - Queried for various identities under the modified constraints.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13444

Differential Revision: https://secure.phabricator.com/D20908
2019-11-19 09:37:44 -08:00
epriestley
df0f5c6cee Make repository identity email address association case-insensitive
Summary:
Ref T13444. Currently, identities for a particular email address are queried with "LIKE" against a binary column, which makes the query case-sensitive.

  - Extract the email address into a separate "sort255" column.
  - Add a key for it.
  - Make the query a standard "IN (%Ls)" query.
  - Deal with weird cases where an email address is 10000 bytes long or full of binary junk.

Test Plan:
  - Ran migration, inspected database for general sanity.
  - Ran query script in T13444, saw it return the same hits for "git@" and "GIT@".

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13444

Differential Revision: https://secure.phabricator.com/D20907
2019-11-19 09:37:26 -08:00
epriestley
5b36f0c97a Add default branch, description, and metrics (commit count, recent commit) to "diffusion.repository.search"
Summary: Fixes T13430. Provide more information about repositories in "diffusion.repository.search".

Test Plan: Used API console to call method (with new "metrics" attachment), reviewed output. Saw new fields returned.

Maniphest Tasks: T13430

Differential Revision: https://secure.phabricator.com/D20862
2019-10-24 17:16:58 -07:00
epriestley
06edcf2709 Fix an issue where ancestors of permanent refs might not be published during import or if a branch is later made permanent
Summary:
Fixes T13284. See that task for substantial discussion. There are currently two cases where we'll skip over commits which we should publish:

  - if a branch is not permanent, then later made permanent; or
  - in some cases, the first time we examine branches in a repository.

In both cases, this error is one-shot and things work correctly going forward. The root cause is conflation between the states "this ref currently permanent" and "this ref was permanent the last time we updated refs".

Separate these pieces of state and cover all these cases. Also introduce a "--rebuild" flag to fix the state of bad commits.

Test Plan:
See T13284 for the three major cases:

  - initial import;
  - push changes to a nonpermanent branch, update, then make it permanent;
  - push chanegs to a nonpermanent branch, update, push more changes, then make it permanent.

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13284

Differential Revision: https://secure.phabricator.com/D20829
2019-09-25 08:54:50 -07:00
epriestley
b6420e0f0a Allow repository service lookups to return an ordered list of service refs
Summary:
Ref T13286. To support request retries, allow the service lookup method to return an ordered list of structured service references.

Existing callsites continue to immediately discard all but the first reference and pull a URI out of it.

Test Plan: Ran `git pull` in a clustered repository with an "up" node and a "down" node, saw 50% serivce failures and 50% clean pulls.

Maniphest Tasks: T13286

Differential Revision: https://secure.phabricator.com/D20775
2019-09-03 10:05:40 -07:00
epriestley
3c26e38487 Provide a simple read-only maintenance mode for repositories
Summary:
Ref T13393. While doing a shard migration in the Phacility cluster, we'd like to stop writes to the migrating repository. It's safe to continue serving reads.

Add a simple maintenance mode for making repositories completely read-only during maintenance.

Test Plan: Put a repository into read-only mode, tried to write via HTTP + SSH. Viewed web UI. Took it back out of maintenance mode.

Maniphest Tasks: T13393

Differential Revision: https://secure.phabricator.com/D20748
2019-08-29 15:23:10 -07:00
epriestley
82cf97ad65 When many commits are discovered at once, import them at lower priority
Summary:
Ref T13369. See that task for discussion.

When the discovery daemon finds more than 64 commits to import, demote the worker queue priority of the resulting tasks.

Test Plan:
  - Pushed one commit, ran `bin/repository discover --verbose --trace ...`, saw commit import with "at normal priority" message and priority 2500 ("PRIORITY_COMMIT").
  - Pushed 3 commits, set threshold to 3, ran `bin/repository discover ...`, saw commist import with "at lower priority" message and priority 4000 ("PRIORITY_IMPORT").

Maniphest Tasks: T13369

Differential Revision: https://secure.phabricator.com/D20712
2019-08-12 12:59:00 -07:00
epriestley
4450c90881 When triggering audits, respect committer identities when importing commits
Summary:
Ref T13311. We currently don't use committer identity mappings when triggering audits, so if a user is only associated with an identity via manual mapping we won't treat them as the author.

Instead, use the identity and manual mapping if they're available.

Test Plan:
  - Pushed a commit as `xyz <xyz@example.org>`, an address with no corresponding user.
  - In the UI, manually associated that identity with user `@alice`.
  - Ran `bin/repository reparse --publish <hash>` to trigger audits and publishing for the commit.
  - Before: observed the `$author_phid` was `null`.
  - After: observed the `$author_phid` is Alice.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13311

Differential Revision: https://secure.phabricator.com/D20580
2019-06-17 13:47:31 -07:00
epriestley
56e7bde68d Recognize self-URI links to Diffusion files and give them special rendering behavior
Summary:
Depends on D20530. Ref T13291. When users paste links to files in Diffusion into remarkup contexts, identify them and specialize the rendering.

When the URIs are embedded with `{...}`, parse them in more detail.

This is a lead-up to a `{src ...}` rule which will use the same `View` but give users more options to customize presentation.

Test Plan: {F6463580}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13291

Differential Revision: https://secure.phabricator.com/D20538
2019-05-21 13:07:37 -07:00
epriestley
c0a4d1de13 Merge the "Herald" and "Owners" daemon workers into a single "Publish" worker
Summary:
Depends on D20466. Ref T13277. Currently:

  - The "Owners" worker writes ownership relationships (e.g., commit X affects package Y, because it touches a path in package Y) -- these are just edges.
  - It also triggers audits.
  - Then it queues a "Herald" worker.
  - This formally publishes the commit and triggers Herald.

These aren't really separate steps and can happen more easily in one shot. Merge them.

Test Plan:
  - Ran `bin/repository reparse --publish` to republish various commits, got sensible behavior.
  - Grepped for "IMPORTED_OWNERS", "IMPORTED_HERALD", "--herald", "--owners", and "--force-local" flags.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13277

Differential Revision: https://secure.phabricator.com/D20467
2019-04-25 09:45:59 -07:00
epriestley
0fab41ff3c Show "hold reasons" on commit page, not on "Edit" page
Summary:
Depends on D20465. Ref T13277. Currently, when a commit is unpublished, we put a single line about it on the "Edit Commit" page. This is pretty much impossible to find.

Move it to the main page. This treatment is more big/bold than I'd probably like to end up, but we should probably overshoot on the explanatory text until users get used to this behavior.

Also, allow searching for only published / unpublished commits.

Test Plan: {F6395705}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13277

Differential Revision: https://secure.phabricator.com/D20466
2019-04-25 09:22:49 -07:00
epriestley
8f43c773b8 Remove nearly all remaining references to "Autoclose"
Summary:
Depends on D20464. Ref T13277. Broadly:

  - Move all the "should publish X" and "why aren't we publishing X" stuff to a separate class (`PhabricatorRepositoryPublisher`).
  - Rename things to be more consistent with modern terminology ("Publish", "Permanent Refs").

Test Plan:
This could use some trial-by-fire on `secure`, but:

  - Grepped for all symbols.
  - Viewed various commits.
  - Reparsed commits.
  - Here's a commit with an explanation:

{F6394569}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13277

Differential Revision: https://secure.phabricator.com/D20465
2019-04-24 08:29:41 -07:00
epriestley
45b9859f02 Remove "--force-autoclose" from "bin/repository reparse"
Summary: Depends on D20463. Ref T13277. This flag was added some time before 2015 and I don't think I've ever used it. Just get rid of it.

Test Plan: Grepped for `force-autoclose`, `forceAutoclose`, `AUTOCLOSE_FORCED`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13277

Differential Revision: https://secure.phabricator.com/D20464
2019-04-24 08:24:06 -07:00
epriestley
6449a0ecb2 Rename some internal "Autoclose" mentions to "Permanent Refs"
Summary: Depends on D20428. Ref T13277.

Test Plan: Grep / reading.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13277

Differential Revision: https://secure.phabricator.com/D20432
2019-04-18 05:38:09 -07:00
epriestley
aed755e1d8 Expose repository ref rules via "diffusion.repository.search"
Summary:
Depends on D20425. Ref T13277. See PHI1067. There's currently no way to retrieve branch/ref rules over the API, which makes some management operations against a large number of repositories difficult.

Expose these rules to the API.

Test Plan: Called `diffusion.repository.search`, got rules in the result set.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13277

Differential Revision: https://secure.phabricator.com/D20426
2019-04-18 05:27:37 -07:00
epriestley
ec9237fe13 In repository settings, fold "Autoclose On/Off" into "Publishing On/Off"
Summary:
Depends on D20423. Ref T13277. Repositories currently have separate toggles for "Autoclose" and "Publishing".

Merge the "Autoclose" toggle into the "Publishing" toggle. I'm unaware of any valid use case for enabling one but not the other.

(This doesn't fix all the documentation, yet.)

Test Plan: Edited a repository, saw only one publishing option.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13277

Differential Revision: https://secure.phabricator.com/D20424
2019-04-18 05:16:59 -07:00
epriestley
e910c76e65 Add "Fetch Rules" to observed Git repositories
Summary:
Depends on D20421. Ref T13277. I'd generally like to move away from "Track Only".

Some of the use cases for "Track Only" (or adjacent to "Track Only") are better resolved with "Fetch Rules" -- basically, rules to fetch only some subset of refs from the observed remote.

Add configurable "Fetch Rules" for Git repositories. For example, if you only want to fetch `master`, you can now speify:

```
refs/heads/master
```

If you only want to fetch branches and tags, you can use:

```
refs/heads/*
refs/tags/*
```

In theory, this is slightly less powerful in the general case than "Track Only", but gives us better behavior in some cases (e.g., when the remote has 50K random temporary branches). In practice, I think this and a better "Autoclose Only" will let us move away from "Track Only", get default behavior which is better aligned with what users actually expect, and dodge all the "track tags/refs" questions.

Test Plan: Configured repositories with "Fetch Refs" rules, used `bin/repository pull --verbose --trace ...` to run pulls, saw expected pull/fetch behavior.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13277

Differential Revision: https://secure.phabricator.com/D20422
2019-04-18 05:09:36 -07:00
epriestley
1cda1402c7 Do not publish/notify about commits which are not reachable from any "Autoclose" ref
Summary:
Depends on D20418. Ref T13277. Fixes T11314.

Currently, when you push commits to some arbitrary ref or tag (like `refs/pull/123` on GitHub, `refs/tags/phabricator/diff/123` on Phabricator, or `refs/changes/whatever` on Gerrit), we do not "autoclose" related objects. This means that we don't process `Ref T123` to create links to tasks, and don't process `Differential Revision: xyz` to close revisions.

However, we //do// still publish these commits. "Publish" means: trigger audits, publish feed stories, and run Herald rules.

  - Stop publishing these commits.
  - In the UI, show these commits as "Not Permanent" with a note that they are "Not [on any permanent branch]."

These commits will publish and autoclose if they ever become reachable from an "autoclose" ref (most commonly, if they are later merged to "master").

Test Plan:
  - Pushed a commit to `refs/tags/quack`.
  - Before: got a feed story.
  - After: no feed story, UI shows commit as "Not Permanent".

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13277, T11314

Differential Revision: https://secure.phabricator.com/D20419
2019-04-18 05:05:00 -07:00
epriestley
23b86bae6c Accept pushes with arbitrary Git refs
Summary:
Depends on D20419. Ref T13277. Fixes T8936. Fixes T9383. Fixes T12300. When you push arbitrary refs to Phabricator, the push log currently complains if those refs are not tags or branches.

Upstream Git now features "notes", and there's no reason to prevent writes to arbitrary refs, particularly beause we plan to start using them soon (see T13278).

Allow these writes as affecting raw refs.

Test Plan:
  - Pushed an arbitrary ref.
  - Pushed some Git notes.
  - Wrote a Herald ref rule, saw "ref" in the dropdown.

{F6376492}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13277, T8936, T9383, T12300

Differential Revision: https://secure.phabricator.com/D20420
2019-04-17 12:43:20 -07:00
epriestley
c116deef63 Remove "Effective User" attachment from Repository Identities
Summary:
See <https://discourse.phabricator-community.org/t/phabricatorrepositoryidentity-attacheffectiveuser-must-be-an-instance-of-phabricatoruser-null-given/1820/3>.

It's possible for an Idenitity to be bound to user X, then for that user to be deleted, e.g. with `bin/remove destroy X`. In this case, we'll fail to load/attach the user to the identity.

Currently, we don't actually use this anywhere, so just stop loading it. Once we start using it (if we ever do), we could figure out what an appropriate policy is in this case.

Test Plan: Browsed around Diffusion, grepped for affected "effective user" methods and found no callsites.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20224
2019-03-05 11:35:12 -08:00
epriestley
4c12420162 Replace "URI->setQueryParams()" after initialization with a constructor argument
Summary: Ref T13250. See D20149. In a number of cases, we use `setQueryParams()` immediately after URI construction. To simplify this slightly, let the constructor take parameters, similar to `HTTPSFuture`.

Test Plan: See inlines.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13250

Differential Revision: https://secure.phabricator.com/D20151
2019-02-14 11:46:37 -08:00
epriestley
9f5e6bee90 Make the default behavior of getApplicationTransactionCommentObject() "return null" instead of "throw"
Summary:
Depends on D20115. See <https://discourse.phabricator-community.org/t/transaction-search-endpoint-does-not-work-on-differential-diffs/2369/>.

Currently, `getApplicationTransactionCommentObject()` throws by default. Subclasses must override it to `return null` to indicate that they don't support comments.

This is silly, and leads to a bunch of code that does a `try / catch` around it, and at least some code (here, `transaction.search`) which doesn't `try / catch` and gets the wrong behavior as a result.

Just make it `return null` by default, meaning "no support for comments". Then remove the `try / catch` stuff and all the `return null` implementations.

Test Plan:
  - Grepped for `getApplicationTransactionCommentObject()`, fixed each callsite / definition.
  - Called `transaction.search` on a diff with transactions (i.e., not a sourced-from-commit diff).

Reviewers: amckinley

Reviewed By: amckinley

Subscribers: jbrownEP

Differential Revision: https://secure.phabricator.com/D20121
2019-02-07 14:56:38 -08:00
epriestley
f3e154eb02 Allow "inactive" repositories to be read over SSH for cluster sync
Summary:
Fixes T13192. See PHI1015. When you deactivate a repository, we currently stop serving it.

This creates a problem for intracluster sync, since new nodes can't sync it. If nothing else, this means that if you "ship of theseus" your cluster and turn nodes over one at a time, you will eventually lose the entire repository. Since that's clearly a bad outcome, support sync.

Test Plan:
Testing this requires a "real" cluster, so I mostly used `secure`.

I deactivated rGITTEST and ran this on `secure002`:

```
./bin/repository thaw --demote secure002.phacility.net --force GITTEST && ./bin/repository update GITTEST
```

Before the patch, this failed:

```
[2019-01-31 19:40:37] EXCEPTION: (CommandException) Command failed with error #128!
COMMAND
git fetch --prune -- 'ssh://172.30.0.64:22/diffusion/GITTEST/' '+refs/*:refs/*'

STDOUT
(empty)

STDERR
Warning: Permanently added '172.30.0.64' (RSA) to the list of known hosts.
phabricator-ssh-exec: This repository ("rGITTEST") is not available over SSH.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
```

After applying (a similar patch to) this patch to `secure001`, the sync worked.

I'll repeat this test with the actual patch once this deploys to `secure`.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13192

Differential Revision: https://secure.phabricator.com/D20077
2019-01-31 22:12:13 -08:00
epriestley
881d79c1ea When dirtying repository cluster routing caches after an Almanac edit, discover linked bindings from devices
Summary:
See PHI1030. When you edit an Almanac object, we attempt to discover all the related objects so we can dirty the repository cluster routing cache: if you modify a Device or Service that's part of a clustered repository, we need to blow away our cached view of the layout.

Currently, we don't correctly find linked Bindings when editing a Device, so we may miss Services which have keys that need to be disabled. Instead, discover these linked objects.

See D17000 for the original implementation and more context.

Test Plan:
  - Used `var_dump()` to dump out the discovered objects and dirtied cache keys.
  - Before change: editing a Service dirties repository routing keys (this is correct), but editing a Device does not.
  - After change: editing a Device now correctly dirties repository routing keys.

Reviewers: amckinley

Reviewed By: amckinley

Differential Revision: https://secure.phabricator.com/D20003
2019-01-21 10:32:48 -08:00