Summary:
Ref T13538. See PHI1739. Synthetic Git commits with no author and/or no commit message currently extract `null` and then fail to parse.
Ideally, we would carefully distinguish between `null` and empty string. In practice, that requires significant schema changes (these columns are non-nullable and have indexing requirements) and these cases are degenerate. These commits are challenging to build and can not normally be constructed with `git commit`.
At least for now, merge the `null` cases into the empty string cases so we can survive import.
Test Plan:
- Constructed a commit with no author and no commit message using the approach described in T13538; pushed and parsed it.
- Before: fatals during identity selection and storing the commit message (both roughly NULL inserts into non-null columns).
- After: clean import.
This produces a less-than-ideal UI in Diffusion, but it doesn't break anything:
{F7492094}
Maniphest Tasks: T13538
Differential Revision: https://secure.phabricator.com/D21266
Summary:
See PHI1688. If many refs with a large amount of shared ancestry are deleted from a repository, we can spend much longer than necessary marking their mutual ancestors as unreachable over and over again.
For example, if refs A, B and C all point near the head of an obsolete "develop" branch and have about 1K shared commits reachable from no other refs, deleting all three refs will lead to us performing 3,000 mark-as-unreachable operations (once for each "<ref, commit>" pair).
Instead, we can stop exploring history once we reach an already-unreachable commit.
Test Plan:
- Destroyed 7 similar refs simultaneously.
- Ran `bin/repository refs`, saw 7 entries appear in the `oldref` table.
- Ran `bin/repository discover` with some debugging statements added, saw sensible-seeming behavior which didn't double-mark any newly-unreachable refs.
Differential Revision: https://secure.phabricator.com/D21056
Summary:
Depends on D21053. Ref T11968. Three things have changed:
- Overseers can no longer use FutureIterator to continue execution of an arbitrary list of futures from any state. Use FuturePool instead.
- Same with repository daemons.
- Probably (?) fix an API change in the Harbormaster exec future.
Test Plan:
- Ran "bin/phd debug task" and "bin/phd debug pull", no longer saw Future-management related errors.
- The Harbormaster future is easiest to test by just seeing if production works once this change is deployed there.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21054
Summary:
Fixes T13479. The behavior of "git rev-parse --show-toplevel" has changed in Git 2.25.0, and it now fails in bare repositories.
Instead, use "git rev-parse --git-dir" to sanity-check the working copy. This appears to have more stable behavior across Git versions, although it's a little more complicated for our purposes.
Test Plan:
- Ran `bin/repository update ...` on an observed, bare repository.
- ...on an observed, non-bare ("legacy") repository.
- ...on a hosted, bare repository.
Maniphest Tasks: T13479
Differential Revision: https://secure.phabricator.com/D20945
Summary: Ref T13444. Allow the effects of performing an identity rebuild to be previewed without committing to any changes.
Test Plan: Ran "bin/repository rebuild-identities --all-identities" with and without "--dry-run".
Maniphest Tasks: T13444
Differential Revision: https://secure.phabricator.com/D20922
Summary:
Fixes T13457. Ref T13444. When we iterate over commits in a particular repository, the default iteration strategy can't effectively use the keys on the table.
Tweak the ordering so the "<repositoryID, epoch, [id]>" key can be used.
Test Plan:
- Ran `bin/audit delete --repository X` and `bin/repository rebuild-identities --repository X` before and after changes.
- With just the key changes, performance was slightly better. My local data isn't large enough to really emphasize the key changes.
- With the page size changes, performance was a bit better (~30%, but on 1-3 second run durations).
- Used `--trace` and ran `EXPLAIN ...` on the new queries, saw them select the "<repositoryID, epoch, [id]>" key and report a bare "Using index condition" in the "Extra" column.
Maniphest Tasks: T13457, T13444
Differential Revision: https://secure.phabricator.com/D20921
Summary:
Ref T13444. Currently, many mutations to users and email addresses (particularly: user creation; and user and address destruction) do not propagate properly to repository identities.
Add hooks to all mutation workflows so repository identities get rebuilt properly when users are created, email addresses are removed, users or email addresses are destroyed, or email addresses are reassigned.
Test Plan:
- Added random email address to account, removed it.
- Added unassociated email address to account, saw identity update (and associate).
- Removed it, saw identity update (and disassociate).
- Registered an account with an unassociated email address, saw identity update (and associate).
- Destroyed the account, saw identity update (and disassociate).
- Added address X to account A, unverified.
- Invited address X.
- Clicked invite link as account B.
- Confirmed desire to steal address.
- Saw identity update and reassociate.
Maniphest Tasks: T13444
Differential Revision: https://secure.phabricator.com/D20914
Summary:
Ref T13444. Repository identities have, at a minimum, some bugs where they do not update relationships properly after many types of email address changes.
It is currently very difficult to fix this once the damage is done since there's no good way to inspect or rebuild them.
Take some steps toward improving observability and providing repair tools: allow `bin/repository rebuild-identities` to effect more repairs and operate on identities more surgically.
Test Plan: Ran `bin/repository rebuild-identities` with all new flags, saw what looked like reasonable rebuilds occur.
Maniphest Tasks: T13444
Differential Revision: https://secure.phabricator.com/D20911
Summary: Ref T13444. Send all repository identity/detection through a new "DiffusionRepositoryIdentityEngine" which handles resolution and detection updates in one place.
Test Plan:
- Ran `bin/repository reparse --message ...`, saw author/committer identity updates.
- Added "goose@example.com" to my email addresses, ran daemons, saw the identity relationship get picked up.
- Ran `bin/repository rebuild-identities ...`, saw sensible rebuilds.
Maniphest Tasks: T13444
Differential Revision: https://secure.phabricator.com/D20910
Summary: Ref T13444. This is an ancient event and part of the old event system. It is not likely to be in use anymore, and repository identities should generally replace it nowadays anyway.
Test Plan: Grepped for constant and related methods, no longer found any hits.
Maniphest Tasks: T13444
Differential Revision: https://secure.phabricator.com/D20909
Summary:
Ref T13444. You can currently explicitly unassign an identity (useful if the matching algorithm is misfiring). However, this populates the "currentEffectiveUserPHID" with the "unassigned()" token, which mostly makes things more difficult.
When an identity is explicitly unassigned, convert that into an explicit `null` in the effective user PHID.
Then, realign "assigned" / "effective" language a bit. Previously, `withAssigneePHIDs(...)` actualy queried effective users, which was misleading. Finally, bulk up the list view a little bit to make testing slightly easier.
Test Plan:
- Unassigned an identity, ran migration, saw `currentEffectiveUserPHID` become `NULL` for the identity.
- Unassigned a fresh identity, saw NULL.
- Queried for various identities under the modified constraints.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13444
Differential Revision: https://secure.phabricator.com/D20908
Summary:
Ref T13444. Currently, identities for a particular email address are queried with "LIKE" against a binary column, which makes the query case-sensitive.
- Extract the email address into a separate "sort255" column.
- Add a key for it.
- Make the query a standard "IN (%Ls)" query.
- Deal with weird cases where an email address is 10000 bytes long or full of binary junk.
Test Plan:
- Ran migration, inspected database for general sanity.
- Ran query script in T13444, saw it return the same hits for "git@" and "GIT@".
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13444
Differential Revision: https://secure.phabricator.com/D20907
Summary:
Fixes T13448. We currently "git clone" to initialize repositories, but this will fetch too many refs if "Fetch Refs" is configured.
In modern Phabricator, there's no apparent reason to "git clone"; we can just "git init" instead. This workflow naturally falls through to an update, where we'll do a "git fetch" and pull in exactly the refs we want.
Test Plan:
- Configured an observed repository with "Fetch Refs".
- Destroyed the working copy.
- Ran "bin/repository pull X --trace --verbose".
- Before: saw "git clone" pull in the world.
- After: saw "git init" create a bare empty working copy, then "git fetch" fill it surgically.
Both flows end up in the same place, this one is just simpler and does less work.
Maniphest Tasks: T13448
Differential Revision: https://secure.phabricator.com/D20894
Summary:
Ref T13448. The default behavior of "git fetch '+refs/heads/master:refs/heads/master'" is to follow and fetch associated tags.
We don't want this when we pass a narrow refspec because of "Fetch Refs" configuration. Tell Git that we only want the refs we explicitly passed.
Note that this doesn't prevent us from fetching tags (if the refspec specifies them), it just stops us from fetching extra tags that aren't part of the refspec.
Test Plan:
- Ran "bin/repository pull X --trace --verbose" in a repository with a "Fetch Refs" configuration, saw only "master" get fetched (previously: "master" and reachable tags).
- Ran "git fetch --no-tags '+refs/*:refs/*'", saw tags fetched normally ("--no-tags" does not disable fetching tags).
Maniphest Tasks: T13448
Differential Revision: https://secure.phabricator.com/D20893
Summary:
Ref T13448. When "Fetch Refs" is configured:
- We switch to a narrow mode when running "ls-remote" against the local working copy. This excludes surplus refs, so we'll incorrectly detect that the local and remote working copies are identical in cases where the local working copy really has surplus refs.
- We rely on "--prune" to remove surplus local refs, but it only prunes refs matching the refspecs we pass "git fetch". Since these refspecs are very narrow under "Fetch Only", the pruning behavior is also very narrow.
Instead:
- When listing local refs, always list everything. If we have too much stuff locally, we want to get rid of it.
- When we identify surplus local refs, explicitly delete them instead of relying on "--prune". We can just do this in all cases so we don't have separate "--prune" and "manual" cases.
Test Plan:
- Created a new repository, observed from a GitHub repository, with many tags/refs/branches. Pulled it.
- Observed lots of refs in `git for-each-ref`.
- Changed "Fetch Refs" to "refs/heads/master".
- Ran `bin/repository pull X --trace --verbose`.
On deciding to do something:
- Before: since "master" did not change, the pull declined to act.
- After: the pull detected surplus refs and deleted them. Since the states then matched, it declined further action.
On pruning:
- Before: if the pull was forced to act, it ran "fetch --prune" with a narrow refspec, which did not prune the working copy.
- After: saw working copy pruned explicitly with "update-ref -d" commands.
Also, set "Fetch Refs" back to the default (empty) and pulled, saw everything pull.
Maniphest Tasks: T13448
Differential Revision: https://secure.phabricator.com/D20892
Summary: Ref T13440. This feature is used in only one interface which I'm about to rewrite, so throw it away.
Test Plan: Grepped for all affected symbols, didn't find any hits anywhere.
Maniphest Tasks: T13440
Differential Revision: https://secure.phabricator.com/D20882
Summary:
Fixes T13435. If you move Phabricator or copy data from one environment to another, the repository URI index currently still references the old URI, since it writes the URI as a plain string. This may make "arc which" and similar workflows have difficulty identifying repositories.
Instead, store the "phabricator.base-uri" domain and the "diffusion.ssh-host" domain as tokens, so lookups continue to work correctly even after these values change.
Test Plan:
- Added unit tests to cover the normalization.
- Ran migration, ran daemons, inspected `repository_uriindex` table, saw a mixture of sensible tokens (for local domains) and static domains (like "github.com").
- Ran this thing:
```
$ echo '{"remoteURIs": ["ssh://git@local.phacility.com/diffusion/P"]}' | ./bin/conduit call --method repository.query --trace --input -
Reading input from stdin...
>>> [2] (+0) <conduit> repository.query()
>>> [3] (+3) <connect> local_repository
<<< [3] (+3) <connect> 555 us
>>> [4] (+5) <query> SELECT `r`.* FROM `repository` `r` LEFT JOIN `local_repository`.`repository_uriindex` uri ON r.phid = uri.repositoryPHID WHERE (uri.repositoryURI IN ('<base-uri>/diffusion/P')) GROUP BY `r`.phid ORDER BY `r`.`id` DESC LIMIT 101
<<< [4] (+5) <query> 596 us
<<< [2] (+6) <conduit> 6,108 us
{
"result": [
{
"id": "1",
"name": "Phabricator",
"phid": "PHID-REPO-2psrynlauicce7d3q7g2",
"callsign": "P",
"monogram": "rP",
"vcs": "git",
"uri": "http://local.phacility.com/source/phabricator/",
"remoteURI": "https://github.com/phacility/phabricator.git",
"description": "asdf",
"isActive": true,
"isHosted": false,
"isImporting": false,
"encoding": "UTF-8",
"staging": {
"supported": true,
"prefix": "phabricator",
"uri": null
}
}
]
}
```
Note the `WHERE` clause in the query normalizes the URI into "<base-uri>", and the lookup succeeds.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13435
Differential Revision: https://secure.phabricator.com/D20872
Summary: Fixes T13430. Provide more information about repositories in "diffusion.repository.search".
Test Plan: Used API console to call method (with new "metrics" attachment), reviewed output. Saw new fields returned.
Maniphest Tasks: T13430
Differential Revision: https://secure.phabricator.com/D20862
Summary:
Depends on D20853. See PHI1474. If the list of "--not" refs is sufficiently long, we may exceed the maximum size of a command.
Use "--stdin" instead, and swap "--not" for the slightly less readable but functionally equivalent "^hash", which has the advantage of actually working with "--stdin".
Test Plan: Ran `bin/repository refs ...` with nothing to be done, and with something to be done.
Differential Revision: https://secure.phabricator.com/D20854
Summary: See PHI1474. This query can become large enough to exceed reasonable packet limits. Chunk the query so it is split up if we have too many identifiers.
Test Plan: Ran `bin/repository refs ...` on a repository with no new commits and a repository with some new commits.
Differential Revision: https://secure.phabricator.com/D20853
Summary:
Fixes T13284. See that task for substantial discussion. There are currently two cases where we'll skip over commits which we should publish:
- if a branch is not permanent, then later made permanent; or
- in some cases, the first time we examine branches in a repository.
In both cases, this error is one-shot and things work correctly going forward. The root cause is conflation between the states "this ref currently permanent" and "this ref was permanent the last time we updated refs".
Separate these pieces of state and cover all these cases. Also introduce a "--rebuild" flag to fix the state of bad commits.
Test Plan:
See T13284 for the three major cases:
- initial import;
- push changes to a nonpermanent branch, update, then make it permanent;
- push chanegs to a nonpermanent branch, update, push more changes, then make it permanent.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13284
Differential Revision: https://secure.phabricator.com/D20829
Summary:
Ref T13286. To support request retries, allow the service lookup method to return an ordered list of structured service references.
Existing callsites continue to immediately discard all but the first reference and pull a URI out of it.
Test Plan: Ran `git pull` in a clustered repository with an "up" node and a "down" node, saw 50% serivce failures and 50% clean pulls.
Maniphest Tasks: T13286
Differential Revision: https://secure.phabricator.com/D20775
Summary:
Ref T13393. While doing a shard migration in the Phacility cluster, we'd like to stop writes to the migrating repository. It's safe to continue serving reads.
Add a simple maintenance mode for making repositories completely read-only during maintenance.
Test Plan: Put a repository into read-only mode, tried to write via HTTP + SSH. Viewed web UI. Took it back out of maintenance mode.
Maniphest Tasks: T13393
Differential Revision: https://secure.phabricator.com/D20748
Summary:
Ref T13369. See that task for discussion.
When the discovery daemon finds more than 64 commits to import, demote the worker queue priority of the resulting tasks.
Test Plan:
- Pushed one commit, ran `bin/repository discover --verbose --trace ...`, saw commit import with "at normal priority" message and priority 2500 ("PRIORITY_COMMIT").
- Pushed 3 commits, set threshold to 3, ran `bin/repository discover ...`, saw commist import with "at lower priority" message and priority 4000 ("PRIORITY_IMPORT").
Maniphest Tasks: T13369
Differential Revision: https://secure.phabricator.com/D20712
Summary:
Fixes T13315. See that task for discussion.
Without `--background`, we currently treat this as a catastrophic failure, but it's relatively routine for some repository states. We can safely continue reparsing other steps.
Test Plan: Ran `bin/repository reparse --all X --message` with commits faked to all be unreachable. Got warnings instead of a hard failure on first problem.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13315
Differential Revision: https://secure.phabricator.com/D20588
Summary:
Ref T13311. We currently don't use committer identity mappings when triggering audits, so if a user is only associated with an identity via manual mapping we won't treat them as the author.
Instead, use the identity and manual mapping if they're available.
Test Plan:
- Pushed a commit as `xyz <xyz@example.org>`, an address with no corresponding user.
- In the UI, manually associated that identity with user `@alice`.
- Ran `bin/repository reparse --publish <hash>` to trigger audits and publishing for the commit.
- Before: observed the `$author_phid` was `null`.
- After: observed the `$author_phid` is Alice.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13311
Differential Revision: https://secure.phabricator.com/D20580
Summary:
See PHI1118. That issue may describe more than one bug, but the recent ordering changes to the import pipeline likely make this at least part of the problem.
Previously, commits would always close associated revisions before we made it to the "publish" step. This is no longer true, so we might be triggering audits on a commit before the associated revision actually closes.
Accommodate this by counting a revision in either "Accepted" or "Published (Was Previously Accepted)" as "reviewed".
Test Plan:
- With commit C affecting paths in package P with "Audit Unreviewed Commits and Commits With No Owner Involvement", associated with revision R, with both R and C authored by the same user, and "R" in the state "Accepted", used `bin/repository reparse --publish <hash>` to republish the commit.
- Before change: audit by package P triggered.
- After change: audit by package P no longer triggered.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20564
Summary:
Ref T13290. Ref T13291. Now that a full URI is a "mention", the full URI in "Differential Revision: ..." also triggers a mention.
Stop it from doing that, since these mentions are silly/redundant/unintended.
The API here is also slightly odd; simplify it a little bit to get rid of doing "append" with "get + append + set".
Test Plan: Used `bin/repository reparse --publish` to republish commits with "Differential Revision: ..." and verified that the revision PHID was properly dropped from the mention list.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13291, T13290
Differential Revision: https://secure.phabricator.com/D20544
Summary:
Depends on D20530. Ref T13291. When users paste links to files in Diffusion into remarkup contexts, identify them and specialize the rendering.
When the URIs are embedded with `{...}`, parse them in more detail.
This is a lead-up to a `{src ...}` rule which will use the same `View` but give users more options to customize presentation.
Test Plan: {F6463580}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13291
Differential Revision: https://secure.phabricator.com/D20538
Summary:
Depends on D20467. Ref T13277. Currently, the "MessageParserWorker" writes this property on commits, then Herald and Audit both read it.
Make them share code so this property has one writer and one reader. This property isn't great, but at least now the badness is hidden.
Currently, we can't just use edges because they may not have been written yet. I am likely to just do this, soon:
- Just write the edges (in "MessageParserWorker").
- Hide the edges from mail.
However, we'll sort-of lose the "revisionMatchData" explanation thing if I do that. Maybe this is fine? But when commits match because hashes match, it legitimately isn't obvious.
For now, just reduce the amount of harm/badness here.
Test Plan:
- Ran `bin/repository reparse --publish ...`.
- Ran a Herald "Audit" rule using the "Accepted Differential revision" field.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20468
Summary:
Depends on D20466. Ref T13277. Currently:
- The "Owners" worker writes ownership relationships (e.g., commit X affects package Y, because it touches a path in package Y) -- these are just edges.
- It also triggers audits.
- Then it queues a "Herald" worker.
- This formally publishes the commit and triggers Herald.
These aren't really separate steps and can happen more easily in one shot. Merge them.
Test Plan:
- Ran `bin/repository reparse --publish` to republish various commits, got sensible behavior.
- Grepped for "IMPORTED_OWNERS", "IMPORTED_HERALD", "--herald", "--owners", and "--force-local" flags.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20467
Summary:
Depends on D20465. Ref T13277. Currently, when a commit is unpublished, we put a single line about it on the "Edit Commit" page. This is pretty much impossible to find.
Move it to the main page. This treatment is more big/bold than I'd probably like to end up, but we should probably overshoot on the explanatory text until users get used to this behavior.
Also, allow searching for only published / unpublished commits.
Test Plan: {F6395705}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20466
Summary:
Depends on D20464. Ref T13277. Broadly:
- Move all the "should publish X" and "why aren't we publishing X" stuff to a separate class (`PhabricatorRepositoryPublisher`).
- Rename things to be more consistent with modern terminology ("Publish", "Permanent Refs").
Test Plan:
This could use some trial-by-fire on `secure`, but:
- Grepped for all symbols.
- Viewed various commits.
- Reparsed commits.
- Here's a commit with an explanation:
{F6394569}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20465
Summary: Depends on D20463. Ref T13277. This flag was added some time before 2015 and I don't think I've ever used it. Just get rid of it.
Test Plan: Grepped for `force-autoclose`, `forceAutoclose`, `AUTOCLOSE_FORCED`.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20464
Summary:
Depends on D20462. Ref T13276. Currently, the "Message" parser also updates related tasks and revisions when a commit is published.
For PHI1165, which ran into a race with message parsing, I originally believed we needed to separate this logic and lock + yield to avoid the race. D20462 provides what is probably a better approach for avoiding the race.
Still, I think separating these "update related revisions" and "updated related tasks" chunks into separate workers is a net improvement. There may still be some value in doing lock + yield in the future to deal with other issues, and when we occasionally run into problems with pulling a diff out of the repository to update the revision (usually because the diff is too big) this isolates the problem better and allows the commit to import.
I think the only thing to watch out for here is that Herald may now run before the revision and commit are attached to one another. This is fine for all current Herald rules, we just need to be mindful in implementing new rules.
Test Plan: Used `bin/repository reparse --message` on various commits, including commits that close revisions and close tasks.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13276
Differential Revision: https://secure.phabricator.com/D20463
Summary: Depends on D20459. Ref T13276. I'll file a followup to actually destroy the table.
Test Plan:
- Grepped for `TABLE_COMMIT`.
- Ran `bin/storage upgrade -f`, got a clean bill of health.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13276
Differential Revision: https://secure.phabricator.com/D20461
Summary:
Ref T13280. In D20421, I changed our observe strategy to `git fetch <uri>` in all cases.
This doesn't work in an ancient, non-bare repository if `master` is checked out and `master` is also fetch: `git` refuses to overwrite the local ref unless we pass `--update-head-ok`. Pass this flag.
Also, remove some code which examines branches and tags in a special way for non-bare working copies. The old `git fetch <origin>` code without explicit revsets meant that `refs/remotes/orgin/heads/xyz` got updated instead of `refs/heads/xyz`. We now update our local refs in all cases (bare and non-bare) so we can throw away this special casing.
Test Plan:
- Replaced a modern bare working copy with a non-bare working copy by explicitly using `git clone` without `--bare`.
- Ran `bin/repository update`, hit the `--update-head-ok` error. Applied the patch, got a clean update.
- Used the "repository.branchquery" API method...
- ...with "contains" to trigger the "git branch" case. Got identical results after removing the special casing.
- ...without "contains" to trigger the "low level ref" case. Got identical results after removing the special casing.
- Grepped for `isWorkingCopyBare()`. The only remaining callsites deal with hook paths, and genuinely need to be specialized.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13280
Differential Revision: https://secure.phabricator.com/D20450
Summary:
Ref T13279. Charting changes alter how the "line-chart" behavior works, but the "Burnup Chart" still relies on the old behavior.
Although I'm intending to remove "Maniphest > Reports" once Facts is a minimally sufficient replacement, copy this behavior to keep it working until we're ready to pull the trigger.
Also fix a leftover typo from D20435.
Test Plan: Viewed a legacy Maniphest burnup rate report.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13279
Differential Revision: https://secure.phabricator.com/D20449
Summary:
Depends on D20434. Fixes T5963. Broadly, the issue here is that when:
- You create a new, empty repository.
- Then, you work on some branch other than `master`, without ever creating `master`.
...you get a warning on `git clone`:
> warning: remote HEAD refers to nonexistent ref, unable to checkout
To fix this, point the symbolic-ref HEAD at `refs/heads/<default-branch>` after installing commit hooks.
This fixes the warning, and also means that `git clone` will check out the repository default branch by default, which is nice.
There are a few caveats about this behavior (see T5963 for discussion) but nothing too substantial.
The only real issue is that Git prevents deletion of the default branch without a config setting. Just set that settting.
Test Plan:
See T5963.
In a repository, set `HEAD` to point somewhere invalid. Ran `bin/repository update ...`. Saw HEAD pointed back at the repository default branch.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T5963
Differential Revision: https://secure.phabricator.com/D20435
Summary:
Depends on D20427. Ref T13277. As an optimization, when we discover that a commit which was previously only on a non-permanent ref ("tmp-epriestley-123") is now reachable from a permanent ref ("master"), we currently queue only a new "message" parse step.
This is an optimization because these commits previously got the full treatment (feed, publish, audit, etc) as soon as they were discovered. Now, those steps only happen once a commit is reachable from a permanent ref, so we need to run everything.
Test Plan:
- Pushed local "tmp-123" branch to remote tag "tmp-123".
- Updated repository with "bin/repository update", saw commit import as a "not on any permanent ref" commit, with no herald/audit/etc.
- Merged "tmp-123" tag into "master".
- Pushed new "master".
- Updated repository with "bin/repository refs ... --trace --verbose", saw commit detected as now reachable from a permament ref.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20428
Summary:
Depends on D20425. Ref T13277. See PHI1067. There's currently no way to retrieve branch/ref rules over the API, which makes some management operations against a large number of repositories difficult.
Expose these rules to the API.
Test Plan: Called `diffusion.repository.search`, got rules in the result set.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20426
Summary:
Depends on D20423. Ref T13277. Repositories currently have separate toggles for "Autoclose" and "Publishing".
Merge the "Autoclose" toggle into the "Publishing" toggle. I'm unaware of any valid use case for enabling one but not the other.
(This doesn't fix all the documentation, yet.)
Test Plan: Edited a repository, saw only one publishing option.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20424
Summary:
Depends on D20422. Ref T13277. Currently, "track only", "publish", and "autoclose" are three separate ideas. I'd like to generally merge them into a more natural idea called "permanent refs".
Since "Autoclose" effectively now controls both "autoclose" and "publish", rename it.
This doesn't rename all the methods or internals, and the documentation needs an update, but it renames most of the UI-facing stuff.
(You also can only specify branches as "Permanent Refs" today, but we may let you specify tags and other arbitrary refs in the future.)
Test Plan: Grepped, poked around the UI, saw UI show "Permanent" / "Permanent Refs" more often and "Autoclose" less.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20423
Summary:
Depends on D20421. Ref T13277. I'd generally like to move away from "Track Only".
Some of the use cases for "Track Only" (or adjacent to "Track Only") are better resolved with "Fetch Rules" -- basically, rules to fetch only some subset of refs from the observed remote.
Add configurable "Fetch Rules" for Git repositories. For example, if you only want to fetch `master`, you can now speify:
```
refs/heads/master
```
If you only want to fetch branches and tags, you can use:
```
refs/heads/*
refs/tags/*
```
In theory, this is slightly less powerful in the general case than "Track Only", but gives us better behavior in some cases (e.g., when the remote has 50K random temporary branches). In practice, I think this and a better "Autoclose Only" will let us move away from "Track Only", get default behavior which is better aligned with what users actually expect, and dodge all the "track tags/refs" questions.
Test Plan: Configured repositories with "Fetch Refs" rules, used `bin/repository pull --verbose --trace ...` to run pulls, saw expected pull/fetch behavior.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20422
Summary:
Depends on D20420. Ref T13277. We currently spend substantial effort trying to detect and correct the URL of the "origin" remote in Git repositories.
I believe this is unnecessary, and we can always `git fetch <url> ...` to get the desired result instead of `git muck-with-origin + git fetch origin ...`. We already do this in the more recent parts of the codebase (e.g., intracluster sync) and it works correctly in every case I'm aware of.
Test Plan:
- Grepped for `origin`, ` origin `.
- Ran `bin/repository update ...` to fetch a mirrored repository.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277
Differential Revision: https://secure.phabricator.com/D20421
Summary:
Depends on D20418. Ref T13277. Fixes T11314.
Currently, when you push commits to some arbitrary ref or tag (like `refs/pull/123` on GitHub, `refs/tags/phabricator/diff/123` on Phabricator, or `refs/changes/whatever` on Gerrit), we do not "autoclose" related objects. This means that we don't process `Ref T123` to create links to tasks, and don't process `Differential Revision: xyz` to close revisions.
However, we //do// still publish these commits. "Publish" means: trigger audits, publish feed stories, and run Herald rules.
- Stop publishing these commits.
- In the UI, show these commits as "Not Permanent" with a note that they are "Not [on any permanent branch]."
These commits will publish and autoclose if they ever become reachable from an "autoclose" ref (most commonly, if they are later merged to "master").
Test Plan:
- Pushed a commit to `refs/tags/quack`.
- Before: got a feed story.
- After: no feed story, UI shows commit as "Not Permanent".
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277, T11314
Differential Revision: https://secure.phabricator.com/D20419
Summary:
Depends on D20419. Ref T13277. Fixes T8936. Fixes T9383. Fixes T12300. When you push arbitrary refs to Phabricator, the push log currently complains if those refs are not tags or branches.
Upstream Git now features "notes", and there's no reason to prevent writes to arbitrary refs, particularly beause we plan to start using them soon (see T13278).
Allow these writes as affecting raw refs.
Test Plan:
- Pushed an arbitrary ref.
- Pushed some Git notes.
- Wrote a Herald ref rule, saw "ref" in the dropdown.
{F6376492}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13277, T8936, T9383, T12300
Differential Revision: https://secure.phabricator.com/D20420
Summary:
Ref T12164. Ref T13276. Currently, the parsing pipeline copies the author and committer names and PHIDs into the transcaction record as metadata. They are then rendered directly from the metadata.
This makes planned changes to the parsing pipeline (to prevent races when multiple commits matching a single revision are pushed simultaneously) more difficult, and generally won't work with repository identities.
Instead, load the commit and use its identities.
Test Plan: Loaded a revision, saw the same story rendering for a "Closed by commit" story.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13276, T12164
Differential Revision: https://secure.phabricator.com/D20418
