Summary:
See T13572. FutureIterator does not release futures, so long-lived iterators (like the one that FuturePool may build) can end up leaking memory.
This affects the FuturePool used by the daemon overseer.
See T13572 for more discussion.
Test Plan:
- Ran the simple FutureIterator script from T13572. Before: memory held during iteration, script grows without bound. After: memory released, script uses stable memory.
- Ran the overseer with memory logging and an immediate wakeup from hibernation. Before: saw memory usage grow without bound at a rate of ~300MB/day. After: saw memory usage stable.
Differential Revision: https://secure.phabricator.com/D21466
Summary: Ref T13581. This message can be slightly more helpful in some cases by showing which method call failed.
Test Plan: Ran `arc branches` under the error condition in D21462, got a more useful error.
Maniphest Tasks: T13581
Differential Revision: https://secure.phabricator.com/D21463
Summary:
Ref T13577. After the lint rule fix in D21453, it can identify more errors. Fix the errors it identifies in "arcanist/".
These all seem fairly obscure/benign.
Test Plan: Ran `arc lint` on the files before and after these changes. Did not specifically re-test these particular messages, but they mostly very obscure.
Maniphest Tasks: T13577
Differential Revision: https://secure.phabricator.com/D21456
Summary:
Ref T13577. I'd like to `arc lint --everything` to find other bad calls to `pht()` and similar functions, but `n_HEREDOC` nodes currently can not generate a response to "getStringLiteralValue()".
Support literal extraction from heredocs.
Test Plan: Added a test, made it pass. Will lint everything.
Maniphest Tasks: T13577
Differential Revision: https://secure.phabricator.com/D21455
Summary: Ref T13577. This call is missing a parameter. After D21453, this is detected properly by lint. Provide the parameter.
Test Plan: Ran `arc lint` on HTTPSFuture before and after the change.
Maniphest Tasks: T13577
Differential Revision: https://secure.phabricator.com/D21454
Summary:
Ref T13577. This lint rule correctly detects the error in `pht('x %s y')` but the narrow test for `n_STRING_SCALAR` prevents it from detecting the error in `pht('x %s y'.'z')`.
Make the test broader.
Test Plan:
- Ran `arc lint` on `HTTPSFuture.php`, got a detection of the issue in T13577.
- Added a failing test and made it pass.
Maniphest Tasks: T13577
Differential Revision: https://secure.phabricator.com/D21453
Summary:
Ref T13555. Currently:
- If an exception is raised in "start()", the exception state is not set on the future.
- Futures do not always call "startFuture()" before starting, and do not always call "endFuture()" once they become resolvable.
- If you start an ExecFuture which immediately fails and then call "getPID()" on it, you get an unclear exception.
Simplify these behaviors:
- In FutureIterator, only start futures which have not already started.
- When starting a future on any pathway, run start code.
- When a future becomes resolvable on any pathway, run end code.
- Raise a more clear exception when calling "getPID()" on a future with no subprocess.
Test Plan: Faked a failing subprocess with "$proc = null", ran "bin/phd debug taskmaster" etc. Got clearer errors and more consistent future lifecycle workflows.
Maniphest Tasks: T13555
Differential Revision: https://secure.phabricator.com/D21423
Summary:
See PHI1808. Currently, "arc land <some bookmark>" does not destroy the bookmark in Mercurial. There are three issues here:
- "hg rebase --keep --collapse" moves bookmarks to the rewritten commits;
- "hg strip" moves bookmarks backwards;
- "hg prune" moves bookmarks backwards.
To get around "hg rebase", save and restore bookmark state.
To get around "hg strip" and "hg prune", explicitly destroy bookmarks pointing at commits before we strip/prune those commits.
Test Plan:
- Ran "arc land <some bookmark> --trace". Saw arc reset the bookmark position after rebasing, and destroy the bookmark explicitly before stripping.
- When the workflow exited, saw no more bookmark (previously: bookmark existed and pointed at a possibly-intermediate state).
Differential Revision: https://secure.phabricator.com/D21397
Summary:
Ref PHI1808. In Mercurial, we must save and restore bookmark state explicitly.
- Save and restore bookmarks.
- Clean up concepts in "arc-ls-markers" slightly, so we don't need separate "isCurrent" and "isActive" flags, hopefully.
I believe the totality of Mercurial state is:
- A (non-bare) working copy points at exactly one commit (which might be the empty/null commit, in an empty repository).
- A working copy has exactly one active branch.
- Each branch has zero or more heads.
- Each head may be closed.
- Each (non-null) commit belongs to exactly one branch.
- Note that the active branch may have zero heads and zero commits which belong to it!
- A working copy has zero or one active bookmark.
To capture this, we now emit:
- A list of branch heads. If a branch head is a working copy commit, that head is flagged as active.
- A list of bookmarks. If a bookmark is the current bookmark, that bookmark is flagged as active.
- A single "branch-state" virtual marker. This covers the case where you have run "hg branch X" to create X, but no objects in the working copy actually correspond to X yet. It also covers the case where you are on a concrete branch, but not any head of that branch.
- A single "commit-state" virtual marker. This always shows the current commit in the working copy.
Test Plan:
- Useful states to test are:
- Empty repository (not all commands currently work here).
- Normal repository, on a bookmark.
- Normal repository, no bookmark.
- "hg up 123" to update to somewhere in history.
- "hg branch X", to start a new branch with no commits.
- Ran "arc branches" and "arc bookmarks" in various states. Saw generally sensible output.
- Ran "arc land --hold ..." in various states against a failing remote. Saw generally sensible output, and saw working properly restored to the original state.
Differential Revision: https://secure.phabricator.com/D21396
Summary: Ref PHI1808. Currently, push failures are messaged awkwardly. Make this exception handling more selective and the user-facing behavior more readable.
Test Plan: Ran "arc land" against a failing remote, saw a human-readable message instead of a stack trace.
Differential Revision: https://secure.phabricator.com/D21395
Summary: See PHI1808. Some refactoring of the "passthru" API resulted in error conditinos here being dropped. Instead, raise them as exceptions.
Test Plan: Forced "hg push" to fail, used "arc land" against a failed push, saw error behavior instead of "success" feedback.
Differential Revision: https://secure.phabricator.com/D21394
Summary:
Ref PHI1805. Under some combination of versions (Python 3.8?), "arc-ls-markers" is running into additional Python runtime issues.
Sprinkle more "b" around to resolve them? Also clean up a couple of plain "arc" issues.
Test Plan:
Landed a change in Mercurial.
Some of this works fine without changes in Python 3.7/2.7 against Mercurial 4.7/5.4, so this may not be exhaustive.
Differential Revision: https://secure.phabricator.com/D21393
Summary:
Ref T13546. See PHI1805. Currently, the "arc-ls-markers" extension doesn't run under Python 3:
- some stuff needs "b'...'" to mark it as a byte string;
- "dict.iteritems()" is gone in Python 3, and "mercurial.pycompat" isn't always available;
- in Python 3, "json" refuses to print byte strings; and
- the compiler caching behavior in Python 3 has changed.
Try to get these things working in the same way under Python 2 and Python 3.
Test Plan:
Ran this command (with `python` as Python 2, locally):
```
$ python /usr/local/bin/hg --config 'extensions.arc-hg=/Users/epriestley/dev/core/lib/arcanist/support/hg/arc-hg.py' arc-ls-markers --
```
...and this command:
```
$ python3 /usr/local/bin/hg --config 'extensions.arc-hg=/Users/epriestley/dev/core/lib/arcanist/support/hg/arc-hg.py' arc-ls-markers --
```
..and saw the same output in both cases (previously, `python3 ...` fataled in various ways).
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21392
Summary: See PHI1805. This call is constructed improperly and can lead to a fatal in `arc patch` under Mercurial.
Test Plan: In Mercurial, ran a valid `arc patch` operation.
Differential Revision: https://secure.phabricator.com/D21391
Summary:
Ref T13546. Try using unicode box drawing characters to render a more obvious tree struture in "arc branches".
Unclear if this has enough support to use, but seems okay so far.
Test Plan: Ran "arc branches", saw a nicer tree display.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21390
Summary:
Correct two minor Conduit future issues:
- FutureAgent shows up in the trace log as "<mystery>". Since it isn't doing anything useful, solve the mystery and drop it from the log.
- Simply the ConduitFuture code for interacting with the service profiler now that a more structured integration is available.
Test Plan: Ran "arc branches --trace", saw conduit calls and no more "<mystery>" clutter.
Differential Revision: https://secure.phabricator.com/D21388
Summary:
See PHI1802. After D21384, "arc land" and similar with no credentials now properly raise a useful exception, but it isn't formatted readably.
Update the display code to make it look prettier.
Test Plan: Ran "arc land" with no and invalid credentials, got properly formatted output.
Differential Revision: https://secure.phabricator.com/D21387
Summary:
Currently, modern "arc" workflows accept and parse "--anonymous" but don't do anything with it.
I intend to move away from anonymous workflows, which only really supported a tiny subset of unusual workflows on open source installs but added significant complexity to login/auth behavior.
Drop support for this flag.
Test Plan: Grepped for "anonymous", didn't turn up any relevant hits.
Differential Revision: https://secure.phabricator.com/D21386
Summary:
Currently, if you type "arc upload <tab>", we do not autocomplete the working directory. We should, but the "current argument" is the empty string and that's technically a prefix of every flag, so we suggest that you might want a flag instead.
You probably don't. Suggest paths in this case.
Test Plan:
- Ran "arc upload <tab>", saw path completions.
- Ran "arc land <tab>" (this workflow does NOT take paths as arguments), saw flag completions.
Differential Revision: https://secure.phabricator.com/D21385
Summary:
See PHI1802. Currently, we can't raise a "you must login" error in a generic way at the beginning of a workflow because we don't know if a workflow needs credentials or not.
For example, "arc help" does not need credentials but "arc diff" does.
Additionally, some actual Conduit calls do not need credentials ("conduit.ping", "conduit.getcapabilities") and others do.
Although I'd like to simplify this eventually and move away from anonymous/unauthenticated "arc", this isn't trivial today. It's also possible for third-party code to add authenticated calls to "arc help", etc., so even if we could execute these tests upfront it's not obvious we'd want to.
So, for now, we raise "you must login" at runtime, when we receive an authentication error from Conduit.
This got implemented for Toolsets in a well-intentioned but not-so-great way somewhere in wilds/experimental, with an "ArcanistConduitCall" that behaves a bit like a future but is not really a future. This implementation made more sense when ConduitEngine was serving as a future engine, and FutureProxy could not rewrite exceptions.
After the Toolsets code was first written, ConduitEngine has stopped serving as a future engine (this is now in "HardpointEngine"). Since HardpointEngine needs a real future, this "show the user a login message" code gets bypassed. This results in user-visible raw authentication exceptions on some workflows:
```
[2020-06-30 21:39:53] EXCEPTION: (ConduitClientException) ERR-INVALID-SESSION: Session key is not present. at [<arcanist>/src/conduit/ConduitFuture.php:76]
```
To fix this:
- Allow FutureProxy to rewrite exceptions (see D21383).
- Implement "ArcanistConduitCall" as a FutureProxy, not a future-like object.
- Collapse the mixed-mode future/not-quite-a-future APIs into a single "real future" API.
Test Plan:
- Created a paste with "echo hi | arc paste --".
- Uploaded a file with "arc upload".
- Called a raw method with "echo {} | arc call-conduit conduit.ping --".
- Invoked hardpoint behavior with "arc branches".
- Grepped for calls to either "resolveCall()" method, found none.
- Grepped for calls to "newCall()", found none.
- Grepped for "ArcanistConduitCall", found no references.
Then:
- Removed my "~/.arcrc", ran "arc land", got a sensible and human-readable (but currently ugly) exception instead of a raw authentication stack trace.
Differential Revision: https://secure.phabricator.com/D21384
Summary:
See PHI1764. See PHI1802. Address two resolution behaviors for FutureProxy:
- FutureProxy may throw an exception directly from iteration via "FutureIterator" (see PHI1764). This is wrong: futures should throw only when resolved.
- FutureProxy can not change an exception into a result, or a result into an exception, or an exception into a different exception. Being able to proxy the full range of result and exception behavior is useful, particularly for Conduit (see PHI1802).
Make "FutureProxy" more robust in how it handles exceptions from proxied futures.
Test Plan:
Used this script to raise an exception during result processing:
```
<?php
require_once 'support/init/init-script.php';
final class ThrowingFutureProxy
extends FutureProxy {
protected function didReceiveResult($result) {
throw new Exception('!');
}
}
$future = new ImmediateFuture('quack');
$proxy = new ThrowingFutureProxy($future);
$iterator = new FutureIterator(array($proxy));
foreach ($iterator as $resolved) {
try {
$resolved->resolve();
} catch (Exception $ex) {
echo "Caught exception properly on resolution.\n";
}
}
```
Before this change, the exception is raised in the `foreach()` loop. After this change, the exception is raised at resolution time.
Differential Revision: https://secure.phabricator.com/D21383
Summary: This method is private and has no callers. The code has moved to "FileUploader" in a prior change.
Test Plan: Grepped for callers, found none.
Differential Revision: https://secure.phabricator.com/D21382
Summary: Ref T13546. Fixes some issues where marker selection in Mercurial didn't work, and selects "draft()" as the set of commits to show, which is at least somewhat reasonable.
Test Plan: Ran "arc branches" and "arc bookmarks" in Mercurial, got more reasonable output.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21380
Summary: Ref T13546. Allow the commit graph to be queried by date range, and Git to be queried for multiple disjoint commits.
Test Plan: Ran "arc branches" and future code which searches for alternate commit ranges for revisions.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21379
Summary: Ref T13546. If your history includes a long linear sequence of published revisions, summarize them.
Test Plan: Ran "arc branches", saw better summarization of linear published revision sequences.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21367
Summary: Ref T13546. In cases where a given set has a large number of commits, summarize them in the output.
Test Plan: Ran "arc branches", saw long lists of commits (like the history of "stable" summarized).
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21366
Summary: Ref T13546. Make "arc branches" use a flexible grid width and try to match the content to the display width in a reasonable way.
Test Plan: Ran "arc branches" at various terminal widths, got generally sensible output.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21365
Summary: Ref T13546. This is no longer necessary after the introduction of "msortv_natural()", which can handle natural string sorting.
Test Plan: Ran "arc branches", saw the same sorting applied.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21364
Summary:
Ref T13546. Currently, each "land" workflow executes custom graph queries to find commits: move toward abstracting this logic.
The "land" workflow also has a potentially dangerous behavior: if you have "master > A > B > C" and "arc land C", it will land A, B, and C. However, an updated version of A or B may exist elsewhere in the working copy. If it does, "arc land" will incorrectly land an out-of-date set of changes.
To find newer versions of "A" and "B", we need to search backwards from all local markers to the nearest outgoing marker, then compare the sets of changes we find to the sets of changes selected by "arc land".
This is also roughly the workflow that "arc branches", etc., need to show local markers as a tree, and starting in "arc branches" allows the process to be visualized.
As implemented here ,this rendering is still somewhat rough, and the selection of "outgoing markers" isn't good. In Mercurial, we may plausibly be able to use phase markers, but in Git we likely can't guess the right behavior automatically and probably need additional configuration.
Test Plan: Ran "arc branches" and "arc bookmarks" in Git and Mercurial.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21363
Summary:
Ref T13546. When running "arc branches", we want to show all unpublished commits. This is often a different set of commits than "commits not present in any remote".
Attempt to identify published commits by using the permanent ref rules in Phabricator.
Test Plan: Ran "arc look published", saw sensible published commits in Git.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21378
Summary:
Ref T13546. Query and associate known Phabricator repositories to working copy remotes by normalizing and comparing URIs.
This primarily gives us access to "permanentRefRules" so we can tell which branches have published changes.
Test Plan: Ran "arc look remotes" in Git and Mercurial working copies, saw repositories map properly.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21377
Summary:
Ref T13546. Currently, "arc which" provides some amount of inspection but it generally isn't very helpful to users and is too limited and inflexible. "arc inspect" is an internal/debugging workflow.
The new "arc look" is much more aggressively unhelpful.
Test Plan: I'm not sure if this command should allow you to continue at night, because it's too dark.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21376
Summary: Ref T13546. Allow construction of remote refs in Git; previously they were only supported in Mercurial.
Test Plan: Ran "arc inspect remote(origin)" in Git, got a ref.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21375
Summary: Ref T13546. Expose remote refs for inspection via "arc inspect". For now, this only works in Mercurial.
Test Plan: Ran "arc inspect remote(default)" in Mercurial, got a ref out.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21374
Summary: Ref T13546. Move toward smarter remote repository lookup by providing URI normalization code in Arcanist. This diff duplicates code from Phabricator; the next change will collapse it.
Test Plan: Ran unit tests.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21372
Summary:
Ref T13546. Pull some small utility changes out of the deeper stack of "land/markers" changes.
"phutil_partition()" makes it easier to write code that loops over a list grouping elements, then acts on each group. This kind of code is not terribly common, but often feels awkward when implemented with raw primitives.
"msortv()" can support "natural" sorting, which sorts "feature1", "feature2", ..., "feature10" in a more human-readable order.
Test Plan: Ran unit tests, used new behaviors elsewhere in "arc markers" workflows.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21371
Summary:
Ref T13546. In a future change, I'm providing a fancier version of "arc branches" that requires more sophisticated table rendering.
Implement a new view which can do some fancier things, like handle alignment of multi-line table cells.
Test Plan: See future changes.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21360
Summary: Ref T13546. The old "differential.query" call is still used to fill refs when all we have locally is hashes. Add some mappings to improve the resulting refs.
Test Plan: Viewed "arc branches", saw statuses colored more consistently.
Reviewers: ptarjan
Reviewed By: ptarjan
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21355
Summary: Ref T13546. If "arc land" would create a branch, warn the user before it does.
Test Plan: Ran "arc land --onto mtarse", a typo of "master".
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21354
Summary: Ref T13546. All of LandEngine, LocalState, and RepositoryAPI implement "getDisplayHash()". Always use the RepositoryAPI implementation.
Test Plan: Grepped for symbols.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21353
Summary:
Ref T13546. Show ongoing and failed builds more clearly in "arc land" output.
Also rename "DisplayRef" (which is not a "Ref") to "RefView" with the goal of improving clarity, and let callers "build...()" it so they can add more status, etc., information.
Get rid of "[DisplayRef|RefView]Interface". In theory, future refs (say, in Phabricator) might not do anything here, but every Ref just ends up implementing it. This could perhaps be subclassed more narrowly in the future if necessary.
Test Plan: Ran "arc land", grepped for various symbols.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21352
Summary:
See PHI1782. If a linter raises a message at a line which does not exist in the file, render a confused warning rather than fataling.
This is a long-existing issue which was exacerbated by D21044.
Test Plan: Modified a linter to raise issues at line 99999. Before change: fatal in console rendering. After change: reasonable rendering.
Differential Revision: https://secure.phabricator.com/D21357
Summary:
Ref T13546. Ref T9948.
- Make "--hold" show the same set of commands to manually push that the normal workflow would use.
- Make save/restore state work.
- Make bookmark creation prompt for confirmation.
- Improve / provide some additional warnings and help text.
Test Plan: Ran various increasingly complex "arc land" workflows, e.g. "arc land --hold --onto fauxmark1 --onto fauxmark2 --into default . --revision 118 --trace"
Maniphest Tasks: T13546, T9948
Differential Revision: https://secure.phabricator.com/D21351
Summary: Ref T9948. Ref T13546. Clean up some minor behaviors to allow "arc land" to function in the simplest cases again. Also, do a capability test for "prune" rather than just falling back.
Test Plan: Ran "arc land <mark>" in Mercurial, got changes pushed.
Maniphest Tasks: T13546, T9948
Differential Revision: https://secure.phabricator.com/D21350
Summary:
Ref T9948. Ref T13546. To identify remote branch heads -- not just modified heads -- use "branchmap" like "hg outgoing" does.
I wasn't able to find any other way to get what we want: for example, with a bundlerepo, "peer.heads()" and "peer.changelog.heads()" include local branches which are not present in the remote.
It generally seems difficult (perhaps impossible?) to distinguish between these cases by using "getremotechanges()":
- branch X exists at position Y in both the local and remote;
- branch X exists at positino Y in the local, but not the remote.
In any case, this seems to work properly and //should// do less work than "getremotechanges()" since we don't need to pull as much data over the wire.
Test Plan: Ran "hg arc-ls-markers" in various repositories, got what appeared to be a faithful representation of the remote branch and bookmark state.
Maniphest Tasks: T13546, T9948
Differential Revision: https://secure.phabricator.com/D21349
Summary:
Ref T9948. Ref T13546. This change moves toward a functional "arc land" in Mercurial.
Because of how "bundlerepo.getremotechanges()" works, "hg arc-ls-markers" does not actually list markers in the remote that aren't different from local markers so it's hard to get anywhere with this.
Test Plan: Got somewhat-encouraging output from "arc land" and "hg arc-ls-markers", but too many things are still broken for this to really work yet.
Maniphest Tasks: T13546, T9948
Differential Revision: https://secure.phabricator.com/D21348
Summary: Ref T13546. In the new "arc land": actually reach build warnings; and show buildable URIs.
Test Plan: Ran "arc land ..." with intentionally broken builds, got more useful build warnings.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21347
Summary: Ref T13546. The modern constant from the modern API method for this state is "published", and this more narrowly covers the desired behavior (notably, excluding "Abandoned" revisions).
Test Plan: Ran "arc land ... --revision X" where "X" is a published revision, got an appropriate prompt.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21345
