Summary:
Ref T13544. These flags change the behavior of the "arc lint" subprocess.
I believe there is no reason to ever use "arc diff --lintall". If you want to find extra warnings to fix, you can use "arc lint --lintall" to express this intent.
Use of "arc diff --only-new" almost certainly means your linters are raising messages at "error" severity which should instead be raised at "warning" severity. If you only care about fixing a particular type of error in changed code, it should be raised as a "warning". The correct remedy is to adjust the severity, not use "--only-new", which is a very broad, slow, complicated hammer.
Test Plan: Searched for "lintall" and "only-new" in this workflow. These flags still exist in "arc lint", but may be changed in the future. Generated this change.
Maniphest Tasks: T13544
Differential Revision: https://secure.phabricator.com/D21298
Summary:
Ref T13544. Long ago, "arc diff" started prompting the user to provide "excuses" when they submitted changes with failing lint or unit tests.
At the time, "arc" was generally more heavy-handed and the review workflow had fewer points where it provided feedback about lint and test issues.
As the workflow has evolved, there is now significantly more feedback (promotion behavior from Draft in Differential, warnings on "arc land", etc). These days, these prompts feel archaic and like they're just getting in the way.
When lint/unit have Harbormaster-triggered components, this prompt is also too early (since Harbormaster tests may fail or raise lint messages later). A modern version of this would look more like putting revisions in some kind of locked state until authors explain issues. It's possible that's worth building, but I'd like to see more interest in it. I suspect this feature is largely just a "nag" feature these days with few benefits.
Test Plan: Grepped for "advice", "excuse", "handleServerMessage", "sendMessage", "getSkipExcuse", "getErrorExcuse", got no hits. Generated this revision.
Maniphest Tasks: T13544
Differential Revision: https://secure.phabricator.com/D21297
Summary:
Ref T13544. This flag was introduced in D1385 (2012) as part of a workflow which no longer exists. I can't recall anyone ever reporting an issue which involves its use and believe it is likely unused. It's not obvious to me why someone would use it in modern "arc".
(The same goal can be accomplished with "--message-file ...", although this requires more steps.)
Test Plan: Grepped for "use-commit-message" and "getCommitMessageFromCommit", ran "arc diff".
Maniphest Tasks: T13544
Differential Revision: https://secure.phabricator.com/D21296
Summary: Ref T13546. Add support for short "-x" flags and repeatable "--say moo --say quack" flags.
Test Plan: In future changes, used "arc diff -m" and "arc land --onto ... --onto ...".
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21310
Summary: Ref T13546. Future "arc land" workflows use this element to provide "hint" or "next step" suggestions to make error resolution easier.
Test Plan: Ran future "arc land" workflows, saw tips like "use --revision to do such-and-such" or "add these files to .gitignore".
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21309
Summary:
Ref T13546. Several substeps in the new "arc land" flow benefit from this. For example:
- When prompting "land revisions you don't own?", it's used to show authors.
- When prompting "land revisions in the wrong state?", it's used to show the current states.
Test Plan: Ran future "arc land" workflows, got relevant contextual information via this mechanism.
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21308
Summary:
Ref T13546. If we try to resolve several hundred hardpoint queries which execute subprocesses, we can currently hit system limits.
For now, limit resolution to 32 simultaneous futures. In the future, this should switch to `FuturePool` and possibly become more nuanced.
Test Plan: In a future change, ran `arc land --into-empty ...` to land thousands of commits. Before change, got a "proc_open()" error when launching too many simultaneous subprocesses. After change, this "worked".
Maniphest Tasks: T13546
Differential Revision: https://secure.phabricator.com/D21307
Summary: See PHI1765. You can find version information with "arc version". Remove this undocumented special-case.
Test Plan:
- Ran `arc --version`, no longer got a workflow API exception.
- Searched the documentation for references to `arc --version` (instead of `arc version`), found none.
Differential Revision: https://secure.phabricator.com/D21306
Summary: This simplifies merging a list-of-lists, which is a common pattern when asking a list of extensions to each provide some kind of list of items.
Test Plan: Used elsewhere in Piledriver.
Differential Revision: https://secure.phabricator.com/D21294
Summary:
With some frequency, code wants to assert that some "$o->m()" returns a list of objects of type X, possibly with unique values for some "getKey()"-style method result.
Existing checks via `PhutilTypeMap` and `assert_instances_of()` aren't quite powerful enough to do this while producing an easily understandable error state. We want to know that the error arose from a call to "$o->m()" in particular.
Test Plan: Currently used elsewhere, in Piledriver code.
Differential Revision: https://secure.phabricator.com/D21293
Summary: This is a path argument, and shell completion should suggest tabs.
Test Plan: Typed "arc liberate s<tab>", got path suggestions.
Differential Revision: https://secure.phabricator.com/D21292
Summary:
Ref T13543. Currently, the `cpplint` tests do not function because `cpplint` is passed a path which does not end in a suffix it recognizes.
Change the tempfile / path code to pass `linter path/to/example.c`-style linters a path they expect.
Then, correct some older code which was playing it fast-and-loose with "null" vs "0".
Test Plan: Ran `arc unit --everything`, got a clean bill of health on all the linters I have installed. (This is probably not all tests, since I have only a subset of linters installed locally that we have code for.)
Maniphest Tasks: T13543
Differential Revision: https://secure.phabricator.com/D21291
Summary:
Ref T13543. If a file ends in spaces and no newline, we'll emit a message suggesting removal of the spaces. This will effectively remove the line, but the code will then attempt to highlight text within the line.
Prior to D21044 this continued without raising an error and produced a reasonable result, but it now fatals. Insetad, don't try to highlight lines which no longer exist.
Test Plan: See T13543 for details.
Maniphest Tasks: T13543
Differential Revision: https://secure.phabricator.com/D21290
Summary:
Ref T13543. The "cpplint.py" script may emit messages on line 0, but Arcanist doesn't accept these messages.
This is a small piece of a whole set of broader issues, but stop the bleeding for now.
Test Plan:
- Ran `arc lint example.h` on a file with no `#ifndef` guard, and `cpplint` configured.
- Cpplint raised a message at line "0".
- Before change: arc choked when trying to render this.
- After change: arc survives rendering.
Maniphest Tasks: T13543
Differential Revision: https://secure.phabricator.com/D21289
Summary:
See PHI1735. "ConduitEngine" was once a future pool, but this has moved to "HardpointEngine". This class may no longer make much sense.
In Phacility code, "bin/host upload" depends on using the Uploader, which needs a "ConduitEngine", not a "ConduitClient". This workflow may use asymmetric key signing, which "ConduitEngine" does not support.
To unblock PHI1735, provide glue code between "Client" and "Engine". But a "more correct" change is probably removal of "Engine".
Test Plan:
- Ran `bin/host upload`, uploaded files (with additional changes to wrap the Client).
- Created this revision.
Differential Revision: https://secure.phabricator.com/D21260
Summary: Fixes T13533. This is a narrow, fragile API for a particular Kerberos use case on one install.
Test Plan:
- Set a non-scalar key, got an exception.
- Set <"duck", "quack">, got an exception from cURL that the value was invalid.
- Set a bunch of made-up options to arbitrary values, no errors. cURL accepts anything so there's nothing we can do about this.
- Set `CURLOPT_NOBODY` and saw the request behavior change, demonstrating that the call can produce effects.
Maniphest Tasks: T13533
Differential Revision: https://secure.phabricator.com/D21251
Summary: Ref T13520. In unusual cases where there are no changes in a changeset list (e.g., empty commits) we can fatal when trying to iterate over an empty list of vectors.
Test Plan:
- Created an empty commit.
- Used "git show | pbcopy" to create a diff from it.
- Viewed it in the web UI.
- Before: fatal when iterating on `null`.
- After: clean page.
Maniphest Tasks: T13520
Differential Revision: https://secure.phabricator.com/D21221
Summary:
Ref T13528. For consistency with other commands ("arc upload", "arc diff"), support a "--browse" flag to "arc paste".
Support "--input" as a more robust alternative to `x | y` (see T6996).
Test Plan: Ran `arc paste --browse --input X`, got a new paste in a browser window. Ran other variations of flags and parameters.
Maniphest Tasks: T13528
Differential Revision: https://secure.phabricator.com/D21203
Summary:
Ref T13528. Provide a "--browse" flag to open files after they are uploaded.
Update the code to use modern query strategies.
This impacts the output of "--json", which was just raw "file.info" output before and could not represent the same path being passed several times (`arc upload X X`).
Test Plan: Ran `arc upload` with `--browse` and `--json`.
Maniphest Tasks: T13528
Differential Revision: https://secure.phabricator.com/D21202
Summary:
Ref T13528. If we start a subprocess that immediately exits and then write to it, we can get a broken pipe error.
Recent versions of PHP appear to raise this as an actual warning, and recent changes upgrade the warning to a runtime exception.
I can't find any way to tell if the RuntimeException is a broken pipe or something else, except by examining the text of the error string.
At least for now, treat this like a "blocked pipe" condition. Since the subprocess has exited and the bytes didn't write, this should generally be reasonable.
Test Plan:
- Viewed a file in Paste with an extension that Pygments does not have a lexer for.
- This causes Pygments to exit immediately with an "unrecognized lexer" error. This closes the pipe, and the next write will fail with a broken pipe error.
- Before patch: fatal on broken pipe.
- After patch: clean resolution of the future and error condition.
Maniphest Tasks: T13528
Differential Revision: https://secure.phabricator.com/D21199
Summary:
Ref T13528. When you call `$future->resolve()`, we currently guarantee it is resolved by calling `FutureIterator->resolveAll()`.
`resolveAll()` does not actually "resolve()" futures: it guarantees that they are ready to "resolve()", but does not actually call "resolve()".
In particular, this means it does not throw exceptions.
This can lead to a case where a Future has "resolve()" called directly (e.g., via a FutureProxy), uses "FutureIterator" to resolve itself, throws an exception inside "FutureIterator", the exception is captured and attached to the Futuer, then the outer future tries to access results. This fails since it's out-of-order.
This can happen in practice with syntax highlighting futures, which may proxy pygments futures.
Instead, "resolveAll()" before testing for exaceptions.
Test Plan:
- Locally, tried to highlight a Paste with an unrecognized lexer extension using Pygments.
- Before patch: fatal when trying to access results of a Future with no results (because it has an exception instead).
- After patch: resolution throws the held exception properly.
- (See also next change.)
Maniphest Tasks: T13528
Differential Revision: https://secure.phabricator.com/D21198
Summary: Fixes T13527. Some versions of PHP strictly require that we pass a string value, and reject "stringlike" objects (objects which implement "__toString()").
Test Plan: Ran unit test, although this is somewhat aspirational because my local PHP version isn't affected.
Maniphest Tasks: T13527
Differential Revision: https://secure.phabricator.com/D21193
Summary:
See D21190. The ".gitattributes" approach fails when ".gitattributes" is in a subdirectory (or global). These are probably unusual cases, but at least one is known in the wild.
Instead:
- Restore the ":(attr:filter=lfs)" test, which seems to be the fastest accurate test available in modern Git.
- If the test fails, assume the repository is not LFS. This only impacts users running very old versions of Git.
Test Plan:
- In LFS and non-LFS repositories, created diffs. Saw correct detection again.
- Broke the command on purpose, saw LFS detection conclude "no LFS", but not fail disastrously.
Subscribers: ptarjan
Differential Revision: https://secure.phabricator.com/D21192
Summary:
See PHI1718. See also <https://discourse.phabricator-community.org/t/arc-diff-fails-due-to-git-cmd-fails/3680/>.
Currently, `arc diff` detects Git LFS with `git ls-files -z -- ':(attr:filter=lfs)'` magic. This is an accurate test, but does not work on older Git.
Try a simpler, dumber test and see if that will work. If this also has issues, we can try this stuff:
- do version detection;
- pipe the whole tree to `git check-attr`;
- try a command like `git lfs ls-files` instead, which is probably a wrapper on one of these other commands.
Test Plan:
- In a non-LFS repository, ran "arc diff" and saw the repository detect as non-LFS.
- In an LFS repository, ran "arc diff" and saw the repository detect as LFS.
Differential Revision: https://secure.phabricator.com/D21190
Summary:
Ref T13520. Replace "FileTree" with a "VectorTree" that does roughly the same thing. The major goals are:
- Compress trees which contain sequences of child directories with no sibilings.
- Build hierarchies of paths where path components may include renames.
This is approximately similar to "FileTree" and similar to client logic in the new paths panel.
Test Plan: See next change.
Maniphest Tasks: T13520
Differential Revision: https://secure.phabricator.com/D21182
Summary:
Ref T13518. The result format of this call changed in PHP 7.4, which causes us to emit "-1" matches because "-1" survives `array_filter()`.
Filter results in a way that should survive both result formats.
Test Plan: Ran `arc unit --everything` under PHP 7.4.
Maniphest Tasks: T13518
Differential Revision: https://secure.phabricator.com/D21173
Summary: Ref T13507. In Phabricator, we perform a specific "Accept-Encoding: gzip" setup test and want to manually decode the result. Allow callers to disable handling of "Content-Encoding".
Test Plan: Ran all Phabricator setup checks.
Maniphest Tasks: T13507
Differential Revision: https://secure.phabricator.com/D21121
Summary:
Ref T13507. For various messy reasons we can't blindly assume the server supports "gzip" -- but if the server tells us it does, we're on firmer ground.
If the server returns an "X-Conduit-Capabilities: gzip" header and we have compression support locally, compress subsequent requests.
This restores D21073, which was reverted by D21076.
Test Plan: With a gzip-asserting server, added debugging code and ran various "arc" commands. Saw the 2nd..Nth calls hit compression code.
Maniphest Tasks: T13507
Differential Revision: https://secure.phabricator.com/D21119
Summary:
Ref T13507. Add "Accept-Encoding: gzip" to requests if we can decompress responses.
When we receive a compressed response, decompress it.
Test Plan: Added debugging code, ran some commands, saw smaller payloads over the wire and inline decompression.
Maniphest Tasks: T13507
Differential Revision: https://secure.phabricator.com/D21118
Summary:
Ref T13507. Currently, this function is a bit conservative about what it encodes, and passing it a string of binary garbage may result in an output which is not valid UTF8.
This could be refined somewhat, since it's less than ideal if the input has valid UTF8. The ideal behavior for byte sequences where all bytes are larger than 0x7F is probably a variation of "phutil_utf8ize()" that replaces bytes with "<0xXX>" instead of the Unicode error glyph.
For now, just err on the side of mangling.
Test Plan: Dumped various binary payloads in the new gzip setup check, saw sensible output in the web UI.
Maniphest Tasks: T13507
Differential Revision: https://secure.phabricator.com/D21117
Summary: Ref T13490. This code is reachable from Phabricator binaries; only inject the legacy stuff if we're in an Arcanist stack.
Test Plan: Ran `bin/conduit help` from `phabricator/`.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21113
Summary: Ref T13490. More-or-less straightforward upgrade to modern calls.
Test Plan: Created and viewed pastes.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21104
Summary:
Ref T13490. One of the biggest issues users are hitting in modern "arc" is that workflows don't appear in "arc help" until they're updated.
Since there's still some work to do and gluing them in isn't terribly difficult, at least get things connected for now.
Test Plan: Ran "arc help", "arc help diff".
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21101
Summary: Ref T13490. These workflows don't seem worth the maintenance cost, see T13488 for discussion.
Test Plan: Grepped for methods which looked like they might only be called by these flows, only dug up the backout stuff.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21100
Summary: Ref T13490. The "RevisionRef" is currently based on "differential.query" data, but all calls other than the hash-based lookup can move to "differential.revision.search".
Test Plan: Ran revision workflows like `arc inspect` and `arc browse`.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21099
Summary: Ref T13490. I recently made "tsprintf()" more strict, but we do sometimes use "%d" and this is reasonable to support.
Test Plan: Ran "arc branch".
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21098
Summary:
Ref T13490. This is mostly straightforward.
- Drop "--show" in favor of "--as -".
- Drop support for 4+ year old "file.info" API.
- Use modern stream-to-disk support so we get a real progress bar and don't need to buffer files into memory.
Test Plan: Downloaded various files, including large files.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21097
Summary: Ref T13490. Moves "arc amend" to Toolsets with modern ref/hardpoint code.
Test Plan: Ran "arc amend --show", "--revision", etc. Hit all the prompts and errors, probably?
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21095
Summary:
Ref T13490. I'm continuing to move toward modernizing "amend", which needs to load some objects by symbol at top level.
Add an API to support this.
Test Plan: Added an API caller to "arc inspect", ran it and hit the new code. Things seemed to work.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21094
Summary: Ref T13490. Make terminal strings work more like HTML does in Phabricator, and make it easier to display refs.
Test Plan: Added some display code, ran `arc inspect` to hit it, saw a nice ref printed.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21093
Summary:
Ref T13490. I'm attempting to update "arc amend", but it needs to fetch revision authors to raise an "amending a revision you don't own" error.
Support user-symbol resolution.
Along the way, this introduces some infrastructure for abstracting away iteration over a multi-page Conduit result set.
Test Plan:
- Ran "arc inspect ..." for various "user(...)" queries.
- Set page size to 3 and issued a general query, saw the future page it away properly.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21092
Summary:
Ref T13490.
- Support resolution of revision symbols into revision objects.
- Align commit inspection better against commit symbols.
- Make "arc inspect --explore" recursively load all object hardpoints.
- Add a "commit message" hardpoint to "RevisionRef".
Test Plan: Used "arc inspect" to resolve commits and revisions. Used "--explore" to see the whole tree.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21091
Summary:
Ref T13490. Frequently, we know the symbolic name of a commit (like "master") but need the immutable identifier for it (the commit hash).
Provide a Ref and Query for doing this lookup.
Test Plan: Ran `arc inspect symbol(...)` with various symbols, saw appropriate resolutions, nulls, or errors.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21090
Summary:
Ref T13490. "ExecFuture" can raise a "CommandException" when the subprocess fails. In most cases, this is desirable, since we usually do not expect subprocesses to fail.
Currently, "execx()" (which raises these exceptions) is the synchronous default (with the more verbose "exec_manual()" as an alternative if you expect the command may return an error code), but the Future variation has no way to hint that external resolution should raise an exception if the process exits with an error.
Since the "HardpointEngine" yield construct can resolve external futures, add a mode for this to simplify implementing "HardpointQuery" classes a bit. Without this, they either need to retain "$futures" and manually call "resolvex()", or check the "$err" result and throw some other exception, both of which are low-value boilerplate (queries that want to do this still can, of course).
This is basically like providing a hint that the futures should be resolved with "resolvex()" instead of "resolve()".
Also, make this the default behavior of a new "$api->newFuture()" wrapper.
Test Plan: Intentionally broke a command in a HardpointQuery, got a sensible exception automatically during external future resolution.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21089
Summary: Ref T13490. There's more `array(...)` happening in this API than necessary. Sugar it slightly.
Test Plan: Grepped for "loadHardpoints()", ran a couple workflows.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21088
Summary: Ref T13490. A few older "defineHardpoint()" calls are sticking around. They no longer have callers; get rid of them.
Test Plan: Grepped for this symbol, no hits.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21087
Summary:
Ref T13490. These workflows are aliases of one another, which is a little silly, but currently all identify as "arc feature" in "arc help".
Straighten that out, at least.
Test Plan: Ran "arc help", "arc branch".
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21086
Summary:
Ref T13490. Bring "arc prompts" from "wilds" and hook it into the prompt in "arc shell-complete".
See D21069. Fix an issue where the shell hook tested for a path other than the path it writes to.
Test Plan: Ran "arc shell-complete" with no hook and got a prompt. Shell completed things. Ran "arc prompts shell-complete".
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21085
Summary: Ref T11968. "RefQuery" is now "HardpointEngine". "HardpointLoader" is now "HardpointQuery".
Test Plan: Grepped for affected symbols.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21083
Summary:
Ref T11968. "arc browse", "arc branch", and "arc diff" currently may execute into the RefQuery engine. Reroute them to the HardpointEngine.
This removes older-generation "Ref" objects and renames the replacement "RefPro" objects to "Ref".
Test Plan: Ran "arc branch", "arc browse <various things>", "arc diff", searched for affected symbols.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21082
Summary: Ref T11968. Merge the modern hardpoint queries and refs for the "browse" workflow as "pro" variations.
Test Plan: Ran `arc inspect browse(...)` for objects, paths, commits, and revisions.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21080
Summary:
Ref T11968. Continue bringing modern yield-based hardpoint code into "master" in the parallel "Pro" classtree.
Adds "working-copy(commit-hash)" as an inspectable ref.
Test Plan: Inspected working copy refs, saw them resolve revisions by commit hash and commit message.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21079
Summary:
Ref T11968. Inches toward the new ref/hardpoint code by introducing the modern refs as "RefPro" objects and supporting an "arc inspect <object>" to load objects and hardpoints.
This doesn't impact any existing runtime behavior.
Test Plan: Ran "arc inspect [--all] commit(...)", got hardpoint queries and yield-based data fetching.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21078
Summary:
Depends on D21074. Ref T13490. Ref T11968. Before toolsets, Arcanist has a "WorkingCopyIdentity" object. After toolsets, it has a "WorkingCopy" object.
Most workflows don't access either one, so make a slightly-breaking API change to simplify the transition.
- "getWorkingCopy()" now returns a modern object (a "WorkingCopy").
- "getWorkingCopyIdentity()" now returns an older object (a "WorkingCopyIdentity").
This isn't backward-compatible, but out-of-date code should get an explicit failure with clear resolution steps.
Test Plan: Ran "arc lint", "arc branch", and a few other commands. Grepped for "getWorkingCopy()".
Maniphest Tasks: T13490, T11968
Differential Revision: https://secure.phabricator.com/D21075
Summary:
Ref T11968. Ref T13490. These are the workflows which currently use the intermediate-level hardpoint code (which made hardpoints formal, but didn't do the yield stuff).
Move toward updating them by doing some basic bookkeeping, with a few compatibility adjustments to the parent Workflow class.
Test Plan: Ran "arc branch" and "arc browse" with various arguments.
Maniphest Tasks: T13490, T11968
Differential Revision: https://secure.phabricator.com/D21074
Summary:
This reverts commit 7e25288f49. See T13507 for discussion, shortly.
This needs to be switched to capability detection because some server configurations can not actually accept these requests.
Test Plan: Straight revert.
Differential Revision: https://secure.phabricator.com/D21076
Summary: Ref T13507. Enable compression in the body of Conduit requests if we have client support for it; we expect the server should always support it.
Test Plan: Created this revision, ran random "arc" commands that use Conduit. Added debugging code, saw payload size drop.
Maniphest Tasks: T13507
Differential Revision: https://secure.phabricator.com/D21073
Summary:
Depends on D21071. Ref T11968. Currently, "ConduitEngine" tries to lightly parallelize futures. This was a compromise when the initial "hardpoint" change didn't plan to pursue real request paralleization.
Now that the newer hardpoint change does, we don't need onboard resolution in ConduitEngine. Throw it away.
When the engine is supposed to resolve a future, it now just resolves that future on its own. This should be functionally identical to the previous behavior, except that it may be slower.
(In practice, because HTTP futures are backed by an internal cURL request pool, this proably has little effect anywhere. Moving to modern hardpoints will make performance no worse than it was prior to this change, in any case.)
Test Plan: Ran various modern "arc" commands.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21072
Summary:
Depends on D21070. Ref T11968. Adds "yield"-aware query classes for parallelizing service calls.
These will replace the similar (but not yield-aware) "Hardpoint" classes introduced previously. This is an API change but most of the old classes still exist and still do the same thing, just with more "yield" statements.
This just adds a bunch of new code with no callers and no API updates.
Test Plan: See future changes.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21071
Summary:
Ref T11968.
- Allow "WorkingCopy" objects to maintain an API object and update callers ("get...()" instead of "new...()").
- Always generate a WorkingCopy object and a RepositoryAPI object.
Currently, code has to look like this:
```
$working_copy = ...
if ($working_copy) {
$repository_api = ...
if ($repository_api [instanceof ... ]) {
```
This is clunky. There's also no reason some "arc" commands can't run outside a VCS working directory without special-casing how they interact with the filesystem.
Conceptually, model the filesystem as a trivial VCS (which stores exactly one commit, always amends onto it, and discards history). Provide a trivial WorkingCopy and API for it.
(This change isn't terribly interesting on its own, but chips away at landing the new Hardpoint infrastructure.)
Test Plan: Ran `arc version`, `arc upgrade`.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21070
Summary:
Depends on D21067. Ref T13492. Converting unit tests to be readable exposed this error in the grammar.
Normally, "grammar_rule" rules emit a standalone Node. In this case, the bottom-level grammar rule is a collection of trivial rules and the callers configure the Node. Wrap the bottom-level rule in a configuration rule so the node is configured correctly and consistently, in exactly one place.
Test Plan: Ran unit tests.
Maniphest Tasks: T13492
Differential Revision: https://secure.phabricator.com/D21068
Summary:
Depends on D21066. Ref T13492. The switch of unit test data to stable/readable output exposed this bug in parsing of variadic calls: some nodes are not given types properly.
Fix the parser and update the test.
Test Plan: Ran the test, which now works.
Maniphest Tasks: T13492
Differential Revision: https://secure.phabricator.com/D21067
Summary:
Depends on D21065. Ref T13492. Swap existing "expect" blocks from unstable, unreadable JSON to readable, stable trees.
(There are two "INVALID TYPE" outputs which this update effectively detects and which future changes correct.)
Test Plan: Ran "arc unit --everything", got a clean build.
Maniphest Tasks: T13492
Differential Revision: https://secure.phabricator.com/D21066
Summary:
Depends on D21064. Ref T13492. Earlier, see D17819. This is essentially the same change, although I inlined the token stream into the node list.
This intentionally breaks most tests since it just has the new "expect" generator; the next change will fix them by swapping the test bodies.
Test Plan: Ran "arc unit --everything" after the next change (which fixes all the tests), got a clean build. This change on its own fails all existing XHPAST tests since the block formats don't match.
Maniphest Tasks: T13492
Differential Revision: https://secure.phabricator.com/D21065
Summary: Ref T13490. The new Arcanist runtime supports workflow signal handling, but Phage isn't quite able to make use of it. Clean up the last few pieces so it can work.
Test Plan: Ran "phage", hit ^C, got status information.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21060
Summary: Ref T11968. Phage has another "sustained pool of Futures" use case, and needs some slight adjustments after Future API changes.
Test Plan: Ran `bin/phage status ...`, got a clean result instead of a JSON decoding failure.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21058
Summary:
Ref T11968. "FutureIterator" recently became non-rewindable, and starting a Future twice is now an error.
This complicates a handful of use cases where a mostly-constant pool of futures is maintained over a long period of time, notably in daemon overseers and repository pull daemons.
They previously relied on being able to do "new FutureIterator($futures)" to continue resolution of a list of futures from any state. This no longer works quite like it used to, since Futures generally may not belong to more than one iterator now (and this property is desirable).
Introduce "FuturePool", which maintains exactly one iterator but manages the small amount of glue around adding and removing Futures from it, destroying it if the pool empties, and rebuilding it if the pool fills.
Test Plan: See next change, which makes use of this.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21053
Summary:
Depends on D21051. Ref T13504. Ref T13209. This very old Mercurial code uses "X=Y hg ..." on Linux and "set X=Y & hg ..." on Windows.
The latter construct no longer works because we bypass the shell. The former construct is obsolete.
Additionaly, delete some ancient "branch merge" code which has no callers.
Test Plan: Created a diff in a Mercurial repository on Linux. I minimally vetted this on Windows since I don't have a "hg + Windows" environment at the moment.
Maniphest Tasks: T13504, T13209
Differential Revision: https://secure.phabricator.com/D21052
Summary:
Ref T13504. Ref T13209. We currently use "escapeshellarg()" under Windows, which destructively replaces "%" symbols with spaces. This is wrong, breaks general behavior with "git log --format=...", and has led to a lot of weird workarounds in `arc` code where we don't escape arguments we should be escaping to tiptoe around issues with "%".
Now that execution occurs via "bypass_shell", we can safely (probably?) escape things, although programs are still apparently free to parse the command string however they want.
Test Plan: Added unit tests, ran them under Mac and Windows, got clean results. Ran "arc version" on Mac and Windows.
Maniphest Tasks: T13504, T13209
Differential Revision: https://secure.phabricator.com/D21051
Summary:
Ref T13504. The API has changed here slightly, and if you run "arc version" without "arcanist/" being a Git working copy, it currently fatals in a misleading way.
Instead, reach the error properly.
Test Plan: Ran "arc version" after moving aside ".git/", got a helpful error message instead of a confusing "call on null" exception.
Maniphest Tasks: T13504
Differential Revision: https://secure.phabricator.com/D21050
Summary:
Fixes T13504. This fixes three issues:
# In ExecFuture, "proc_open()" on an invalid binary could fail with an unconditional exception.
# In ExecPassthru, "proc_open()" on an invalid binary could fail with an unconditional exception.
# In "arc browse", "start <url>" does not work when the shell is bypassed.
In (1) and (2), the desired behavior is to fail with an exit code which is sometimes upgraded to an exception depending on calling convention.
Issue (1) most commonly manifested as "find" failing when run via "cmd.exe".
Issue (2) most commonly manifested as "arc browse" failing.
Issue (3) was entangled with issue (2).
In cases (1) and (2), assume "proc_open()" failures under Windows are because of bad binaries and treat them like bogus commands on Linux/Mac.
In case (3), use "cmd /c start" instead of "start" as a default browser on Windows.
Test Plan:
- On Windows, did mime type detection in cmd.exe. Before patch: proc_open() exception in "find". After patch: clean (albeit not terribly useful) detection.
- On Windows, did "arc browse ...". Before patch: proc_open() exception in "start". After patch: clean browser execution.
Maniphest Tasks: T13504
Differential Revision: https://secure.phabricator.com/D21047
Summary:
Ref T11968.
- Rekeying natural lists of futures creates more problems than it solves. For now, don't try to deal with this.
- The scope of the future autokey was incorrect (per subclass, not per process).
Test Plan: Ran various future-oriented flows on new HardpointEngine code.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21046
Summary:
Depends on D21036. Ref T11968. Ref T13177. Currently, each Future integrates separately with ServiceProfiler, but much of the code is similar.
Move integration to the base class and lift up most of the implementation details.
Test Plan: Ran `arc diff --trace`, saw sensible output.
Maniphest Tasks: T13177, T11968
Differential Revision: https://secure.phabricator.com/D21038
Summary: Depends on D21035. Ref T11968. This allows a "FutureIterator" to hold futures which are blocked because of unresolved dependencies, and makes the resolution process more structured.
Test Plan: Ran unit tests, created this revision.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21036
Summary: Depends on D21033. Ref T11968. This is just an incremental step in modernizing Future and making it more robust. Currently, subclasses are expected to write directly to `$this->result`, but this isn't consistent with how modern classes generally work.
Test Plan: Ran unit tests, created this revision.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21034
Summary:
Depends on D21032. Ref T11968. Currently, "Future" and "FutureIterator" can both resolve futures.
Treat "Future->resolve()" as sugar on resolving an iterator of size 1.
Test Plan: Ran tests, created this revision.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21033
Summary:
Ref T11968. This future-level parameter has no nontrivial callers and makes the "fate of FutureGraph" changes more difficult.
Callers that are genuinely interested in this behavior can wrap the Future in a FutureIterator and use "setUpdateInterval()" to get the same behavior.
Test Plan: Grepped for "resolve()" and "resolvex()", updated callers.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21031
Summary:
One of our users ran into an issue where they were running `arc diff` in
a subdirectory and it so happened that in that directoy there wasn't any `lfs`
files so the repo was marked as `$is_lfs == false`. Lets allow `arc` to be run
from anywhere.
Test Plan: Internally this patch worked
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D21045
Summary:
Ref T13499. Currently, we throw most language errors as exceptions, but this list isn't exhaustive and errors we don't specifically make more severe are allowed to slip through.
This is generally undesirable, particularly in the case of "Undefined index:" errors. See T13499 for a specific case where this caused behavior to be more difficult to understand and diagnose than it should have been.
Make this the default behavior instead, except for "E_USER" errors, which we never expect to arise from first-party code.
This may be slightly too aggressive, but future changes can selectively reduce the severity of some types of errors if problems arise.
Test Plan:
- Executed code which intentionally accessed an undefined index, got an exception.
- Poked around Phabricator and Arcanist without any further issues cropping up, but I don't have a good way to develop confidence that the "reduced severity" list should genuinely be empty.
Maniphest Tasks: T13499
Differential Revision: https://secure.phabricator.com/D21044
Summary: Fixes T13498. Currently, `arc diff` may invoke the file upload flow using an older variation of the API. Both the modern and fallback clients build an Engine, so switching to the modern API is trivial.
Test Plan:
- Created a local commit with a 1MB binary file.
- Ran `arc diff --only`.
- Before patch: fatal on old API call.
- After patch: clean upload.
Maniphest Tasks: T13498
Differential Revision: https://secure.phabricator.com/D21043
Summary:
I'm working on a fairly large monorepo using phabricator and we are
running into a `git-lfs` issue. Every single `arc diff` we have to wait
many seconds while all our lfs files are walked before the push goes through.
I chatted with the `git-lfs` folks in
https://github.com/git-lfs/git-lfs/issues/4076 and they suggested this workflow
change. I tested it on our repo and it does indeed seem to fix the issue. Three
things were change:
1. The remote url is named
2. Use branches instead of tags
3. Do a `git fetch`
(Also, hello after all this time! Hope you're all doing well over at phacility.)
Test Plan:
Before
```
$ arc diff
Linting...
LINT OKAY No lint problems.
Running unit tests...
UNIT OKAY No unit test failures.
PUSH STAGING Pushing changes to staging area...
Uploading LFS objects: 100% (8211/8211), 1.0 GB | 0 B/s, done.
Enumerating objects: 31, done.
Counting objects: 100% (31/31), done.
Delta compression using up to 12 threads
Compressing objects: 100% (21/21), done.
Writing objects: 100% (24/24), 1.77 KiB | 8.00 KiB/s, done.
Total 24 (delta 19), reused 0 (delta 0)
remote: Resolving deltas: 100% (19/19), completed with 6 local objects.
To github.com:robinhoodmarkets/web-staging.git
* [new tag] 2581578b59b6341c1023377fd8741814e89fcd0c -> phabricator/base/427095
* [new tag] 7d9be50fbca590f15742d8cc20edd4f082dfbb3b -> phabricator/diff/427095
Updated an existing Differential revision:
Revision URI: https://phabricator.robinhood.com/D135442
Included changes:
M __shared/tools/arcanist-js/src/workflow/RHArcanistDiffWorkflow.php
```
After
```
$ arc diff
Linting...
LINT OKAY No lint problems.
Running unit tests...
UNIT OKAY No unit test failures.
PUSH STAGING Pushing changes to staging area...
Enumerating objects: 15, done.
Counting objects: 100% (15/15), done.
Delta compression using up to 12 threads
Compressing objects: 100% (7/7), done.
Writing objects: 100% (8/8), 638 bytes | 2.00 KiB/s, done.
Total 8 (delta 6), reused 0 (delta 0)
remote: Resolving deltas: 100% (6/6), completed with 6 local objects.
To github.com:robinhoodmarkets/web-staging.git
* [new branch] 2581578b59b6341c1023377fd8741814e89fcd0c -> phabricator/base/427105
* [new branch] 274cc3a25eb7856291c1d4c078b28b76af8a43b2 -> phabricator/diff/427105
Updated an existing Differential revision:
Revision URI: https://phabricator.robinhood.com/D135442
Included changes:
M __shared/tools/arcanist-js/src/workflow/RHArcanistDiffWorkflow.php
```
(notice the lack of `Uploading LFS objects`)
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D21041
Summary:
Depends on D21031. Ref T11968. This clears some lint with the word "Future", but is just a minor cleanup patch not really related to the main goal in that task.
We currently warn on `preg_quote()` with no second parameter, but this is valid and correct:
```
preg_match('('.preg_quote($x).')', ...)
```
This is the modern recommended style for this sort of expression, so the warning is often a false positive; drop it.
The "__CLASS__" warning looks for hard-coded class names in strings, but currently looks for them as a word anywhere in the string.
This is often a false positive and sometimes makes error messages or exceptions untranslatable. For example, translators can't do anything with this:
> Filesystem path "%s" does not exist.
...if it's written as:
> %s path "%s" does not exist.
...just because it happens to appear in the class "Filesystem". Some other classes, including "Future", suffer from this.
Even when the detection is correct, it's clunky and a mistake here (failing to update the class name in an error message) is unlikely to ever do any real damage.
Test Plan: Ran unit tests and `arc lint`.
Maniphest Tasks: T11968
Differential Revision: https://secure.phabricator.com/D21032
Summary:
See PHI1663. If "phabricator.uri" is not configured, we try to fall back to "default", but doesn't have a modern config specification and fails.
Instead, read "default" as a raw config value to preserve compatible behavior. My intent is to eventually remove it.
Test Plan:
In a directory with no "phabricator.uri" config available, ran `echo {} | arc call-conduit conduit.ping`. After the patch, got a reasonable error instead of a fatal.
In a directory with "default" configured but not "phabricator.uri", ran the same command. After the patch, got a ping.
Maniphest Tasks: T13497
Differential Revision: https://secure.phabricator.com/D21039
Summary: Ref T13490. This largely makes "arc upload" work, although the Future stuff is still a bit wonky. See T11968.
Test Plan: Ran "arc upload README.md", got an upload.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21030
Summary: Ref T13490. This fixes one bug in D21025 (loading relative to "arcanist/" rather than the parent directory) and improves behavior when a library fails to load (we prompt the user to continue).
Test Plan:
- Ran `arc list` with bad library specifications, got a sensible error and an option to continue.
- Ran `arc list` with a library specification that existed next to "arcanist/", got a library load instead of an error.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21029
Summary: Ref T13490. This makes the "phage" toolset work properly and updates external library behavior to match the "classic" behavior, except that "--library" is now supported.
Test Plan: Ran "phage remote" workflows.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21025
Summary:
Ref T13490. There's one simple "implode()" order issue here, and one slightly more complex one that uses "DIRECTORY_SEPARATOR" as glue.
Add test coverage for this, update the lint check to detect constants used as glue, and fix the callsites.
Test Plan:
- Added a failing test and made it pass.
- Ran `arc lint --everything` and looked for remaining implode warnings, found none.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21024
Summary: Ref T13490. Allows "arc upgrade" to run through the new Toolsets infrastructure.
Test Plan: Ran "arc upgrade", although you can't upgrade feature branches so this can't be entirely tested until it hits "master".
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21006
Summary: Depends on D21004. Ref T13490. The "wilds" branch removed a bunch of dead features but some of them were resurrected by the merge. I don't think any of these are desirable to retain, so purge them all again, even in classic mode.
Test Plan: Grepped for affected symbols.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21005
Summary:
Depends on D21001. Ref T13490.
- Require "--" to terminate arguments when running noninteractively.
- Don't correct spelling when running noninteractively (we still suggest corrections).
- Remove old "phage" wrapper script.
- Fix an issue where "argv" could implicitly generate a parseable "--argv" flag.
- Accept "--" as an argument terminator even when there is no argument list.
- Update some strings to use more modern quoting style.
- Make workflows decline to handle signals by default.
- Allow "arc weld" to weld non-UTF8 files together.
Test Plan: Ran various commands. Welded non-UTF8 files.
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21002
Summary: Ref T13490. Update the "weld" workflow and the "anoid" workflow. Incorporates D20938.
Test Plan: Ran "arc weld". Ran "arc anoid".
Maniphest Tasks: T13490
Differential Revision: https://secure.phabricator.com/D21001
Summary: Depends on D20996. Ref T13395. Ports the updated version of this workflow from "experimental".
Test Plan: Ran `arc shell-complete` to install the hook, then shell-completed commands.
Maniphest Tasks: T13395
Differential Revision: https://secure.phabricator.com/D20997
Summary: Depends on D20993. Ref T13395. Merges the more-modern "alias" out of "experimental".
Test Plan: Defined and invoked aliases. This has some rough edges: notably, no easy list/delete flow.
Maniphest Tasks: T13395
Differential Revision: https://secure.phabricator.com/D20996
Summary: Ref T13395. Depends on D20991. Make "arc liberate" run as a toolset command, not a classic command.
Test Plan: Ran "arc liberate"; created this diff.
Maniphest Tasks: T13395
Differential Revision: https://secure.phabricator.com/D20992
Summary: Ref T13395. Make "arc help" run as a toolset command, not a classic command.
Test Plan: Ran "arc help".
Maniphest Tasks: T13395
Differential Revision: https://secure.phabricator.com/D20991
Summary:
Depends on D20988. Ref T13395. Ref T13098. Ref T13203.
This brings all the "toolsets" code into "master". We try to run commands as toolsets commands first, then fall back to older code.
Since the "toolsets" class tree is mostly parallel to the older class tree, this isn't completely broken. Currently, all commands fall back.
Test Plan: Created this diff, ran various other commands. But this is probably a long shot from finished.
Maniphest Tasks: T13395, T13203, T13098
Differential Revision: https://secure.phabricator.com/D20990
Summary:
Ref T13395. Merge a lot of stuff which doesn't break existing workflows:
- Merge a UTF8 fix for "cmd.exe" on Windows.
- Merge minor changes to JSON linters.
- Merge some shell completion stuff.
- Merge some "arc anoid" fixes.
- Merge various Windows improvements to unit tests which interact with processes / the filesystem.
- Merge some other Windows path fixes.
- Merge a UTF8 character class fix.
- Merge script initialization.
- Merge unit test support scripts.
- Merge some initialization code.
- Merge Windows stdout/stderr-as-files code.
- Merge a bunch of code for making exec tests work on Windows.
- Merge more Windows unit test fixes.
- Merge "continue on failure" mode when loading symbols.
- Merge "GPC" order CLI fixes.
Test Plan: Ran `arc unit --everything`; created this change. There's likely some less-than-perfect code here.
Maniphest Tasks: T13395
Differential Revision: https://secure.phabricator.com/D20988
Summary: Fixes T13489. See that task for discussion.
Test Plan: Ran `arc lint` and `arc lint --output xml`. Faked a missing extension, ran `arc lint` (no problems) and `arc lint --output xml` (sensible error message).
Maniphest Tasks: T13489
Differential Revision: https://secure.phabricator.com/D20989
Summary:
Depends on D20986. Ref T13395. This //mostly// collapses the entire "experimental" branch into "master".
I plan to change the "Ref/Hardpoint" pattern to become future oriented, but this is more steps forward than sideways.
Test Plan: Ran various `arc` workflows.
Maniphest Tasks: T13395
Differential Revision: https://secure.phabricator.com/D20987
Summary: Ref T13395. Collapse changes to the linter workflow from "experimental" into "master".
Test Plan: Ran `arc lint`. Noted flag changes in changelog.
Maniphest Tasks: T13395
Differential Revision: https://secure.phabricator.com/D20986
Summary: Ref T13010. This brings the "--draft" flag (and a handful of related flags) to "master" from "experimental".
Test Plan: Ran "arc diff". This code has been in use on "experimental" for a long time.
Maniphest Tasks: T13010
Differential Revision: https://secure.phabricator.com/D20985
Summary: Ref T13395. Fixes a file-locking test and removes a logging test that's more trouble than it's worth.
Test Plan: Ran "arc unit --everything" locally, got a clean bill of health.
Maniphest Tasks: T13395
Differential Revision: https://secure.phabricator.com/D20984
Summary: Ref T13395. Currently, "phage" is required for various cluster operations. Bring the working code out of "experimental". This isn't the final form; "wilds" has a fancier version.
Test Plan: Ran phage workflows against the cluster.
Maniphest Tasks: T13395
Differential Revision: https://secure.phabricator.com/D20982
Summary: Ref T13395. Moves all remaining code in "libphutil/" into "arcanist/".
Test Plan: Ran various arc workflows, although this probably has some remaining rough edges.
Maniphest Tasks: T13395
Differential Revision: https://secure.phabricator.com/D20980
Summary:
Ref T13395. This code relies on PhutilHTML, but that's moving to Phabricator.
It has no callers in libphutil or Arcanist.
Test Plan: Looked for callers.
Maniphest Tasks: T13395
Differential Revision: https://secure.phabricator.com/D20978
Summary: Ref T13481. Some older versions of Git appear to not support "--" in these commands. Just drop it. This can lead to ambiguous results with certain obviously-silly remote names, but doesn't appear to lead to anything dangerous.
Test Plan: Will followup with user on ancient Git.
Maniphest Tasks: T13481
Differential Revision: https://secure.phabricator.com/D20952
Summary: Fixes T13481. When identifying the URI for a remote, fall back from "git remote get-url" to "git ls-remote --get-url" to "git config remote.<name>.url" based on command output and version tests.
Test Plan: Ran `arc land --hold`, rigged the subcommands to fail to try all three fallbacks, ran `arc land --hold --remote asdfasdf` to get an explicit failure.
Maniphest Tasks: T13481
Differential Revision: https://secure.phabricator.com/D20950
Summary: Fixes T13458. Emit an explicit message when "arc close-revision --finalize" bails out because the revision is not "Accepted".
Test Plan: Ran `arc close-revision [--finalize] ...` on various revisions, saw more clear messaging.
Maniphest Tasks: T13458
Differential Revision: https://secure.phabricator.com/D20915
Summary:
Ref T13432. Git has a "diff.suppressBlankEmpty" config option which makes it emit nonstandard diffs with trimmed trailing whitespace on unchanged blank lines.
Currently, we don't parse these diffs correctly. Even if we do in the future, emitting a more standard diff is desirable.
Explicitly disable this option when executing "git diff" so we build more standard diffs.
Test Plan:
- Configured this option.
- Modified a file with a blank line in it without changing the blank line, got this goofy display diff:
{F6985234}
- Applied patch, rediffed the same change, saw "-c diff.suppressBlankEmpty" in "--trace" output and got this sensible diff:
{F6985235}
Maniphest Tasks: T13432
Differential Revision: https://secure.phabricator.com/D20877
Summary: Ref T13434. Since "git p4 submit" gets more complicated when submitting merges, and empty merges (as with "--no-ff") seem to vanish, and it's not clear this is desirable or useful anyway, just make the "merge" strategy an explicit error with Perforce remotes.
Test Plan: Ran "arc land --merge ..." in a Git/Perforce repository, got an explicit error. Ran "arc land --squash ...", got existing working behavior.
Maniphest Tasks: T13434
Differential Revision: https://secure.phabricator.com/D20871
Summary:
Fixes T10650. It's valid to `arc land --remote origin --onto master` without first fetching that ref. If we can't find a local ref for a specified remote branch, try to fetch it before giving up.
(In the long run, this should be valid even if the remote branch does not exist at all and the user intends to create it -- see T12876 -- but this is a step toward that.)
Test Plan:
- Ran `rm .git/refs/remotes/origin/master`, then landed into "master".
- Before: "arc land" bailed out immediately.
- After: "arc land" fetches the missing ref.
```
$ arc land
TARGET Landing onto "master", the default target under git.
REMOTE Using remote "origin", the default remote under Git.
TARGET No local ref exists for branch "master" in remote "origin", attempting fetch...
FETCHED Fetched branch "master" from remote "origin".
...
```
Maniphest Tasks: T10650
Differential Revision: https://secure.phabricator.com/D20870
Summary: Ref T13434. Move some git-engine-specific handling of "arc land" arguments into the Git engine. This prepares to handle Perforce workflows.
Test Plan: Will "arc land" this change.
Maniphest Tasks: T13434
Differential Revision: https://secure.phabricator.com/D20867
Summary:
Ref T13428. Historically, "implode()" accepts arguments in either order. PHP 7.4+ warns about glue not being first.
Add a lint check when the second parameter is a static scalar, implying it is the glue parameter.
Test Plan: Ran unit tests. See next change.
Maniphest Tasks: T13428
Differential Revision: https://secure.phabricator.com/D20857
Summary:
Fixes T10321. Some reasonable but less-common workflows involve running `arc land` from a detached HEAD state.
When users do this, we currently try to delete the raw hash as though it were a branch during cleanup. Instead, detect if the thing we're thinking about deleting is a branch or not, and just leave it alone if it isn't.
Test Plan:
- Ran `git checkout <some hash>`, then `arc land --revision <some revision>`.
- Before, everything worked but cleanup tried to `git branch -D <some hash>`.
- After, everything worked and cleanup skipped branch deletion.
Maniphest Tasks: T10321
Differential Revision: https://secure.phabricator.com/D20786
Summary:
The current code assumes git-svn is always working from a remote called
`trunk`, but if the repository is initialized without the `-T` option it
will instead be called `git-svn`, and if `--prefix` is used (which is
set by default to `origin/` in Git 2+) the remote name will have the
specified prefix as well.
Instead, look at the `fetch` target refspec set in the git-svn config.
Fixes T13293.
Test Plan:
`arc land` without errors (or manually creating a `trunk` branch) from a
checkout made with Git 2.18.0 (verified this manually on a non-`-T`
checkout as well).
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Maniphest Tasks: T13293
Differential Revision: https://secure.phabricator.com/D19681
Summary:
Fixes T6854. The current format for `lint-test` files is somewhat inflexible and does not allow us to make assertions regarding the code or name of the linter messages (of class `ArcanistLintMessage`) that are raised. Specifically, the `${severity}:${line}:${char}` format is hardcoded in `ArcanistLinterTestCase`. In this diff, I extend the this format to achieve the following goals:
- Allow for the lint message code and name to be specified. Specifically, the full format is `${severity}:${line}:${char}:${code}:${name}`.
- Make all fields optional. `error:3:` will match any and all errors occuring on line 3.
- Provide more useful output when assertions fail. Specifically, output //all// lint messages that are missing and/or surplus. Previously, only the first lint message was output.
Test Plan: `arc unit`
Reviewers: #blessed_reviewers, epriestley, chad
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T6854
Differential Revision: https://secure.phabricator.com/D11176
Summary:
I am writing a proxy linter that can be used to wrap any `ArcanistExternalLinter` and execute all commands within a Docker container (see [[https://github.com/freelancer/flarc/blob/master/src/lint/linter/ArcanistDockerContainerLinterProxy.php |`ArcanistDockerContainerLinterProxy`]] from [[https://github.com/freelancer/flarc | `flarc`]]). In order for `ArcanistDockerContainerLinterProxy` to behave like the `ArcanistExternalLinter` that is being proxied, `final` needs to be removed from some methods.
I figured this was reasonable to submit upstream as a similar change ({D19630}) was previously accepted.
Test Plan: N/A
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D19730
Summary:
I am writing a proxy linter that can be used to wrap any `ArcanistExternalLinter` and execute all commands within a Docker container (see [[https://github.com/freelancer/flarc/blob/master/src/lint/linter/ArcanistDockerContainerLinterProxy.php |`ArcanistDockerContainerLinterProxy`]] from [[https://github.com/freelancer/flarc | `flarc`]]). In order for `ArcanistDockerContainerLinterProxy` to behave like the `ArcanistExternalLinter` that is being proxied, `final` needs to be removed from some methods.
I figured this was reasonable to submit upstream as a similar change ({D19630}) was previously accepted.
Test Plan: N/A
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D20514
Summary: The recently-added, build-plan-behavior-aware check here does all of its own prompting, so we should skip the other prompting if it doesn't throw.
Test Plan: Will `arc land` something sketchy sooner or later.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20494
Summary:
See PHI1238. This "," in the regex can only make the lint binding fail if there is no "," in the version output string. In modern versions of Pylint, there is (apparently) no comma in the version string. Remove it.
See also <https://discourse.phabricator-community.org/t/arcanistpylintlinter-version-regex-issue/2688>
Test Plan:
```
$ pip install pylint
Traceback (most recent call last):
File "/Users/epriestley/Library/Python/2.7/bin/pip", line 6, in <module>
from pip._internal import main
ImportError: No module named pip._internal
```
¯\_(ツ)_/¯
Reviewers: amckinley, joshuaspence
Reviewed By: joshuaspence
Differential Revision: https://secure.phabricator.com/D20505
Summary:
Since I'm in here for PHI1083:
- Add some "--" so we get correct behavior when you have a file named "master", a branch named "README.txt", etc.
- Stop using "%C" unnecessarily.
- Fix some untranslatable strings.
Test Plan: Ran `arc patch` a couple of times, ran the variations of `git` commands to catch anything weird with "--" handling.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20254
Summary:
See PHI1083. Previously, see PHI648 and D19475.
When you apply a submodule patch in Git, it leaves you with a working copy that has the "submodule pointer" dirtied but the actual submodule untouched:
```
$ git status
On branch ...
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
modified: philter
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: philter (new commits)
```
So, if you're applying `D123` and `submodule/` was previously pointed at commit "A" but `D123` updates it to point at commit "B", you get this after `git apply ...`:
- Git index says "submodule/ = B".
- On disk, "submodule/ = A".
Now, if you `git add --all` or `git commit --all`, git picks up the "change" on disk as an intended modification of the submodule. This puts the submodule back to "A" and overwrites/undoes the "pointer" update that's trying to make it point to "B".
To avoid this, update submodules after applying the patch.
Also, every time we modify the working copy, just update submodules.
Test Plan:
- Add a submodule in branch "B1", pointed at commit "A".
- Branch to create branch "B2". Update the submodule to point at commit "B". Commit this and `arc diff` it.
- Go back to "B1". Use `arc patch D...` to apply the revision you just created.
- Before change:
- "arc patch" applies the submodule change, so "pointer = B", "disk = A".
- "arc patch" runs "git commit --all", which looks at disk and sets "pointer = A".
- This isn't a change, so we fail with an empty commit.
- After change:
- "arc patch" applies the submodule change, so "pointer = B", "disk = A".
- "arc patch" updates submodules, so "pointer = B", "disk = B".
- "arc patch" runs "git commit --all", which now has a change, and commits "submodule = B".
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20253
Summary:
Ref T13258. This makes "arc land" respect the new "Warn When Landing" behavior.
This will only work if you have very up-to-date APIs. Just fall back to the older code if the new API calls fail.
Test Plan: Ran `arc land` on a revision with builds in various states and with the different "Warn When Landing" behaviors. Saw appropriate warnings.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13258
Differential Revision: https://secure.phabricator.com/D20236
Summary:
See PHI1104. The older "differential.querydiffs" method includes the entire raw diff text for all the diffs associated with a revision in its response, but we: only care about the most recent diff; and don't care about the text at all.
For reasonably large changes with several updates, this can be significantly slow.
We can get this same information more efficiently from the modern "differential.diff.search", since D19386 (April 2018). The only trick is that we need a "revisionPHID", which we don't have on hand.
For now, just fetch the revision PHID. In the future, we can likely make adjustments so that we have the revision PHID already by the time we get here.
This may slow down the normal case very slightly (since we now do two calls instead of one), but it speeds up the bad cases dramatically.
Test Plan:
Ran `arc diff` to update a change in a local repository. `var_dump()`'d the old and new algorithm results, saw the same outcome.
Used `arc diff --trace` on an update to a change to verify that `differential.diff.search` is called but `differential.querydiffs` is not.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D20221
Summary:
Ref T13161. See D20181. This allows the intraline highlighter to accept new ">" and "<" spans and apply a different style for them.
The input pattern is `list<segment>`. Each segment is `pair<wild kind, int byte_length>`, i.e. wrap the next `byte_length` bytes in a span of kind `kind`.
Before this change, the possible kinds of segements are `0` (no intraline diff, do not highlight) or `1` (intraline diff, highlight in bright color).
D20181 adds `<` (depth decreased) and `>` (depth increased). These are like `1`, but add a different class so the UI can handle them differently.
Test Plan: See D20181.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13161
Differential Revision: https://secure.phabricator.com/D20182
Summary:
Ref T13249. See PHI810. We currently warn you when //all// reviewers are away, but not when only some reviewers are away.
This makes some amount of sense under the "anyone can accept anything" rules we sort of recommend, but a lot of installs realistically have tons of owner/package rules now.
Instead, if any reviewers are away, show the user exactly who is away and until when, then make sure they don't want to make any adjustments.
(We can do a better job of this after the toolsets change when we can use the new APIs, but this is an easy fix for now.)
Test Plan: Created a revision with multiple reviewers, either some or all of whom were away. Got appropriate output and prompt behavior.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13249
Differential Revision: https://secure.phabricator.com/D20172
Summary: Ref T13249. We currently allow `if (function_exists('X')) { X(); }` but not `if (defined('X')) { X; }`. Allow the latter.
Test Plan: See D20145, which linted clean with this patch in place.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13249
Differential Revision: https://secure.phabricator.com/D20146
Summary: See 30 prior patches. This is a fatal in PHP7, let's just hunt these down.
Test Plan: Ran unit tests. See next diff for results.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D19931
Summary:
See PHI904. Ref T13210. Ref T13209. Currently, we have an `hg cat` construction which attempts to pass a literal `%p` to Mercurial. This fails because you can't pass `%` through `%s` outside of `wilds`.
It also uses `%C` to pass a list of file paths. This is broadly unsafe and can cause command execution if you modify a file named, e.g., `; rm -rf xyz` or similar. I think it would be difficult to turn this into an attack but it's fairly bad. This dates from D5144 in 2013.
Test Plan: With this patch, created D19757 which has valid binary data (see F5962134).
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13210, T13209
Differential Revision: https://secure.phabricator.com/D19758
Summary:
Ref T13098. See PHI858. If you write this at the end of a message in `arc diff`:
```
Subscribers:
#projectname
# NEW DIFFERENTIAL REVISION
# Describe the changes in this new revision.
# ...
```
...we'll currently eat the `#projectname` as an instructional comment, even if it is followed by an empty line.
Instead, stop eating stuff once we hit the first empty line. (We escape empty lines in comments already.)
After T13098 I'll maybe adjust this to use a more explicit instruction escape, like `##`, since there's no reason we're bound to `#`.
Test Plan: Added a unit test and made it pass.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13098
Differential Revision: https://secure.phabricator.com/D19639
Summary: I'm not sure if the upstream will be interested in this change, but we are writing a linter which works by running an external command on the entire repository. This can't be done with `ArcanistExternalLinter` at the moment, which meant that we ended up copy-pasting most of `ArcanistFutureLinter`. This would be a lot easier if we could override `willLintPaths` and `didLintPaths`, but these methods are currently marked as `final`. An alternative solution would be some sort of `ArcanistLinter::transformPath` method.
Test Plan: N/A
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: faulconbridge, Korvin
Differential Revision: https://secure.phabricator.com/D19630
Summary:
Ref T13187. See PHI838. If two hunks are separated by 7 lines of context, we can render them as either:
```lang=diff
+ Hunk A
Context 1
Context 2
Context 3
Context 4
Context 5
Context 6
Context 7
+ Hunk B
```
...or:
```lang=diff
+ Hunk A
Context 1
Context 2
Context 3
@@ +1,2 -3,4 @@
Context 5
Context 6
Context 7
+ Hunk B
```
Since we get the same number of output lines either way and the first one is more human-readable, we picked that one.
However, `diff -u` does the second one. Since human-readability is probably less important than compatibility, change the behavior to be more similar to `diff -u`.
Test Plan: Added unit tests for the edge cases with default parameters (6 context lines, 7 context lines) and made them pass.
Reviewers: amckinley
Maniphest Tasks: T13187
Differential Revision: https://secure.phabricator.com/D19603
Summary:
See https://discourse.phabricator-community.org/t/arc-not-supporting-git-2-17-1/.
When treating it a large file binary, we try to get the "old" and "new" content using `git ls-tree` and `cat-file`.
If the file is new or deleted, there is no old file, so we try to work with filename `null`.
Under git < 2.17.1, that gets treated as `git ls-tree -- .`, which falls in the next condition under "no such path".
In git 2.18, etc, this is an error.
Explicitly bail out if there is no filename.
Test Plan: Add a new, large (>4Mb) file, arc diff.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D19513
Summary: See PHI718. Modern Mercurial with the "evolve" extension enabled may emit this field.
Test Plan: As D19262.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D19498
Summary:
Ref T13151. See PHI648. With `arc patch --nobranch`, we update submodules a little too early.
I //believe// it is safe to just update them a little later, after the intermediate branch management logic runs.
Test Plan: Ran `arc patch --nobranch`, saw submodule update run later. Not 100% sure this doesn't cause weird issues, but I can't anticipate any.
Reviewers: amckinley, jmeador
Reviewed By: jmeador
Maniphest Tasks: T13151
Differential Revision: https://secure.phabricator.com/D19475
Summary:
The Configuration Manager is supported by ArcanistUnitTestEngine but not support by the ArcanistConfigurationDrivenUnitTestEngine.
Added the configuration manager as one of the initially set properties of an ArcUnitTestEngine created by the ArcanistConfigurationDrivenTestEngine
Test Plan: Ran arc unit against a project without the change, verified the Configuration was none. Added this change and ran again and verified it was set
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D19465
Summary:
Ref T13137. See PHI592. When you have a diff with 600MB of videos, we want to bail out of diff generation early (as soon as we realize what we're dealing with), not build an 800MB text diff in memory and then throw it away.
Support bailout //during// diff generation once we realize we're in over our heads.
This is approximate, but since the limit is fairly large (512KB by default) it isn't too important to be precise.
Test Plan: Rigged some callers to set various byte limits, generated diffs including diffs with large binaries. Got an appropriate diff or exeception depending on how low the limit was.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13137
Differential Revision: https://secure.phabricator.com/D19444
Summary: Fixes T1246. See PHI637. See T13137. Computers have gotten a bit faster so we can probably bump this up a little and see if it causes problems. This is `O(N^2)` so the this should be less than twice as expensive in the worst case.
Test Plan:
Created a diff affecting characters on a very long line separated by more than 80 but fewer than 100 characters, got a good intraline diff out of it:
{F5605162}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T1246
Differential Revision: https://secure.phabricator.com/D19442
Summary:
Ref T13130. I wasn't able make this much better, but it looks like this is about ~20% faster on my system.
This kind of thing is somewhat difficult to micro-optimize because XHProf tends to over-estimate the cost of function calls. In XHProf, this looks much much faster than the old version (~100% faster) but the actual cost of `bin/conduit call --method differential.getrawdiff` hasn't improved that much. Still, it seems consistently faster across multiple runs.
Test Plan:
- Pulled binary diffs over Conduit with `bin/conduit call --method differential.getrawdiff`.
- Verified that they are byte-for-byte identical with the pre-change diffs, and look like they're ~20% faster.
- Profiled the differences and saw a far more dramatic improvement, but I believe XHProf is exaggerating the effect of this change because it tends to overestimate function call cost.
- Ran unit tests (from D19407), got byte-for-byte identical output under both 32bit and 64bit mode.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13130
Differential Revision: https://secure.phabricator.com/D19408
Summary:
Ref T13130. I want to take a crack at improving performance here, but two possible approaches (inlining the actual encoding; using integers if they're big enough) aren't easy to test right now.
Restructure the tests so they can support these kinds of refactoring.
The "32bit" and "64bit" modes currently do the same thing, but I expect to introduce introduce separate encoding pathways in a future change, if the profiler says it actually helps.
(I'll hold this and everything that comes after it until I make meaningful performance improvements.)
Test Plan: Ran `arc unit`, got passes on tests.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13130
Differential Revision: https://secure.phabricator.com/D19407
Summary:
Filenames are last in `git status --porcelain=2` lines; they
are not escaped in any way, despite the fields being
whitespace-delimited. `explode` thus happily chops apart filenames
with spaces in them, causing later git operations to operate only on
the filename up to the first space.
Split the lines into the right number of elements -- in all cases,
this is one more than the index we're using, since filenames come last.
Test Plan:
Altering a file with a space in its path, and running `arc diff -a`.
Added tests.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D19389
Summary: Ref T13110. We now degrade very large changes and I'm not convinced any user ever entered "n" at this prompt.
Test Plan: Ran `arc diff` to create this very revision.
Maniphest Tasks: T13110
Differential Revision: https://secure.phabricator.com/D19299
Summary: See PHI527. Ref T13116. The `--revision` flag currently fails if the argument is in the form `D123` instead of `123`. Normalize monogram arguments.
Test Plan: Ran `arc patch --revision Dxxx`, `arc patch --revision xxx`, `arc patch --revision xxx --diff yyy`, `arc patch`; got good behavior on the good ones and sensible error messages on the other ones.
Maniphest Tasks: T13116
Differential Revision: https://secure.phabricator.com/D19292
Summary:
STDERR output with `%`s in it could cause:
```
ERROR 2: fprintf(): Too few arguments at [/usr/local/arcanist/src/workflow/ArcanistFeatureWorkflow.php:170]
```
Test Plan: Untested.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D19261
Summary:
In Go 1.10 the output for tests was changed to have also a "(cached)" mode in
addition to the normal timing info printed. This is on by default. This adds
support for parsing these lines instead of erroring out on the regex.
Test Plan: Have a unit test included, and will continue to poke at it locally.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D19161
Summary: Fixes T8236. I played around with a lot of variations of this but in the end it felt like the simple version was best.
Test Plan: Ran `arc weld a.txt b.txt`, observed very robust fusion of materials.
Maniphest Tasks: T8236
Differential Revision: https://secure.phabricator.com/D19081
Summary:
Fixes T13061. Both `arc lint` and `arc unit` accept an `--everything` flag, but the documentation isn't quite clear about what these flags do.
They act as though every //tracked// file in the repository (`git ls-files`, `hg manifest`, or `svn list -R`) is included in the argument list.
They do not lint/test ignored files (and I think almost all users would be very surprised if they did).
They also don't lint/test untracked files (files you have not yet used `git add`, `svn add`, or `hg add` on). This is slightly more contentious but we have good reasons for doing it (e.g., `git ls-files` often outperforms `find .` by a large margin) and I believe users very rarely use `--everything` in a situation where they have untracked files. The only real exception I can come up with is linter configuration/development, as in PHI343, and it seems okay to have a slightly surprising behvaior here.
Make the documentation more clear about what is in scope.
We could also rename these to `--nearly-everything` or whatever, but I think the name is probably clear enough given current information about how confusing this is (specifically: only rarely, in unusual cases).
Test Plan:
- Grepped for documentation about these flags.
- Ran `arc help lint`, `arc help unit`, `arc unit --everything x`, `arc lint --everything x` and read all the new messages.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13061
Differential Revision: https://secure.phabricator.com/D18989
Summary:
Fixes T8768. See PHI294. See that task for more details.
Git, Mercurial, `diff`, and `patch` have conspired to make things weird. To correctly handle files with spaces in the way everything else does and expects, we need to emit semantic trailing whitespace literals.
Test Plan:
- Created a file with spaces in it in a Mercurial repositroy, committed it, diffed it into a revision.
- Used `arc patch` to apply the change to a clean copy of the repository.
- Before patch: Mercurial incorrectly creates a file named `X`, not a file named `X Y.txt`.
- After patch: `arc patch` commit is identical to genuine commit.
- Also added test coverage. The other general behaviors here are fairly well covered already.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T8768
Differential Revision: https://secure.phabricator.com/D18869
Summary:
Currently, `arc` on `git` uses the following commands to examine the
state of the working tree and history; example times for a no-op diff in a
165k-file working tree are also shown:
```
1) git diff --no-ext-diff --no-textconv --submodule=short --raw 'fb062d4ecce5d9c1786b7bfc8a0dedf6b11fdd96' --
= 1,722,514 us
2a) git diff --no-ext-diff --no-textconv --submodule=short --raw 'HEAD' --
= 1,715,507 us
2b) git ls-files --others --exclude-standard
= 2,359,202 us
3) git diff-files --name-only
= 1,333,274 us
```
Steps (2a) and (2b) are run concurrently; this results in a total elapsed
wallclock time of approximately 5.4 seconds. This is inefficient -- all four of
the above steps must both load the index and examine the working copy, which may
be slow operations when large repositories are used. Additionally, none of the
effort of those stat calls on the working tree, or load time of the index, is
shared across the processes.
Step (1) is called from `getCommitRangeStatus`, which was split out in D4095; it
is currently never called on its own, only ever from `getWorkingCopyStatus`,
where it it combined with `getUncommittedStatus`. The current behavior of the
method is to return the set of changes //either// in local commits //or//
uncommitted in the working tree, which duplicates work that
`getUncommittedStatus` is intended to do. Changing the behavior of this method
(in Git, and other VCSes) to only examine _committed_ status seems both inline
with the name of the method and the original description of it in D4095 -- and
also serves to make it much faster, as it is an operation that need not inspect
the working tree at all.
Steps (2a), (2b), and (3) attempt to gather the state of the working copy, and
as such are all I/O bound but must examine nearly identical data. For git
2.11.0 and higher, we can instead rely on the machine-parseable `git status
--porcelain=2` format, which provides the information from all of these commands
at once. It also allows additional performance improvements, as `git status`
has been the focus of several optimizations in the latest versions of git (the
untracked cache and fsmonitor services, for instance), which are not available
in the lower-level `diff`, `ls-files`, and `diff-files` commands.
This has the added benefit of fixing a bug noticed in T9455, in that uncommitted
or unstaged changes in modules can now be detected, regardless of if they also
have changed their base commit. It further resolves a bug where `.gitmodules`
appeared to have unstaged changes, when in reality the unstaged changes were in
submodules elsewhere in the tree.
For backwards compatibility with versions of git < 2.11.0, the old code is left
in place. It is possible that the simpler output from v1 of `git status
--porcelain` would also suffice for some of the above benefits, but the payoff
of parsing yet another format is deemed insufficient; users wishing improved
performance should simply upgrade `git`.
Alltogether, these result in the following, for a no-op diff in a
165k-working-file tree:
```
1) git diff --no-ext-diff --no-textconv --submodule=short --raw 'fb062d4ecce5d9c1786b7bfc8a0dedf6b11fdd96' HEAD --
= 9,227 us
2) git status --porcelain=2 -z
= 739,964 us
```
...for a total of 749ms, an improvement of 4.7s.
Depends on D18841.
Test Plan: Existing tests.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D18842
Summary:
This adds tests that detail the current behavior of `arc` in
the presence of `git` submodules.
Test Plan: No behavior change; wrote the tests such that they pass.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D18841
Summary:
See PHI261. Currently "arc land" shows every build staus (passed, failed, building, etc) as yellow. Intended behavior is that passed builds are green, failed builds are red, and so on.
This is because of an unintended API change a while ago in D16356. Since the only impact was a cosmetic color issue, this escaped notice until now.
Additionally, try to use the modern `harbormaster.build.search` if it is available.
Test Plan:
- Ran `arc land` with running builds, got reasonable coloration.
- Faked the new method not being available, still got sensible behavior from the old method.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18837
Summary:
Most users, if they have gone through the trouble of
accepting the auto-fixes, are most likely going to want to take those
changes and attempt to land with them. Assuming "Y" for this prompt
streamlines for the more likely flow.
Test Plan: `arc lint`
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D18824
Summary: Fixes T10233. See PHI231. Users sometimes believe this warning is a bug and/or don't understand how they're supposed to resolve it.
Test Plan: Ran `arc land` on a revision in "Changes Planned", got a sensible prompt. Ran `arc land` on a revision in another non-accepted state, got more or less the old prompt.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T10233
Differential Revision: https://secure.phabricator.com/D18807
Summary: See PHI191. This is a rehash of an earlier fix, but we didn't have a test case for this half yet.
Test Plan:
- Added a failing test, made it pass.
- Added a linter like the one in PHI191, ran it, got a valid lint result instead of an exception.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18759
Summary:
See PHI162. This corrects a couple more bugs:
- If the old file didn't end in a newline, we could end up printing two lines next to each other in the output.
- If the patch targeted "Line 6, character 1" instead of "line 5, character 3" in a file "1\n2\n3\n4\n5\n", we would fail to figure out what that meant when computing an offset because the last line has 0 characters on it.
Test Plan: Added failing unit tests, made them pass. Also tested with some fake linters similar to the ones described in PHI162.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18716
Summary: See PHI136. These are already optional on the server side in `HarbormasterBuildLintMessage`, and effectively mean "file-level issue", which is a bit niche but not unreasonable.
Test Plan: Checked that `HarbormasterBuildLintMessage` doesn't care if these keys exist, created this revision.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18711
Summary:
SKIP lines, for instance, often have no UserData; there is no
reason to display a content-less blank line.
Test Plan: `arc unit`
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D18632
Summary: Ref PHI48. If a patch removes all of the lines at the end of a file, we can get some array index errors.
Test Plan: Added failing test, made it pass.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18631
Summary:
Fixes T12981.
See new `arc lint` output: P2071
See new `arc unit` output: P2072
Test Plan: Ran `arc unit/lint/diff` and observed new error instead of a Conduit error
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12981
Differential Revision: https://secure.phabricator.com/D18603
Summary: Ref T9846. See PHI48. See D18538 for a similar fix. We can contract the suffix lines too much if, e.g, a newline after another newline is removed. Prevent contraction to fewer than 0 lines.
Test Plan: Added a failing test, made it pass.
Reviewers: chad
Reviewed By: chad
Subscribers: alexmv
Maniphest Tasks: T9846
Differential Revision: https://secure.phabricator.com/D18541
Summary: Ref T9846. See PHI48. For replacing text in the form "ABC" with "ABBC", the trimmer had a bug.
Test Plan: Added failing tests, fixed 'em.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9846
Differential Revision: https://secure.phabricator.com/D18538
Summary: Fixes T8291. See PHI52. This is papering over the real issue (T8298) but it's a 10-second patch so just improve things slightly for now.
Test Plan: Ran `arc version` locally; patch confirmed on a Windows system by an affected user.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T8291
Differential Revision: https://secure.phabricator.com/D18518
Summary: Fixes T9846. This restores the last missing feature, ANSI highlighting of diff sections.
Test Plan:
Added a mode so we can actually test this stuff, activated that mode, wrote unit tests.
Did a bunch of actual lint locally too and looked at it, all seemed sane.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9846
Differential Revision: https://secure.phabricator.com/D18512
Summary: Ref T9846. This was dropped when I refactored how things are rendered; restore it.
Test Plan: Added unit tests.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9846
Differential Revision: https://secure.phabricator.com/D18511
Summary:
Ref T9846. This rewrites the rendering algorithm in a mostly-compatible way and fixes the major issue.
- Includes test coverage for removing a newline, from T12765.
- Includes test coverage for mangling an XML tag, from T9846.
This omits two features, which I'll port forward separately:
- For one-line patches, highlighting the patched section.
- For zero-line patches, putting a little caret ("^") under the character where the warning occurred.
I'll restore these features in a followup change.
Test Plan: Ran unit tests, linted a few things.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9846
Differential Revision: https://secure.phabricator.com/D18510
Summary:
Ref T9846. Sometimes, a lint message says to replace "the big bad wolf" with "the huge bad wolf": that is, the original and replacement text are the same at the beginning, or the end, or both.
To make this easier for humans to understand, we want to just show that "big" is being replaced with "huge", not that the entire phrase is being replaced.
This logic currently happens inline in console rendering. Pull it out and cover it so a future change can simplify console rendering.
Test Plan: Ran unit tests.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9846
Differential Revision: https://secure.phabricator.com/D18509
Summary: Ref T9846. The algorithm here is fairly invovled, so lay down some test coverage before breaking it.
Test Plan: Ran tests.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9846
Differential Revision: https://secure.phabricator.com/D18508
Summary:
D13794 changed ArcanistPHPCloseTagXHPASTLinterRule to ignore inline HTML blocks, but selectDescendantsOfType returns an AASTNodeList (which always exists).
Instead, check that the count() of the node list is > 0.
empty.lint-test had to be changed, it wouldn't have been accepted had this rule not been broken before it was commited.
Added tests to cover ArcanistPHPCloseTagXHPASTLinterRule in the future.
Test Plan: `arc unit`
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D18271
Summary:
`PhutilConsole->confirm()` is vestigial as noted in efcd70c, as
`PhutilConsole` may be deprecated sometime in the future. `PhutilConsole`
was developed as a tool for T4281 but the feature has been removed
since.
Replace existing occurences of the `PhutilConsole->confirm()` pattern with
`phutil_console_confirm()`. There should be no change in functionality
since the two functions are interchangeable.
Test Plan: Manually tested by running `bin/arc lint`, `bin/arc diff --preview`, `bin/arc land --preview`.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, alexmv
Differential Revision: https://secure.phabricator.com/D18185
Summary:
Fixes T12815. During the last update to "arc land", some flags were disabled but remained in place in case we needed to retain them.
It now seems reasonably clear that we do not. The "rebase" and "merge" strategies for landing were replaced by a better "headless" strategy which seems to avoid the original issues, so these flags no longer do anything or reasonably could do anything.
`--delete-remote` is slightly more ambiguous (e.g., see T12650 and maybe others) but the only real use case is "git push = save changes".
Test Plan:
Ran `arc land --update-with-rebase`, was told the flag does not exist.
Grepped for affected flags/symbols.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12815
Differential Revision: https://secure.phabricator.com/D18108
Summary:
If the commit does not exist locally, aborting still leaves
the user checked out on the branch. In nearly all cases, all that is
necessary is a fetch -- but the branch must also be cleaned up. This
leads to the pattern of:
```
arc patch D12345
[...base commit does not exist...]
^C
git checkout master
git branch -D arcpatch-D12345
git fetch
arc patch D12345
```
Solve this common problem by simply trying to fetch once if the commit does not
exist locally.
Test Plan: Ran `arc patch` on a recent diff.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D17949
Summary: Ref T12655 - this change properly detects trailing spaces or tabs (or combinations of thereof) on end of lines.
Test Plan: Use Text lint with trailing whitespace rule on files with spaces, tabs or combinations of thereof. Should properly detect and fix all those.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, Itms
Maniphest Tasks: T12655
Differential Revision: https://secure.phabricator.com/D17814
Summary:
Ref T12651. Ran into these during D17799:
- Use `getStatusCode()` to put the actual status code into the message.
- If we fail but wrote an empty file to reserve the filename, clean it up.
Test Plan:
- Faked the error, `phlog()`'d the exception.
- Saw sensible exception message.
- Saw empty file get cleaned up.
Reviewers: chad, amckinley
Reviewed By: chad
Maniphest Tasks: T12651
Differential Revision: https://secure.phabricator.com/D17800
Summary: Fixes T12555.
Test Plan:
Added this class to the codebase and ran `arc liberate`:
```
<?php
class FooBar {
public static function doTheFoo() {
return 'foobar';
}
}
```
Ran `arc lint` and observed this warning:
```
Warning (XHP87) Class Not `abstract` Or `final`
This class is neither `final` nor `abstract`, and does not have a
docblock marking it `@concrete-extensible`.
1 <?php
2
>>> 3 class FooBar {
4 public static function doTheFoo() {
5 return "foobar";
6 }
```
Added a `final` modifier to `FooBar`'s declaration and observed the warning went away.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T12555
Differential Revision: https://secure.phabricator.com/D17787
Summary: Fixes T12464. Moves "arc upload" to SHA256 where applicable.
Test Plan: Ran `arc upload` against a server with D17620 twice, saw it skip the actual upload the second time.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12464
Differential Revision: https://secure.phabricator.com/D17622
Summary:
Ref T12464. This is similar to D17619 and prepares us to move to SHA256 in the client.
Note that it's fine if `arc` and Phabricator disagree about hashing algorithms. We don't really trust the client anyway, so if things are mismatched clients will just end up transferring a bit more data instead of getting to cheat when Phabricator already has copies of data.
Test Plan: Ran `arc upload`, got a clean upload.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T12464
Differential Revision: https://secure.phabricator.com/D17621
Summary: Fixes T8348. Just use normal HTTP GET to download files if the server is sufficiently modern.
Test Plan: Downloaded various files with `--as`, `--show`, large files, small files, old server, new server.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T8348
Differential Revision: https://secure.phabricator.com/D17614
Summary:
Because the character offset group is optional, the logic for dealing
with the previous index-based match object was more complex than it
needs to be. Switching to named groups makes this clearer.
As an example of how this was causing a problem, when the character
group *was* present (at index 3), it was being appending to the linter's
name in the call to ->setName(), resulting in confusing linter output.
We may have also been setting the character offset to the error code's
string, too, which is further nonsense.
Because we reliably capture the flake8 error code, I don't think there's
a need to append it to the linter's name at all now, so I removed that
part (so it's always just `flake8`), but it's not a problem to restore.
Lastly, I hoisted `$regex` out of the loop because it's a constant and
de-indenting it gave me enough room to continuing writing it all on one
line.
Test Plan: Tested primarily with flake8 3.3.0
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D17552
