Summary:
Ref T13098. Depends on D19675. This change is headed to the `untamed-wilds` branch.
This change prevents `arc` from running `arc diff`, so I'm copy/pasting it.
This change also completely breaks `arc`, but I'm just generally trying to do this rewrite step-by-step so we have at least a bit of context to refer to in the future.
The major change here is to turn both `arc` and `phage` into scripts which start an `ArcanistRuntime`. This runtime then decides which workflows (like "diff", "patch", or "remote") are available based on `$argv[0]`. This turns `arc` into more of a CLI tools platform: we can build `phage` on it, third parties can build `my-companion-tool-thing`, etc. But all the different entry points can share a lot of infrastructure like: `help`, `alias`, shell-complete, configuration, Conduit, prompting, `--trace`, extension infrastructure, and so on.
`ArcanistRuntime` is roughly a slightly more modern version of `scripts/arcanist.php`. That will be removed eventually, but not everything has ported yet.
Test Plan: This code basically doesn't run yet, although the next patch runs at least a little bit.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13098
Differential Revision: https://secure.phabricator.com/D19677
Summary:
Ref T13187. See PHI838. If two hunks are separated by 7 lines of context, we can render them as either:
```lang=diff
+ Hunk A
Context 1
Context 2
Context 3
Context 4
Context 5
Context 6
Context 7
+ Hunk B
```
...or:
```lang=diff
+ Hunk A
Context 1
Context 2
Context 3
@@ +1,2 -3,4 @@
Context 5
Context 6
Context 7
+ Hunk B
```
Since we get the same number of output lines either way and the first one is more human-readable, we picked that one.
However, `diff -u` does the second one. Since human-readability is probably less important than compatibility, change the behavior to be more similar to `diff -u`.
Test Plan: Added unit tests for the edge cases with default parameters (6 context lines, 7 context lines) and made them pass.
Reviewers: amckinley
Maniphest Tasks: T13187
Differential Revision: https://secure.phabricator.com/D19603
Summary:
See https://discourse.phabricator-community.org/t/arc-not-supporting-git-2-17-1/.
When treating it a large file binary, we try to get the "old" and "new" content using `git ls-tree` and `cat-file`.
If the file is new or deleted, there is no old file, so we try to work with filename `null`.
Under git < 2.17.1, that gets treated as `git ls-tree -- .`, which falls in the next condition under "no such path".
In git 2.18, etc, this is an error.
Explicitly bail out if there is no filename.
Test Plan: Add a new, large (>4Mb) file, arc diff.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D19513
Summary: See PHI718. Modern Mercurial with the "evolve" extension enabled may emit this field.
Test Plan: As D19262.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D19498
Summary:
Ref T13151. See PHI648. With `arc patch --nobranch`, we update submodules a little too early.
I //believe// it is safe to just update them a little later, after the intermediate branch management logic runs.
Test Plan: Ran `arc patch --nobranch`, saw submodule update run later. Not 100% sure this doesn't cause weird issues, but I can't anticipate any.
Reviewers: amckinley, jmeador
Reviewed By: jmeador
Maniphest Tasks: T13151
Differential Revision: https://secure.phabricator.com/D19475
Summary:
The Configuration Manager is supported by ArcanistUnitTestEngine but not support by the ArcanistConfigurationDrivenUnitTestEngine.
Added the configuration manager as one of the initially set properties of an ArcUnitTestEngine created by the ArcanistConfigurationDrivenTestEngine
Test Plan: Ran arc unit against a project without the change, verified the Configuration was none. Added this change and ran again and verified it was set
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D19465
Summary:
Ref T13137. See PHI592. When you have a diff with 600MB of videos, we want to bail out of diff generation early (as soon as we realize what we're dealing with), not build an 800MB text diff in memory and then throw it away.
Support bailout //during// diff generation once we realize we're in over our heads.
This is approximate, but since the limit is fairly large (512KB by default) it isn't too important to be precise.
Test Plan: Rigged some callers to set various byte limits, generated diffs including diffs with large binaries. Got an appropriate diff or exeception depending on how low the limit was.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13137
Differential Revision: https://secure.phabricator.com/D19444
Summary: Fixes T1246. See PHI637. See T13137. Computers have gotten a bit faster so we can probably bump this up a little and see if it causes problems. This is `O(N^2)` so the this should be less than twice as expensive in the worst case.
Test Plan:
Created a diff affecting characters on a very long line separated by more than 80 but fewer than 100 characters, got a good intraline diff out of it:
{F5605162}
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T1246
Differential Revision: https://secure.phabricator.com/D19442
Summary:
Ref T13130. I wasn't able make this much better, but it looks like this is about ~20% faster on my system.
This kind of thing is somewhat difficult to micro-optimize because XHProf tends to over-estimate the cost of function calls. In XHProf, this looks much much faster than the old version (~100% faster) but the actual cost of `bin/conduit call --method differential.getrawdiff` hasn't improved that much. Still, it seems consistently faster across multiple runs.
Test Plan:
- Pulled binary diffs over Conduit with `bin/conduit call --method differential.getrawdiff`.
- Verified that they are byte-for-byte identical with the pre-change diffs, and look like they're ~20% faster.
- Profiled the differences and saw a far more dramatic improvement, but I believe XHProf is exaggerating the effect of this change because it tends to overestimate function call cost.
- Ran unit tests (from D19407), got byte-for-byte identical output under both 32bit and 64bit mode.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13130
Differential Revision: https://secure.phabricator.com/D19408
Summary:
Ref T13130. I want to take a crack at improving performance here, but two possible approaches (inlining the actual encoding; using integers if they're big enough) aren't easy to test right now.
Restructure the tests so they can support these kinds of refactoring.
The "32bit" and "64bit" modes currently do the same thing, but I expect to introduce introduce separate encoding pathways in a future change, if the profiler says it actually helps.
(I'll hold this and everything that comes after it until I make meaningful performance improvements.)
Test Plan: Ran `arc unit`, got passes on tests.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13130
Differential Revision: https://secure.phabricator.com/D19407
Summary:
Filenames are last in `git status --porcelain=2` lines; they
are not escaped in any way, despite the fields being
whitespace-delimited. `explode` thus happily chops apart filenames
with spaces in them, causing later git operations to operate only on
the filename up to the first space.
Split the lines into the right number of elements -- in all cases,
this is one more than the index we're using, since filenames come last.
Test Plan:
Altering a file with a space in its path, and running `arc diff -a`.
Added tests.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D19389
Summary: Ref T13110. We now degrade very large changes and I'm not convinced any user ever entered "n" at this prompt.
Test Plan: Ran `arc diff` to create this very revision.
Maniphest Tasks: T13110
Differential Revision: https://secure.phabricator.com/D19299
Summary: See PHI527. Ref T13116. The `--revision` flag currently fails if the argument is in the form `D123` instead of `123`. Normalize monogram arguments.
Test Plan: Ran `arc patch --revision Dxxx`, `arc patch --revision xxx`, `arc patch --revision xxx --diff yyy`, `arc patch`; got good behavior on the good ones and sensible error messages on the other ones.
Maniphest Tasks: T13116
Differential Revision: https://secure.phabricator.com/D19292
Summary:
STDERR output with `%`s in it could cause:
```
ERROR 2: fprintf(): Too few arguments at [/usr/local/arcanist/src/workflow/ArcanistFeatureWorkflow.php:170]
```
Test Plan: Untested.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D19261
Summary:
See PHI458. The help text and prompting for "arc diff --draft" aren't very clear about whether updating publishes or not, nor about whether you need to keep passing "--draft" every time.
Make these behaviors more clear.
Test Plan:
- Ran `arc help diff`, read text.
- Updated a draft with "--draft", got new warning but it went through.
- Updated a published revision with "--draft", got new error.
Differential Revision: https://secure.phabricator.com/D19229
Summary:
In Go 1.10 the output for tests was changed to have also a "(cached)" mode in
addition to the normal timing info printed. This is on by default. This adds
support for parsing these lines instead of erroring out on the regex.
Test Plan: Have a unit test included, and will continue to poke at it locally.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D19161
Summary: Fixes T8236. I played around with a lot of variations of this but in the end it felt like the simple version was best.
Test Plan: Ran `arc weld a.txt b.txt`, observed very robust fusion of materials.
Maniphest Tasks: T8236
Differential Revision: https://secure.phabricator.com/D19081
Summary:
Fixes T13061. Both `arc lint` and `arc unit` accept an `--everything` flag, but the documentation isn't quite clear about what these flags do.
They act as though every //tracked// file in the repository (`git ls-files`, `hg manifest`, or `svn list -R`) is included in the argument list.
They do not lint/test ignored files (and I think almost all users would be very surprised if they did).
They also don't lint/test untracked files (files you have not yet used `git add`, `svn add`, or `hg add` on). This is slightly more contentious but we have good reasons for doing it (e.g., `git ls-files` often outperforms `find .` by a large margin) and I believe users very rarely use `--everything` in a situation where they have untracked files. The only real exception I can come up with is linter configuration/development, as in PHI343, and it seems okay to have a slightly surprising behvaior here.
Make the documentation more clear about what is in scope.
We could also rename these to `--nearly-everything` or whatever, but I think the name is probably clear enough given current information about how confusing this is (specifically: only rarely, in unusual cases).
Test Plan:
- Grepped for documentation about these flags.
- Ran `arc help lint`, `arc help unit`, `arc unit --everything x`, `arc lint --everything x` and read all the new messages.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13061
Differential Revision: https://secure.phabricator.com/D18989
Summary:
Fixes T8768. See PHI294. See that task for more details.
Git, Mercurial, `diff`, and `patch` have conspired to make things weird. To correctly handle files with spaces in the way everything else does and expects, we need to emit semantic trailing whitespace literals.
Test Plan:
- Created a file with spaces in it in a Mercurial repositroy, committed it, diffed it into a revision.
- Used `arc patch` to apply the change to a clean copy of the repository.
- Before patch: Mercurial incorrectly creates a file named `X`, not a file named `X Y.txt`.
- After patch: `arc patch` commit is identical to genuine commit.
- Also added test coverage. The other general behaviors here are fairly well covered already.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T8768
Differential Revision: https://secure.phabricator.com/D18869
Summary:
Currently, `arc` on `git` uses the following commands to examine the
state of the working tree and history; example times for a no-op diff in a
165k-file working tree are also shown:
```
1) git diff --no-ext-diff --no-textconv --submodule=short --raw 'fb062d4ecce5d9c1786b7bfc8a0dedf6b11fdd96' --
= 1,722,514 us
2a) git diff --no-ext-diff --no-textconv --submodule=short --raw 'HEAD' --
= 1,715,507 us
2b) git ls-files --others --exclude-standard
= 2,359,202 us
3) git diff-files --name-only
= 1,333,274 us
```
Steps (2a) and (2b) are run concurrently; this results in a total elapsed
wallclock time of approximately 5.4 seconds. This is inefficient -- all four of
the above steps must both load the index and examine the working copy, which may
be slow operations when large repositories are used. Additionally, none of the
effort of those stat calls on the working tree, or load time of the index, is
shared across the processes.
Step (1) is called from `getCommitRangeStatus`, which was split out in D4095; it
is currently never called on its own, only ever from `getWorkingCopyStatus`,
where it it combined with `getUncommittedStatus`. The current behavior of the
method is to return the set of changes //either// in local commits //or//
uncommitted in the working tree, which duplicates work that
`getUncommittedStatus` is intended to do. Changing the behavior of this method
(in Git, and other VCSes) to only examine _committed_ status seems both inline
with the name of the method and the original description of it in D4095 -- and
also serves to make it much faster, as it is an operation that need not inspect
the working tree at all.
Steps (2a), (2b), and (3) attempt to gather the state of the working copy, and
as such are all I/O bound but must examine nearly identical data. For git
2.11.0 and higher, we can instead rely on the machine-parseable `git status
--porcelain=2` format, which provides the information from all of these commands
at once. It also allows additional performance improvements, as `git status`
has been the focus of several optimizations in the latest versions of git (the
untracked cache and fsmonitor services, for instance), which are not available
in the lower-level `diff`, `ls-files`, and `diff-files` commands.
This has the added benefit of fixing a bug noticed in T9455, in that uncommitted
or unstaged changes in modules can now be detected, regardless of if they also
have changed their base commit. It further resolves a bug where `.gitmodules`
appeared to have unstaged changes, when in reality the unstaged changes were in
submodules elsewhere in the tree.
For backwards compatibility with versions of git < 2.11.0, the old code is left
in place. It is possible that the simpler output from v1 of `git status
--porcelain` would also suffice for some of the above benefits, but the payoff
of parsing yet another format is deemed insufficient; users wishing improved
performance should simply upgrade `git`.
Alltogether, these result in the following, for a no-op diff in a
165k-working-file tree:
```
1) git diff --no-ext-diff --no-textconv --submodule=short --raw 'fb062d4ecce5d9c1786b7bfc8a0dedf6b11fdd96' HEAD --
= 9,227 us
2) git status --porcelain=2 -z
= 739,964 us
```
...for a total of 749ms, an improvement of 4.7s.
Depends on D18841.
Test Plan: Existing tests.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D18842
Summary:
This adds tests that detail the current behavior of `arc` in
the presence of `git` submodules.
Test Plan: No behavior change; wrote the tests such that they pass.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D18841
Summary:
See PHI261. Currently "arc land" shows every build staus (passed, failed, building, etc) as yellow. Intended behavior is that passed builds are green, failed builds are red, and so on.
This is because of an unintended API change a while ago in D16356. Since the only impact was a cosmetic color issue, this escaped notice until now.
Additionally, try to use the modern `harbormaster.build.search` if it is available.
Test Plan:
- Ran `arc land` with running builds, got reasonable coloration.
- Faked the new method not being available, still got sensible behavior from the old method.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18837
Summary:
Most users, if they have gone through the trouble of
accepting the auto-fixes, are most likely going to want to take those
changes and attempt to land with them. Assuming "Y" for this prompt
streamlines for the more likely flow.
Test Plan: `arc lint`
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D18824
Summary: Fixes T10233. See PHI231. Users sometimes believe this warning is a bug and/or don't understand how they're supposed to resolve it.
Test Plan: Ran `arc land` on a revision in "Changes Planned", got a sensible prompt. Ran `arc land` on a revision in another non-accepted state, got more or less the old prompt.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T10233
Differential Revision: https://secure.phabricator.com/D18807
Summary: See D18651. This was missing an `unset()`.
Test Plan: Looked carefully.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18766
Summary: See PHI191. This is a rehash of an earlier fix, but we didn't have a test case for this half yet.
Test Plan:
- Added a failing test, made it pass.
- Added a linter like the one in PHI191, ran it, got a valid lint result instead of an exception.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18759
Summary:
Experimental branch. Ref T2543. Depends on D18742.
Add an "arc diff --draft" flag which holds revisions as drafts indefinitely.
Test Plan: Ran "arc diff --draft" when creating; ran "arc diff --draft" to try to update a revision and got a failure.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18743
Summary:
Depends on D18741. Switches to "differential.revision.edit" if we get a transaction list back from parsing (added by D18740).
This will allow us to use new transactions, notably "Hold as Draft" for "--draft".
Test Plan: Ran `arc diff`, got a revision via `differential.revision.edit`.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18742
Summary:
Experimental branch. Depends on D18739. Ref T2543.
To prepare for `--draft`, consolidate `--preview` and `--only`.
The old `--preview` is now `arc diff --only`.
The old `--only` is now `arc diff --only --nounit --nolint`.
I think this is more clear than other ways we could do it, since the difference between "draft" and "preview" seems less clear than the difference between "draft" and "only".
Test Plan: Ran `arc diff --only` and `arc diff --only --nounit --nolint`.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18740
Summary:
Experimental branch. Depends on D18738. Ref T2543.
To prepare for adding `--draft`, clean up some of the flags behaviors.
`--only` currently means `--preview --nolint --nounit`. I'm going to:
- Remove `--only`.
- Rename `--preview` to `--only`, since I think `--only` is a better flag for this behavior than `--preview`.
- If you want the old `--only`, you can `arc alias` it to `arc diff --only --nolint --nounit --browse` or similar.
Test Plan: Grepped for `'only'`, `only`, etc.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T2543
Differential Revision: https://secure.phabricator.com/D18739
Summary:
See PHI162. This corrects a couple more bugs:
- If the old file didn't end in a newline, we could end up printing two lines next to each other in the output.
- If the patch targeted "Line 6, character 1" instead of "line 5, character 3" in a file "1\n2\n3\n4\n5\n", we would fail to figure out what that meant when computing an offset because the last line has 0 characters on it.
Test Plan: Added failing unit tests, made them pass. Also tested with some fake linters similar to the ones described in PHI162.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18716
Summary: See PHI136. These are already optional on the server side in `HarbormasterBuildLintMessage`, and effectively mean "file-level issue", which is a bit niche but not unreasonable.
Test Plan: Checked that `HarbormasterBuildLintMessage` doesn't care if these keys exist, created this revision.
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18711
Summary: Depends on D18651. Allows Mercurial to auto-detect depends-on revisions.
Test Plan: See D18652 and D18653. Didn't actually test the empty repository states but they're probably okay. The `experimental` branch is an untamed wild. ¯\_(ツ)_/¯
Reviewers: amckinley
Reviewed By: amckinley
Differential Revision: https://secure.phabricator.com/D18654
Summary:
Depends on D18645. Fixes T11343. Currently, when you "arc diff", we don't detect dependencies even if your working copy is on top of another open revision you authored.
Use the new Ref/Hardpoint infrastructure from the `experimental` branch to look up the revision associated with the base commit when you run `arc diff`. If it exists and there's exactly one open revision you authored associated with it, add "Depends on ..." to the summary. This is a bit rough but the whole `experimental` branch is a bit of a wilderness for now.
Also remove `--cache` (passthru flag for `arc lint --cache`, which I removed in D18643) and `getUnderlyingWorkingCopyRevision()` (no callsites).
(This won't yet work in Mercurial, and does not make sense in SVN.)
Test Plan: Created this revision, got an auto "depends on" up top there.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T11343
Differential Revision: https://secure.phabricator.com/D18651
Summary:
Ref T12996. `arc lint` can render in different formats with `--output <format>`.
Previously, these were a big `switch()` statement and a bunch of hard-coded stuff.
Modularize them so that that third parties could add new renderers, and this code can be refactored toward parallelizing the `lint` step.
This has a small behavioral change: we no longer hide "autofix" messages. I'd like to generally simplify the number of amend/autofix flags here, so this edges us toward that slightly.
Test Plan: Ran `arc lint` with all the `--output` flags in states with warnings and no warnings, saw sensible-seeming behavior.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T12996
Differential Revision: https://secure.phabricator.com/D18645
Summary:
Ref T12996. See also T9749. This option is the opposite of `--lintall`, and means "don't show warnings on changed lines".
However, it's confusing (and I think no one uses it), and we can reasonably infer what you mean if you give us `--rev`. Simplify this logic a bit and choose the "all / only changed" behavior based on the only sensible thing we can reasonably do given the flags.
As with all the other cruft here, I'm pretty sure this came from FB.
Test Plan: Grepped for `only-changed`, `lintAll`, created this revision.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T12996
Differential Revision: https://secure.phabricator.com/D18644
Summary: Ref T12996. This is another half-baked Facebook-specific feature. Clear it out of the way so `arc lint` can be modernized more easily.
Test Plan: Grepped for `cache`, created this revision.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T12996
Differential Revision: https://secure.phabricator.com/D18643
Summary:
Ref T12996. Fixes T9749. This is a very old Facebook-specific thing which relies on other server-side Facebook-specific things and doesn't work properly anyway.
I don't actually remember what the use case for this flag was. It was either "the codebase has a million warnings, so showing warnings on files/lines you touched isn't good enough", or "weird warnings that raise lint on other lines", or some variation of those.
Regardless, I believe this feature benefits at most one install (Facebook circa 2011), and likely zero. It occasionally confuses normal users.
T9749 suggests a possible replacement workflow which is likely more practical, but I'd like to see a real problem description before considering this again.
Test Plan: Created this revision, grepped for `only-new`.
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T12996, T9749
Differential Revision: https://secure.phabricator.com/D18642
