Summary:
Ref T4292. This mostly implements the locking/versioning logic for multi-master repositories. It is only active on Git SSH pathways, and doesn't actually do anything useful yet: it just does bookkeeping so far.
When we read (e.g., `git fetch`) the logic goes like this:
- Get the read lock (unique to device + repository).
- Read all the versions of the repository on every other device.
- If any node has a newer version:
- Fetch the newer version.
- Increment our version to be the same as the version we fetched.
- Release the read lock.
- Actually do the fetch.
This makes sure that any time you do a read, you always read the most recently acknowledged write. You may have to wait for an internal fetch to happen (this isn't actually implemented yet) but the operation will always work like you expect it to.
When we write (e.g., `git push`) the logic goes like this:
- Get the write lock (unique to the repository).
- Do all the read steps so we're up to date.
- Mark a write pending.
- Do the actual write.
- Bump our version and mark our write finished.
- Release the write lock.
This allows you to write to any replica. Again, you might have to wait for a fetch first, but everything will work like you expect.
There's one notable failure mode here: if the network connection between the repository node and the database fails during the write, the write lock might be released even though a write is ongoing.
The "isWriting" column protects against that, by staying locked if we lose our connection to the database. This will currently "freeze" the repository (prevent any new writes) until an administrator can sort things out, since it'd dangerous to continue doing writes (we may lose data).
(Since we won't actually acknowledge the write, I think, we could probably smooth this out a bit and make it self-healing //most// of the time: basically, have the broken node rewind itself by updating from another good node. But that's a little more complex.)
Test Plan:
- Pushed changes to a cluster-mode repository.
- Viewed web interface, saw "writing" flag and version changes.
- Pulled changes.
- Faked various failures, got sensible states.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4292
Differential Revision: https://secure.phabricator.com/D15688
Summary:
Ref T4292. This adds some very basic cluster/device data to the new management view. Nothing interesting yet.
Also deal with disabled bindings a little more cleanly.
Test Plan: {F1214619}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4292
Differential Revision: https://secure.phabricator.com/D15685
Summary:
Ref T4292. This puts a very rough skeleton in place for the new "Manage Repository" UI, somewhat similar to the "Settings" UI.
Right now, it has one panel with no content, and is not reachable from the UI.
Test Plan: {F1214525}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4292
Differential Revision: https://secure.phabricator.com/D15683
Summary: Going to render these all normal case instead of all caps, and bump up the font size. Should be more consistent. Yellow if you green anything orange.
Test Plan: grep, lint
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15645
Summary:
Ref T10262. This removes one-time tokens and makes file data responses always-cacheable (for 30 days).
The URI will stop working once any attached object changes its view policy, or the file view policy itself changes.
Files with `canCDN` (totally public data like profile images, CSS, JS, etc) use "cache-control: public" so they can be CDN'd.
Files without `canCDN` use "cache-control: private" so they won't be cached by the CDN. They could still be cached by a misbehaving local cache, but if you don't want your users seeing one anothers' secret files you should configure your local network properly.
Our "Cache-Control" headers were also from 1999 or something, update them to be more modern/sane. I can't find any evidence that any browser has done the wrong thing with this simpler ruleset in the last ~10 years.
Test Plan:
- Configured alternate file domain.
- Viewed site: stuff worked.
- Accessed a file on primary domain, got redirected to alternate domain.
- Verified proper cache headers for `canCDN` (public) and non-`canCDN` (private) files.
- Uploaded a file to a task, edited task policy, verified it scrambled the old URI.
- Reloaded task, new URI generated transparently.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10262
Differential Revision: https://secure.phabricator.com/D15642
Summary:
Fixes T10721. When trying to load commits by identifier, we would take some bad pathways in Subversion if the repository had no callsign and end up missing the commits.
Fix this logic so it works for either callsigns (e.g., if passed `rXyyy`) or with PHIDs if passed repositories.
Test Plan:
- Viewed SVN commit in a Subversion repository with no callsign.
- Added a callsign, looked at it again.
- Viewed non-SVN commits in callsign and non-callsign repositories.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10721
Differential Revision: https://secure.phabricator.com/D15607
Summary:
Fixes T10385. Two issues:
- `$show_blame` and `$show_color` were improperly swapped.
- Code to hide these columns got dropped somewhere, probably in my recent-ish rewrite.
Test Plan:
- Showed/hid blame.
- Showed/hid colors.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10385
Differential Revision: https://secure.phabricator.com/D15528
Summary:
Ref T10537. For Nuance, I want to introduce new sources (like "GitHub" or "GitHub via Nuance" or something) but this needs to modularize eventually.
Split ContentSource apart so applications can add new content sources.
Test Plan:
This change has huge surface area, so I'll hold it until post-release. I think it's fairly safe (and if it does break anything, the breaks should be fatals, not anything subtle or difficult to fix), there's just no reason not to hold it for a few hours.
- Viewed new module page.
- Grepped for all removed functions/constants.
- Viewed some transactions.
- Hovered over timestamps to get content source details.
- Added a comment via Conduit.
- Added a comment via web.
- Ran `bin/storage upgrade --namespace XXXXX --no-quickstart -f` to re-run all historic migrations.
- Generated some objects with `bin/lipsum`.
- Ran a bulk job on some tasks.
- Ran unit tests.
{F1190182}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10537
Differential Revision: https://secure.phabricator.com/D15521
Summary:
Fixes T10665. See that task for discussion.
Because `$head_map` is not properly re-initialized for each ref we check, pushes which affect multiple branches (say, "A" and "B") can have information bleed from the first branch check to the second branch.
To trigger a problem behavior, you can push one commit which updates an existing branch, plus one commit which creates a new branch. If they process in the right order, the `$head_map` from the updated branch will bleed into the `$head_map` for the new branch and trigger an incorrect head split detection.
Test Plan:
- Pushed a set of changes which updated `branch-a` and created `branch-b`.
- Before change: improper detection of split heads.
- After change: clean push.
- Pushed a set of changes which split the head of `branch-d`.
- Correct detection of split heads.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10665
Differential Revision: https://secure.phabricator.com/D15522
Summary:
D9087 adds a nice typeahead but breaks the existing regex
search by quoting the pattern. Ideally, this change won't break the
typeahead, which as far as I can tell doesn't use the `pattern`
argument.
Test Plan:
Not yet.
RFC as to whether this change makes sense, will fix my local setup and resend if so.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15500
Summary: Ref T7789. Implement proper detection for read-only requests. Previously, we assumed every request was read/write and required lots of permissions, but we don't need "Can Push" permission if you're only cloning/fetching/pulling.
Test Plan:
- Set push policy to "no one".
- Fetched, got clean data out of LFS.
- Tried to push, got useful error.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T7789
Differential Revision: https://secure.phabricator.com/D15499
Summary:
- Fix spacing on InfoView inside collasped boxes
- Fix spacing on stacked PropertyLists in TwoColumn
- Fix spacing on Readmes on Tablets
- Fix unset variable on importing commits
Test Plan: Review each of the above cases.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15496
Summary: Ref T7789. This isn't the most perfect UI imaginable, but it's similar to what GitHub does and seems reasonable.
Test Plan:
{F1180271}
{F1180272}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T7789
Differential Revision: https://secure.phabricator.com/D15494
Summary:
Ref T7789. Ref T10604. This implements the `upload` action, which streams file data into Files.
This makes Git LFS actually work, at least roughly.
Test Plan:
- Tracked files in an LFS repository.
- Pushed LFS data (`git lfs track '*.png'; git add something.png; git commit -m ...; git push`).
- Pulled LFS data (`git checkout master^; rm -rf .git/lfs; git checkout master; open something.png`).
- Verified LFS refs show up in the gitlfsref table.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T7789, T10604
Differential Revision: https://secure.phabricator.com/D15492
Summary: I think I like this better -- but maybe right-aligned?
Test Plan:
{F1180295}
{F1180296}
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D15495
Summary:
Ref T7789. This implements:
- A new table to store the `<objectHash, filePHID>` relationship between Git LFS files and Phabricator file objects.
- A basic response to `batch` commands, which return actions for a list of files.
Test Plan:
Ran `git lfs push origin master`, got a little further than previously:
```
epriestley@orbital ~/dev/scratch/poemslocal $ git lfs push origin master
Git LFS: (2 of 1 files) 174.24 KB / 87.12 KB
Git LFS operation "upload/b7e0aeb82a03d627c6aa5fc1bbfd454b6789d9d9affc8607d40168fa18cf6c69" is not supported by this server.
Git LFS operation "upload/b7e0aeb82a03d627c6aa5fc1bbfd454b6789d9d9affc8607d40168fa18cf6c69" is not supported by this server.
```
With `GIT_TRACE=1`, this shows the batch part of the API going through.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T7789
Differential Revision: https://secure.phabricator.com/D15489
Summary: This updates (all?) of Diffusion/Audit to new UI, included edit and other extra form pages. It's fairly complete but I don't know all the nooks and crannies so to speak to fully verify I didn't mess anything up.
Test Plan: Tested creating new repositories, browsing, searching, auditing. Need more eyes.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15487
Summary:
Ref T7789. This builds on top of `git-lfs-authenticate` to detect LFS requests, read LFS tokens, and route them to a handler which can do useful things.
This handler promptly drops them on the floor with an error message.
Test Plan:
Here's a transcript showing the parts working together so far:
- `git-lfs` connects to the server with SSH, and gets told how to connect with HTTP to do uploads.
- `git-lfs` uses HTTP, and authenticates with the tokens properly.
- But the server tells it to go away, and that it doesn't support anything, so the operation ultimately fails.
```
$ GIT_TRACE=1 git lfs push origin master
12:45:56.153913 git.c:558 trace: exec: 'git-lfs' 'push' 'origin' 'master'
12:45:56.154376 run-command.c:335 trace: run_command: 'git-lfs' 'push' 'origin' 'master'
trace git-lfs: Upload refs origin to remote [master]
trace git-lfs: run_command: git rev-list --objects master --not --remotes=origin
trace git-lfs: run_command: git cat-file --batch-check
trace git-lfs: run_command: git cat-file --batch
trace git-lfs: run_command: 'git' config -l
trace git-lfs: tq: starting 3 transfer workers
trace git-lfs: tq: running as batched queue, batch size of 100
trace git-lfs: prepare upload: b7e0aeb82a03d627c6aa5fc1bbfd454b6789d9d9affc8607d40168fa18cf6c69 lfs/dog1.jpg 1/1
trace git-lfs: tq: sending batch of size 1
trace git-lfs: ssh: local@localvault.phacility.com git-lfs-authenticate diffusion/18/poems.git upload
trace git-lfs: api: batch 1 files
trace git-lfs: HTTP: POST http://local.phacility.com/diffusion/POEMS/poems.git/info/lfs/objects/batch
trace git-lfs: HTTP: 404
trace git-lfs: HTTP: {"message":"Git LFS operation \"objects\/batch\" is not supported by this server."}
trace git-lfs: HTTP:
trace git-lfs: api: batch not implemented: 404
trace git-lfs: run_command: 'git' config lfs.batch false
trace git-lfs: tq: batch api not implemented, falling back to individual
trace git-lfs: ssh: local@localvault.phacility.com git-lfs-authenticate diffusion/18/poems.git upload b7e0aeb82a03d627c6aa5fc1bbfd454b6789d9d9affc8607d40168fa18cf6c69
trace git-lfs: api: uploading (b7e0aeb82a03d627c6aa5fc1bbfd454b6789d9d9affc8607d40168fa18cf6c69)
trace git-lfs: HTTP: POST http://local.phacility.com/diffusion/POEMS/poems.git/info/lfs/objects
trace git-lfs: HTTP: 404
trace git-lfs: HTTP: {"message":"Git LFS operation \"objects\" is not supported by this server."}
trace git-lfs: HTTP:
trace git-lfs: tq: retrying 1 failed transfers
trace git-lfs: ssh: local@localvault.phacility.com git-lfs-authenticate diffusion/18/poems.git upload b7e0aeb82a03d627c6aa5fc1bbfd454b6789d9d9affc8607d40168fa18cf6c69
trace git-lfs: api: uploading (b7e0aeb82a03d627c6aa5fc1bbfd454b6789d9d9affc8607d40168fa18cf6c69)
trace git-lfs: HTTP: POST http://local.phacility.com/diffusion/POEMS/poems.git/info/lfs/objects
trace git-lfs: HTTP: 404
trace git-lfs: HTTP: {"message":"Git LFS operation \"objects\" is not supported by this server."}
trace git-lfs: HTTP:
Git LFS: (0 of 1 files) 0 B / 87.12 KB
Git LFS operation "objects" is not supported by this server.
Git LFS operation "objects" is not supported by this server.
```
Reviewers: chad
Reviewed By: chad
Subscribers: eadler
Maniphest Tasks: T7789
Differential Revision: https://secure.phabricator.com/D15485
Summary:
Ref T7789. This implements a (probably) usable "git-lfs-authenticate" on top of the new temporary token infrastructure.
This won't actually do anything yet, since nothing reads the tokens.
Test Plan:
```
$ ./bin/ssh-exec --phabricator-ssh-user admin --ssh-command 'git-lfs-authenticate'
phabricator-ssh-exec: Expected `git-lfs-authenticate <path> <operation>`, but received too few arguments.
```
```
$ ./bin/ssh-exec --phabricator-ssh-user admin --ssh-command 'git-lfs-authenticate x'
phabricator-ssh-exec: Unrecognized repository path "x". Expected a path like "/diffusion/X/" or "/diffusion/123/".
```
```
$ ./bin/ssh-exec --phabricator-ssh-user admin --ssh-command 'git-lfs-authenticate diffusion/22'
Exception: Expected `git-lfs-authenticate <path> <operation>`, but received too few arguments.
```
```
$ ./bin/ssh-exec --phabricator-ssh-user admin --ssh-command 'git-lfs-authenticate diffusion/22 y'
Exception: Git LFS operation "y" is not supported by this server.
```
```
$ ./bin/ssh-exec --phabricator-ssh-user admin --ssh-command 'git-lfs-authenticate diffusion/22 upload'
{"header":{"Authorization":"Basic QGdpdC1sZnM6NmR2bDVreWVsaXNuMmtnNXBtbnZwM3VlaWhubmI1bmI="},"href":"http:\/\/local.phacility.com\/diffusion\/22\/new-callsign-free-repository.git\/info\/lfs"}
```
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T7789
Differential Revision: https://secure.phabricator.com/D15482
Summary:
First pass at converting Differential, I likely have some buggy-poos but thought I'd toss this up now in case very bad bugs present.
To do:
- Need to put status back on Hovercards
- "Diff Detail" probably needs a better design
Test Plan: Looking at lots of diffs, admittedly I dont have harbormaster, etc, running locally. Checked Diffusion for Table of Content changes on small and large commits.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15463
Summary:
Two minor changes here:
- Replace `get/setUser()` with `get/setViewer()` for consistency with everything else.
- `getViewer()` now throws if no viewer is set. We had a lot of code that either "should" check this but didn't, or did check it in an identical way, duplicating work. In contrast, very little code checks for a viewer but works if one is not present.
Test Plan:
- Grepped for `->user`.
- Attempted to fix all callsites inside `*View` classes.
- Browsed around a bunch of applications, particularly Calendar, Differential and Diffusion, which seemed most heavily affected.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D15412
Summary:
Ref T10449. Currently, we store classes (like "AlmanacClusterRepositoryServiceType") in the database.
Instead, store types (like "cluster.repository").
This is a small change, but types are a little more flexible (they let us freely reanme classes), a little cleaner (fewer magic strings in the codebase), and a little better for API usage (they're more human readable).
Make this minor usability change now, before we unprototype.
Also make services searchable by type.
Also remove old Almanac API endpoints.
Test Plan:
- Ran migration, verified all data migrated properly.
- Created, edited, rebound, and changed properties of services.
- Searched for services by service type.
- Reviewed available Conduit methods.
Reviewers: chad
Reviewed By: chad
Subscribers: yelirekim
Maniphest Tasks: T10449
Differential Revision: https://secure.phabricator.com/D15346
Summary: Fixes T10432. I missed these in making properties non-default.
Test Plan: Diffusion now works again in a cluster configuration.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10432
Differential Revision: https://secure.phabricator.com/D15337
Summary:
These trip me up every time because Differential has:
> Comment, Accept, Request Changes, Resign, Commandeer, Add Reviewers, Add Subscribers
while audits currently show:
> Comment, Add Subscribers, Add Auditors, Accept, Raise Concern, Resign
Now they're more or less in the same order which helps with muscle memory.
Test Plan: Careful inspection.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15323
Summary: Swapping out to PHUIDocumentProView to remove all calls to PHUIDocumentView.
Test Plan: Review the Phabricator Readme.MD in Diffusion
Reviewers: epriestley, avivey
Reviewed By: avivey
Subscribers: avivey, Korvin
Differential Revision: https://secure.phabricator.com/D15308
Summary:
Ref T4245. This could still use a little UI smoothing, but:
- Don't require a callsign on the create flow (you can add one later in "Edit Basic Information" if you want).
- Allow existing callsigns to be removed.
Test Plan:
- Created a new repository with no callsign.
- Cloned it; pushed to it.
- Browsed around Diffusion a bunch.
- Visited a commit URI.
- Added a callsign to it.
- Removed the callsign again.
- Referenced it with `R22` in remarkup.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D15305
Summary:
Ref T4245. This is a prelude to removing them from the "create" screen.
Currently, if you try to delete the callsign you get an unceremonious database error, but the next diff (or maybe two) will permit that, so I didn't put any "this is required yada yada" text in.
This could also maybe use some big flashing warning lights and a "if you edit this, all your URIs break" but I'll save that for later.
Test Plan: Changed the callsign for a repository.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D15304
Summary: Ref T4245. When creating new repositories, set a default local path based on the repository ID instead of callsign.
Test Plan:
- Created a new repository.
- Saw it get a reasonable, ID-based local path.
- Edited a repository to make sure the `applyFinalEffects()` wasn't doing anything whacky.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D15303
Summary: Ref T4245. Consolidates the URI parsing/rewriting logic so that repositories can be served from either `/diffusion/XYZ/` or `/diffusion/123/`, over both HTTP and SSH.
Test Plan:
- Pulled a Git repository by ID and callsign over HTTP and SSH.
- Pulled a Mercurial repository by ID and callsign over HTTP and SSH.
- Pulled a Subversion repository by ID and callsign over SSH (no HTTP support for SVN).
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D15302
Summary:
Ref T4245. Make `/diffusion/123/` work, but redirect the user to `/diffusion/XYZ/` if the repository has a callsign.
(Right now, every repository has a callsign, so this always redirects.)
Also redirect `/R123:abcdef` if the repository has a callsign.
Also also, move the Pull garbage collector somewhere more sensible.
Test Plan:
- Added test coverage.
- Visited `/diffusion/1/`, was redirected.
- Visited `/diffusion/R1:abcdef`, was redirected.
- Browsed Diffusion normally.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D15301
Summary: Ref T4245. This has no callers.
Test Plan:
- Ran `git grep -i 'getCallsign('` and visually verified that no callers could reasonably be `DiffusionRequest` objects (there are only 23 remaining sites, and about half are `$this->...` in `PhabricatorRepository`.
- Browsed around directory/file/branch/content/diff/etc pages in Diffusion.
Reviewers: chad, avivey
Reviewed By: avivey
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D15295
Summary:
Ref T4245. We pass this exclusively for use by additional third-party hooks.
This is technically a backward compatibility break, but I suspect it doesn't affect anyone:
- Probably almost no one is using this (there are few reasons to, even for the tiny number of installs with custom commit hooks).
- If they are, there's a good chance the PHID will work anyway, since nearly all scripts and Conduit methods will accept it in place of a callsign now, and if it's in logging or debugging code the PHID is a reasonable substitute
- Even if it doesn't just keep working, the break should be very obvious in most reasonable cases.
I'll call this out explicitly in the changelog, though -- almost everything else will just continue working, but this is a strict compatibility break.
Test Plan:
- Ugh.
- Picked a hosted Git repo out of Diffusion.
- Went to the path on disk.
- Went into `hooks/`.
- Went into `pre-receive-phabricator.d/`.
- Wrote this hook and gave it `chmod +x`:
```name=stuff.sh
#!/bin/sh
echo $PHABRICATOR_REPOSITORY >> /tmp/stuff.log
```
- Pushed to the repository.
- Saw a PHID show up in the log:
```
$ cat /tmp/stuff.log
PHID-REPO-bqkcdp47euwnwlasrsrh
```
Reviewers: chad, avivey
Reviewed By: avivey
Subscribers: avivey
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D15294
Summary: Fixes T9562. We already do this for tags, but didn't have similar logic for branches. Implement that logic.
Test Plan:
- Set limit to 1, saw "More branches", clicked it, got the correct results.
- Verified that branch table with no specified commit still works properly.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9562
Differential Revision: https://secure.phabricator.com/D15284
Summary:
Fixes T10304. In Mercurial, we must enumerate the whole file tree. Currently, we incorrectly count files within directories (which won't be shown) toward the "100 file" limit at top level, so directories with more than 100 subpaths are truncated improperly.
This is approxiately the same as @richardvanvelzen's fix.
Test Plan: Viewed a large Mercurial repository, saw a complete directory listing.
Reviewers: chad
Reviewed By: chad
Subscribers: richardvanvelzen
Maniphest Tasks: T10304
Differential Revision: https://secure.phabricator.com/D15282
Summary: Moves all the one off object calls to PHUIRemarkupView, adds a "Document" call as well (future plans).
Test Plan: Visited most pages I could get access to, but may want extra careful eyes on this diff.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15281
Summary:
Fixes T10264. I'm reasonably confident that this is the chain of events here:
First, prior to 8269fd6e, we would ignore "Content-Encoding" when reading inbound bodies. So if a request was gzipped, we would read a gzipped body, then give `git-http-backend` a gzipped body with "Content-Encoding: gzip". Everything matched normally, so that was fine, except in the cluster.
In the cluster, we'd accept "gzip + compressed body" and proxy it, but not tell cURL that it was already compressed. cURL would think it was raw data, so it would arrive on the repository host with a compressed body but no "Content-Encoding: gzip". Then we'd hand it to git in the same form. This caused the issue in 8269fd6e: handing it compressed data, but no "this is compressed" header.
To fix this, I made us decompress the encoding when we read the body, so the cluster now proxies raw data instead of proxying gzipped data. This fixed the issue in the cluster, but created a new issue on non-cluster hosts. The new issue is that we accept "gzip + compressed body" and decompress the body, but then pass the //original// header to `git-http-backend`. So now we have the opposite problem from what we originally had: a "gzip" header, but a raw body.
To fix //this//, we could do two things:
- Revert 8269fd6e, then change the proxy request to preserve "Content-Encoding" instead.
- Stop telling `git-http-backend` that we're handing it compressed data when we're handing it raw data.
I did the latter here because it's an easier change to make and test, we'll need to interact with the raw data later anyway, to implement repository virtualization in connection with T8238.
Test Plan: See T10264 for users confirming this fix.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10264
Differential Revision: https://secure.phabricator.com/D15258
Summary: Also adds the commit to the header underneath the title. Ref T7628
Test Plan: Review a few Diffusion pages.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T7628
Differential Revision: https://secure.phabricator.com/D15246
Summary: These are not needed I think? and handy for cut and paste. Fixes T7628
Test Plan: cut and paste easier from commit hash.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T7628
Differential Revision: https://secure.phabricator.com/D15245
Summary:
Ref T10289. This probably doesn't cover everything but should do a little bit better.
Although we should mabye just exlude milestones from this menu completely?
Test Plan: {F1093937}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10289
Differential Revision: https://secure.phabricator.com/D15191
Summary: No UI changes, just some search and replace for UI consistency.
Test Plan: Test person and object hovercards still work. UIExamples too.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15172
Summary: Fixes T10259. There was no real reason to do this `ip2long()` stuff in the first place -- it's very slightly smaller, but won't work with ipv6 and the savings are miniscule.
Test Plan:
- Ran migration.
- Viewed logs in web UI.
- Pulled and pushed.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10259
Differential Revision: https://secure.phabricator.com/D15165
Summary: I believe this got clobbered in rP8b6edaa4e238a809fe78e6d14ad0705545f8179f. This index doesn't seem to be present in the line dictionary and we're now relying on `$line_index` for the current position.
Test Plan:
before {F1085522}
after {F1085521}
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley
Differential Revision: https://secure.phabricator.com/D15156
Summary: Ref T10228. Commands like `git-http-backend` can emit errors with raw bytes in the output. Sanitize these if present so we can log them in JSON format.
Test Plan: Edited this into production. >_> sneaky sneaky <_<
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10228
Differential Revision: https://secure.phabricator.com/D15144
Summary: Fixes T10226. I just made a mistake here when rewriting this recently.
Test Plan: {F1079166}
Reviewers: chad, avivey
Reviewed By: avivey
Maniphest Tasks: T10226
Differential Revision: https://secure.phabricator.com/D15131
Summary:
Ref T10228. This is currently quite limited:
- No UI.
- No SSH support.
My primary goal is to debug the issue in T10228. In the long run we can expand this to be a bit fancier.
Test Plan:
Made various valid and invalid clones, got sucess responses and not-so-successful responses, viewed the log table for general corresponding messages and broad sanity.
Ran GC via `bin/phd debug trigger`, no issues.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10228
Differential Revision: https://secure.phabricator.com/D15127
Summary: Mostly for consistency, we're not using other forms of icons and this makes all classes that use an icon call it in the same way.
Test Plan: tested uiexamples, lots of other random pages.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D15125
Summary: Fixes T10212. This method was removed in D14990, but I missed a callsite.
Test Plan: Disabling blame now works nicely.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10212
Differential Revision: https://secure.phabricator.com/D15100
Summary:
Fixes T10186. After D14970, `diffusion.filecontentquery` puts the content in a file and returns the file PHID.
However, it does this in a way that doesn't go through the chunking engine, so it will fail for files larger than the chunk threshold (generally, 8MB).
Instead, stream the file from the underlying command directly into chunked storage.
Test Plan:
- Made a commit including a really big file: 4dcd4c492b
- Used `diffusion.filecontentquery` to load file content.
- Parsed/imported commit locally.
- Used `diffusion.filecontentquery` to load content for smaller files (README, etc).
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10186
Differential Revision: https://secure.phabricator.com/D15072
Summary:
Ref T4705 (there are also some other adjacent related tasks dealing with URIs).
Currently, we issue a "get repositories matching URIs: ..." query by loading every possible repository and then checking their URIs in PHP.
Instead, put URIs in a separate table. I plan for each repository to potentially have multiple URIs soon, so this prepares for that.
Test Plan:
- Ran migrations.
- Looked at index table, made sure it appeared sensible.
- Ran some queries by `uri` to find repositories, found the repositories I expected.
- Updated the remote URI of a repository, saw queries / index update appropriately.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4705
Differential Revision: https://secure.phabricator.com/D15005
Summary: Fixes T8826. Git tracks an "author date", which may be different from the "committed date". We don't currently extract/show this; do so.
Test Plan: {F1059235}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T8826
Differential Revision: https://secure.phabricator.com/D14995
Summary: Ref T4245. Full commit display names (like `rPaaaa`) are going to be obnoxious soon in some cases (e.g., `rPaaaa` becomes `R123:aaaa`, which is much uglier) so reduce how often we show the repository in cases where it isn't really necessary to include it.
Test Plan:
- Saw no more `rX` on repository list view for Git/Mercurial (still present for Subversion).
- Saw no more `rX` on various repository detail views, except when referencing other commits (e.g., mentions).
- Grepped for removed `getShortName()`.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14990
Summary:
Ref T4245.
- Rename "Clone/Checkout As" to "Short Name" in the UI.
- Allow any repository to have a short name, not just hosted repositories.
Test Plan:
- Ran migration.
- Reviewed old transactions, saw they looked good.
- Edited an existing repository's short name.
- Gave an imported repository a new short name.
- Removed a repository's short name.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14989
Summary:
Fixes T7938.
- Primarily, users can currently shoot themselves in the foot by putting `../../etc/passwd` and other similar nonsense in these fields (this is not dangerous, but also does not work). Require sensible names.
- Enforce uniqueness so these names can be used in URIs and as identifiers in the future.
- (This doesn't start actually using them for anything fancy yet.)
Test Plan:
- Gave several repositories clone names: a valid name, two duplicate names, an invalid, name, some with no names.
- Ran migrations.
- Got clean conversion for valid names, appropriate errors for invalid/duplicate names.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T7938
Differential Revision: https://secure.phabricator.com/D14986
Summary:
Fixes T9323. Two minor fixes:
- On the first commit, don't render a downward line.
- Clean up a 1px spacing issue that had cropped up a while ago when we added icons or something, I think.
Test Plan:
Before:
{F1057248}
After:
{F1057249}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9323
Differential Revision: https://secure.phabricator.com/D14974
Summary:
Fixes T9319. Proxied requests (e.g., in the cluster) for binary files (like images) currently fail because we can not return binary data over Conduit in JSON.
Although Conduit will eventually support binary-safe encodings, a cleaner approach to this is just to return a `filePHID` instead of the raw content. This is generally faster and more flexible, and gives us more opportunities to add caching later.
After making the call, the client pulls the file data separately.
We also no longer need to return a complex data structure because we don't do blame over this call any longer.
Test Plan:
- Viewed images in Diffusion.
- Viewed READMEs in Diffusion.
- Used `bin/differential attach-commit rX Dy` to hit attach pathway.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9319
Differential Revision: https://secure.phabricator.com/D14970
Summary: I was looking at some random un-revisioney repository for most of my testing and missed these.
Test Plan: Viewed blame of a file with some revisions.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14965
Summary: Fixes T2450. If we spend more than 15 seconds in blame, just cut it off.
Test Plan:
- Changed timeout to 0.01 seconds.
- Did blame on a non-highlighted file, got no blame, saw warning.
- Did blame on a highlighted file, got no blame.
- Note: you don't get a warning here because of Ajax stuff. It'd be kind of tricky to add and doesn't seem like a big deal so I'm planning to leave it as-is for now.
Reviewers: chad
Reviewed By: chad
Subscribers: 20after4, chasemp
Maniphest Tasks: T2450
Differential Revision: https://secure.phabricator.com/D14964
Summary:
Ref T2450. This reorganizes code to improve performance.
Mostly, there are a lot of things which are unique per commit (author name, links, short name, etc), but we were rendering them for every line.
This often meant we'd render the same author's name thousands of times. This is slower than rendering it only once.
In 99% of interfaces this doesn't matter, but blame is weird and it's significant on big files.
Test Plan:
Locally, `__phutil_library_map__.php` now has costs of roughly:
- 550ms for main content (from 650ms before the patch).
- 1,500ms for blame content (frrom 1,800ms before the patch).
So this isn't huge, is a decent ~20%-ish performance gain for shuffling some stuff around.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T2450
Differential Revision: https://secure.phabricator.com/D14963
Summary:
Ref T2450. File blame tends to have the same commit a lot of times, and we don't do lookups like this efficiently right now.
In particular, for a file like `__phutil_library_map__.php`, we would issue a query with ~9,000 clauses like this:
```
(repositoryID = 1 AND commitIdentifier LIKE "XYZ%")
```
...but only a few hundred of those identifiers were unique. Instead, issue only one clause per unique identifier.
MySQL also seems to do a little better on "commitIdentifier = X" if we have the full hash, so special case that slightly.
Test Plan:
- Issuing a query for only unique identifiers dropped the cost from 400ms to 100ms locally.
- Swapping to `=` if we have the full hash dropped the cost from 100ms to 75ms locally.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T2450
Differential Revision: https://secure.phabricator.com/D14962
Summary:
Fixes T4366. Two years ago, Facebook put 16,000 files in a directory. Today, the page has nearly loaded.
Paginate large directories.
Test Plan:
- Viewed home and browse views in Git, Mercurial and Subversion.
I put an artificially small page size (5) on home:
{F1055653}
I pushed 16,000 files to a directory and paged through them. Here's the last page, which rendered in about 200ms:
{F1055655}
Our behavior is a bit better than GitHub here, which shows only the first 1,000 files, disables pagination, and can't retrieve history for the files:
{F1055656}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4366
Differential Revision: https://secure.phabricator.com/D14956
Summary:
When looking at a large file in Diffusion:
- disable highlighting if it's huge and show a note about why;
- pick up a few other optimizations.
Test Plan: Locally, this improves the main render of `__phutil_library_map__.php` from 3,200ms to 600ms for me, at the cost of syntax highlighting (we can eventually add view options and let users re-enable it).
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14959
Summary:
Ref T2450. Ref T9319. This is still a bit messy, but not quite so bad as it was: instead of using a single call to get both blame information and file content, use `diffusion.blame` for blame information.
This will make optimizations to both blame and file content easier.
Test Plan: Viewed a bunch of blame (color on/off, blame on/off).
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T2450, T9319
Differential Revision: https://secure.phabricator.com/D14958
Summary:
Fixes T2451. Several motivations here, from strongest to weakest:
- Currently, getting blame and file content are closely entwined. This makes fixing T9319 more difficult, and I want to fix it. I want to separate blame from content so there's more flexibility in how we approach this issue.
- This makes pursuing T2450 easier, if it turns out to be a meaningful win.
- If we can get a win on blame performance, we can do `arc blame` eventually if we want.
Test Plan:
- Blamed in SVN, Git and Mercurial.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T2451
Differential Revision: https://secure.phabricator.com/D14957
Summary: Ref T4245. This is the last of it, and covers the clone/push stuff.
Test Plan:
- Cloned git.
- Pushed git.
- Cloned mercurial.
- Pushed mercurial.
- Visited a `blah.git` URL in my browser just because; got redirected to a human-facing UI.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14949
Summary: Ref T4245. On their best day these don't work all that well, but I'm pretty sure I didn't make anything worse.
Test Plan:
- Viewed global lint.
- Viewed lint for a repository.
- Viewed lint details for a particular message.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14948
Summary: Ref T4245. This gets everything else except serving HTTP requests (complicated) and lint (quite weird).
Test Plan:
- Viewed a diff.
- Viewed externals.
- Viewed history table to see last modified.
- Did path completion and validation in Owners.
- Did tree path search in Diffusion.
- Viewed a repository.
- Created a new repository.
- Looked up symbols.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14947
Summary: Fixes T7852. Although `1` could also indicate other kinds of problems, assume it means "no results".
Test Plan: Searched for nonsense strings in Git and Mercurial. Searched for valid strings in Git and Mercurial.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T7852
Differential Revision: https://secure.phabricator.com/D14943
Summary:
Ref T4245. Browsing is huge and currently split across 5 files using controller delegation.
Although having a huge file isn't great, I think the way it is split up is currently worse, and it gets weird with more flexible repository identifiers.
So this is mostly merging five controllers into one, then a bit of modernization.
I think this can probably be split up better by pulling some of it out into views, instead of using delegation.
Test Plan: Browsed files, directories, and search results.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14942
Summary: Ref T4245. Prepares these controllers to accept alternate identifers, plus minor spacing and layout fixes.
Test Plan: Viewed tags, viewed branches.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14941
Summary:
Ref T4245. This adds support for both ID-based and callsign-based routes, although the ID-based routes don't occur anywhere.
Also moves toward simplifying the DiffusionRequest stuff.
Test Plan: Visited normal callsign-based commit pages; visited new ID-based commit pages.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14940
Summary: Ref T4245. This further reduces the reliance on callsigns in Diffusion.
Test Plan:
- Pretty reasonable test coverage already exists.
- Browsed repository list, browse view, history view, content view, change view, commit view, tag view, branch view of repositories.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14937
Summary: Ref T4245. These mostly relate to building URIs.
Test Plan: Tried to hunt down as many of these in the UI as I could. Some are a bit tricky but they should be low-risk.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14933
Summary:
Ref T4245. Like everything else, accept more identifiers.
This needs a change in `arc`, which I've made a note about elsewhere.
Test Plan: Used "Update Now" from web UI, saw update get scheduled.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14932
Summary: Ref T4245. Pass the whole repository in so it can do something else in a future change.
Test Plan: Loaded changesets in Diffusion.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14931
Summary:
Ref T4245. This was obsoleted long ago and has no callers in Phabricator or Arcanist.
Also some minor cleanup.
Test Plan: `grep` for callers everywhere.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14930
Summary: Ref T4245. More of the same, just narrowing down the easy cases.
Test Plan:
- Called `diffusion.querycommit`.
- Browsed branches.
- Browsed repository.
- Browsed directory.
- Searched for stuff.
- Viewed a commit.
- Viewed a file diff.
- Edited a commit.
- Viewed history.
- Viewed tags.
- Viewed push log.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14929
Summary:
Ref T4245. Broaden support to include "ABCD", "rABCD", "1234", "R1234", etc.
This doesn't change the old behavior, just accepts more stuff.
Test Plan:
- Browsed Diffusion.
- Made various calls via API console.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14928
Summary: Ref T4245. These are all straightforward to remove.
Test Plan:
- Edited paths in a package.
- Ran `bin/audit delete --repositories ...` with various identifiers.
- Searched by repository for `R3`, `rAAAA` in Harbormaster.
- Did a Herald dry run on a commit.
- Browsed commits, made comments.
- Viewed a Releeph product list.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T4245
Differential Revision: https://secure.phabricator.com/D14927
Summary: This logic wasn't quite right.
Test Plan: Hovered over a recognized commit, got a valid hovercard
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14925
Summary:
Ref T8980. Move away from events to EngineExtensions.
This also simplifies hovercards a bit:
- Removes tasks from revision cards.
- Removes blockers/blocked from task cards.
- Removes "Send Message" from user cards.
These mostly felt cluttery to me. Open to arguments to retain them. I think we can make better use of the space, though (e.g., flags, projects + board columns).
Test Plan:
- Viewed people, task, revision, commit and project hovercards.
{F1043256}
{F1043257}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T8980
Differential Revision: https://secure.phabricator.com/D14878
Summary:
Ref T10004. After D14804, we get this behavior by default and no longer need to set it explicitly.
(If some endpoint did eventually need to set it explicitly, it could just change what it passes to `setHref()`, but I believe we currently have no such endpoints and do not foresee ever having any.)
Test Plan:
- As a logged out user, clicked various links in Differential, Maniphest, Files, etc., always got redirected to a sensible place after login.
- Grepped for `setObjectURI()`, `getObjectURI()` (there are a few remaining callsites, but to a different method with the same name in Doorkeeper).
Reviewers: chad
Reviewed By: chad
Subscribers: hach-que
Maniphest Tasks: T10004
Differential Revision: https://secure.phabricator.com/D14805
Summary: Ref T9952. This will let me put a "Branch: [____]" control on the "Land Revision" dialog so users can choose a branch to target.
Test Plan: Used `/typeahead/class/` to vet basic behavior.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9952
Differential Revision: https://secure.phabricator.com/D14732
Summary: Fixes T9941. I think someone from Perforce emailed us about 10 years ago and I added this link in response, but I haven't seen other interest in Perforce since then. Link is now dead.
Test Plan:
- {nav Diffusion > Create Repository > Import Existing}, no more Perforce link.
- Grepped for `perforce`.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9941
Differential Revision: https://secure.phabricator.com/D14720
Summary: Fixes T5788. We already have this as a pre-commit field, add it as a post-commit field too.
Test Plan: Ran this rule on a merge commit. Also ran it on a non-merge commit. Both got the correct value.
Reviewers: avivey, chad
Reviewed By: avivey, chad
Subscribers: avivey
Maniphest Tasks: T5788
Differential Revision: https://secure.phabricator.com/D14685
Summary:
Fixes T9798. That task has good repro instructions.
In sub-views, we don't link the "History" icon correctly -- we only link it to `history/README` instead of `history/path/to/README`. Add the full path.
Also canonicalize the paths in a slightly prettier and more consistenty way.
Test Plan: Viewed root and non-root browse tables, saw links show up properly.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9798
Differential Revision: https://secure.phabricator.com/D14491
Summary:
This diff adds a new mode argument to the Diffusion Conduit API with two options:
- "overwrite": the default, maintains the current behavior of deleting all coverage
in the specified branch before uploading the new coverage
- "update": does not delete old coverage, but will overwrite previous
coverage information if it's for the same file and commit
`DiffusionRequest::loadCoverage` already loads a file's coverage from the
latest available commit, so uploading coverage for different files in different
commits with "update" will result in seeing the latest uploaded coverage in
Diffusion.
Test Plan: manual local verification
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14428
