1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-24 06:20:56 +01:00
Commit graph

851 commits

Author SHA1 Message Date
epriestley
575c01373e Extract repository command construction from Repositories
Summary:
Ref T4292. Ref T10366. Depends on D15751. Today, generating repository commands is purely a function of the repository, so they use protocols and credentials based on the repository configuration.

For example, a repository with an SSH "remote URI" always generate SSH "remote commands".

This needs to change in the future:

  - After T10366, repositories won't necessarily just have one type of remote URI. They can only have one at a time still, but the repository itself won't change based on which one is currently active.
  - For T4292, I need to generate intracluster commands, regardless of repository configuration. These will have different protocols and credentials.

Prepare for these cases by separating out command construction, so they'll be able to generate commands in a more flexible way.

Test Plan:
  - Added unit tests.
  - Browsed diffusion.
  - Ran `bin/phd debug pull` to pull a bunch of repos.
  - Ran daemons.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4292, T10366

Differential Revision: https://secure.phabricator.com/D15752
2016-04-19 04:51:48 -07:00
epriestley
f424f9f2d2 Record more details about where a write is taking place while holding a cluster lock
Summary: Ref T4292. This will let the UI and future `bin/repository` tools give administrators more tools to understand problems when reporting or resolving them.

Test Plan:
  - Pushed fully clean repository.
  - Pushed previously-pushed repository.
  - Forced write to abort, inspected useful information in the database.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4292

Differential Revision: https://secure.phabricator.com/D15748
2016-04-18 11:55:27 -07:00
epriestley
368d2d1ddb Improve robustness of cluster version bookkeeping
Summary:
Ref T4292. Small fixes:

  - There was a bug with the //first// write, where we'd write 1 but expect 0. Fix this.
  - Narrow the window where we hold the `isWriting` lock: we don't need to wait for the client to finish.
  - Release the lock even if something throws.
  - Use a more useful variable name.

Test Plan:
  - Made new writes to a fresh cluster repository.
  - Made sequential writes.
  - Made concurrent writes.
  - Made good writes and bad writes.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4292

Differential Revision: https://secure.phabricator.com/D15747
2016-04-18 11:54:59 -07:00
epriestley
595f203816 Correct RepositoryURI schema and propagate adjust exit code correctly
Summary:
Fixes T10830.

  - The return code from `storage adjust` did not propagate correct.
  - There was one column issue which I missed the first time around because I had a bunch of unrelated stuff locally.

Test Plan:
  - Ran `bin/storage upgrade -f` with failures, used `echo $?` to make sure it exited nonzero.
  - Got fully clean `bin/storage adjust` by dropping all my extra local tables.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10830

Differential Revision: https://secure.phabricator.com/D15746
2016-04-18 08:11:22 -07:00
epriestley
9352ed8abb Add missing RepositoryURI table + run storage adjustments in tests
Summary:
Fixes T10830. Ref T10366. I wasn't writing to this table yet so I didn't build it, but the fact that `bin/storage adjust` would complain slipped my mind.

  - Add the table.
  - Make the tests run `adjust`. This is a little slow (a few extra seconds) but we could eventually move some steps like this to run server-side only.

Test Plan: Ran `bin/storage upgrade -f`, got a clean `adjust`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10366, T10830

Differential Revision: https://secure.phabricator.com/D15744
2016-04-18 07:54:02 -07:00
epriestley
fbfe730452 Support more transactions types in RepositoryEditEngine
Summary:
Ref T10748. This supports more transaction types in the modern editor and improves validation so Conduit benefits.

You can technically create repositories via `diffusion.repository.edit` now, although they aren't very useful.

Test Plan:
  - Used `diffusion.repository.edit` to create and edit repositories.
  - Used `/editpro/` to edit repositories.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10748

Differential Revision: https://secure.phabricator.com/D15740
2016-04-17 16:27:02 -07:00
epriestley
92c50de8aa Rough in the new custom URI panel
Summary: Ref T10748. Ref T10366. No support for editing and no impact on the UI, but get some of the basics in place.

Test Plan: {F1223279}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10366, T10748

Differential Revision: https://secure.phabricator.com/D15742
2016-04-17 16:03:24 -07:00
epriestley
adf42db5ea Trivially implement RepositoryEditEngine and API methods
Summary: Ref T10748. Ref T10337. This technically implements this stuff, but it does not do anything useful yet. This skips all the hard stuff.

Test Plan:
  - Technically used `diffusion.repository.search` to get repository information.
  - Technically used `diffusion.repository.edit` to change a repository name.
  - Used `editpro/` to edit a repository name.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10337, T10748

Differential Revision: https://secure.phabricator.com/D15736
2016-04-17 16:02:13 -07:00
epriestley
d96b6506d7 Disable repository read/write synchronization for now
Summary:
This nearly works but I didn't have time to get back to it and it isn't stable enough to turn on in the cluster yet.

We have enough other stuff going out this week, so just disable it before `stable` gets cut. Should be ready by next week if things go well.

Test Plan: Fetched a Git SSH repo locally.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D15731
2016-04-15 15:39:36 -07:00
June Rhodes
7150aa8e19 Use Conduit in PhabricatorRepositoryGitCommitChangeParserWorker
Summary:
Ref T2783.  This allows this worker to run on a machine different to the one that stores the repository, by routing the execution of Git over Conduit calls.

This API method is super gross, but fixing it isn't straightforward and it runs into other complicated considerations. We can fix it later; for now, just define it as "internal" to limit how much mess this creates.

"Internal" methods do not appear on the console.

Test Plan: Ran `bin/repository reparse --change <commit> --trace` on several commits, saw daemons make a Conduit call instead of running a `git` command.

Reviewers: hach-que, chad

Reviewed By: chad

Subscribers: joshuaspence, Korvin, epriestley

Maniphest Tasks: T2783

Differential Revision: https://secure.phabricator.com/D11874
2016-04-14 04:53:03 -07:00
epriestley
33060d1652 Ignore post-write repository synchronization if no devices are configured
Summary: Fixes T10789. If we aren't configured with a device, we never grabbed a lock in the first place, and should not expect one to be held.

Test Plan: Pushed non-cluster-configured Git SSH repository.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10789

Differential Revision: https://secure.phabricator.com/D15692
2016-04-12 18:31:11 -07:00
epriestley
4244cad990 Move toward multi-master replicated repositories
Summary:
Ref T4292. This mostly implements the locking/versioning logic for multi-master repositories. It is only active on Git SSH pathways, and doesn't actually do anything useful yet: it just does bookkeeping so far.

When we read (e.g., `git fetch`) the logic goes like this:

- Get the read lock (unique to device + repository).
  - Read all the versions of the repository on every other device.
  - If any node has a newer version:
    - Fetch the newer version.
    - Increment our version to be the same as the version we fetched.
- Release the read lock.
- Actually do the fetch.

This makes sure that any time you do a read, you always read the most recently acknowledged write. You may have to wait for an internal fetch to happen (this isn't actually implemented yet) but the operation will always work like you expect it to.

When we write (e.g., `git push`) the logic goes like this:

- Get the write lock (unique to the repository).
  - Do all the read steps so we're up to date.
  - Mark a write pending.
    - Do the actual write.
  - Bump our version and mark our write finished.
- Release the write lock.

This allows you to write to any replica. Again, you might have to wait for a fetch first, but everything will work like you expect.

There's one notable failure mode here: if the network connection between the repository node and the database fails during the write, the write lock might be released even though a write is ongoing.

The "isWriting" column protects against that, by staying locked if we lose our connection to the database. This will currently "freeze" the repository (prevent any new writes) until an administrator can sort things out, since it'd dangerous to continue doing writes (we may lose data).

(Since we won't actually acknowledge the write, I think, we could probably smooth this out a bit and make it self-healing //most// of the time: basically, have the broken node rewind itself by updating from another good node. But that's a little more complex.)

Test Plan:
  - Pushed changes to a cluster-mode repository.
  - Viewed web interface, saw "writing" flag and version changes.
  - Pulled changes.
  - Faked various failures, got sensible states.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4292

Differential Revision: https://secure.phabricator.com/D15688
2016-04-12 08:57:57 -07:00
epriestley
58eef68b7c Rough cut of repository cluster status panel
Summary:
Ref T4292. This adds some very basic cluster/device data to the new management view. Nothing interesting yet.

Also deal with disabled bindings a little more cleanly.

Test Plan: {F1214619}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4292

Differential Revision: https://secure.phabricator.com/D15685
2016-04-12 05:38:10 -07:00
epriestley
0216fac30a Make PullLocal smart about which repositories it should pull
Summary:
Ref T10756. When repositories are properly configured for the cluster (which is hard to set up today), be smart about which repositories are expected to exist on the current host, and only pull them.

This generally allows daemons to pretty much do the right thing no matter how many copies are running, although there may still be some lock contention issues that need to be sorted out.

Test Plan: {F1214483}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10756

Differential Revision: https://secure.phabricator.com/D15682
2016-04-12 05:37:30 -07:00
epriestley
8d6488f290 Fix a typo in bin/repository help update
Summary: Fixes T10741. The workflow is `refs`, not `ref`.

Test Plan: o.O

Reviewers: chad, cspeckmim

Reviewed By: cspeckmim

Maniphest Tasks: T10741

Differential Revision: https://secure.phabricator.com/D15652
2016-04-07 05:39:37 -07:00
epriestley
601aaa5a86 Modularize content sources
Summary:
Ref T10537. For Nuance, I want to introduce new sources (like "GitHub" or "GitHub via Nuance" or something) but this needs to modularize eventually.

Split ContentSource apart so applications can add new content sources.

Test Plan:
This change has huge surface area, so I'll hold it until post-release. I think it's fairly safe (and if it does break anything, the breaks should be fatals, not anything subtle or difficult to fix), there's just no reason not to hold it for a few hours.

- Viewed new module page.
- Grepped for all removed functions/constants.
- Viewed some transactions.
- Hovered over timestamps to get content source details.
- Added a comment via Conduit.
- Added a comment via web.
- Ran `bin/storage upgrade --namespace XXXXX --no-quickstart -f` to re-run all historic migrations.
- Generated some objects with `bin/lipsum`.
- Ran a bulk job on some tasks.
- Ran unit tests.

{F1190182}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10537

Differential Revision: https://secure.phabricator.com/D15521
2016-03-26 11:59:45 -07:00
epriestley
f82db7524b Add a "Build with CircleCI" build step
Summary: Ref T9456. Some rough edges and we can't complete the build yet since I haven't written a webhook, but this mostly seems to be working.

Test Plan:
  - Ran this build on some stuff.
  - Ran a normal HTTP step build to make sure I didn't break that.

{F880301}

{F880302}

{F880303}

Reviewers: chad

Reviewed By: chad

Subscribers: JustinTulloss, joshma

Maniphest Tasks: T9456

Differential Revision: https://secure.phabricator.com/D14286
2016-03-22 12:12:11 -07:00
epriestley
78e36d6b17 Implement DestructibleInterface on GitLFS refs
Summary: Ref T7789. Make sure these get cleaned up when a repository is destroyed.

Test Plan:
  - Created a new repository.
  - Pushed some LFS data to it.
  - Used `bin/remove destroy` to nuke it.
  - Verified the LFS stuff was cleaned up and the underlying files were destroyed (`SELECT * FROM repository_gitlfsref`, etc).

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7789

Differential Revision: https://secure.phabricator.com/D15493
2016-03-18 09:36:54 -07:00
epriestley
f46686ff58 Implement a Git LFS link table and basic batch API
Summary:
Ref T7789. This implements:

  - A new table to store the `<objectHash, filePHID>` relationship between Git LFS files and Phabricator file objects.
  - A basic response to `batch` commands, which return actions for a list of files.

Test Plan:
Ran `git lfs push origin master`, got a little further than previously:

```
epriestley@orbital ~/dev/scratch/poemslocal $ git lfs push origin master
Git LFS: (2 of 1 files) 174.24 KB / 87.12 KB
Git LFS operation "upload/b7e0aeb82a03d627c6aa5fc1bbfd454b6789d9d9affc8607d40168fa18cf6c69" is not supported by this server.
Git LFS operation "upload/b7e0aeb82a03d627c6aa5fc1bbfd454b6789d9d9affc8607d40168fa18cf6c69" is not supported by this server.
```

With `GIT_TRACE=1`, this shows the batch part of the API going through.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7789

Differential Revision: https://secure.phabricator.com/D15489
2016-03-17 17:15:20 -07:00
epriestley
51153a580c Implement "git-lfs-authenticate" over SSH
Summary:
Ref T7789. This implements a (probably) usable "git-lfs-authenticate" on top of the new temporary token infrastructure.

This won't actually do anything yet, since nothing reads the tokens.

Test Plan:
```
$ ./bin/ssh-exec --phabricator-ssh-user admin --ssh-command 'git-lfs-authenticate'
phabricator-ssh-exec: Expected `git-lfs-authenticate <path> <operation>`, but received too few arguments.
```

```
$ ./bin/ssh-exec --phabricator-ssh-user admin --ssh-command 'git-lfs-authenticate x'
phabricator-ssh-exec: Unrecognized repository path "x". Expected a path like "/diffusion/X/" or "/diffusion/123/".
```

```
$ ./bin/ssh-exec --phabricator-ssh-user admin --ssh-command 'git-lfs-authenticate diffusion/22'
Exception: Expected `git-lfs-authenticate <path> <operation>`, but received too few arguments.
```

```
$ ./bin/ssh-exec --phabricator-ssh-user admin --ssh-command 'git-lfs-authenticate diffusion/22 y'
Exception: Git LFS operation "y" is not supported by this server.
```

```
$ ./bin/ssh-exec --phabricator-ssh-user admin --ssh-command 'git-lfs-authenticate diffusion/22 upload'
{"header":{"Authorization":"Basic QGdpdC1sZnM6NmR2bDVreWVsaXNuMmtnNXBtbnZwM3VlaWhubmI1bmI="},"href":"http:\/\/local.phacility.com\/diffusion\/22\/new-callsign-free-repository.git\/info\/lfs"}
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7789

Differential Revision: https://secure.phabricator.com/D15482
2016-03-17 08:08:00 -07:00
epriestley
5520729db3 Remove recently added repository_pathchange key
Summary:
Ref T10560. Reverts D15460. See that task for discussion: we dug up some more information to explain the behavior, and this key was just sort of sidestepping an analyze/cardinality estimate issue on the index.

With proper cardinality estimates it shouldn't be used, so just nuke it.

Test Plan: Ran `bin/storage adjust`, saw key drop.

Reviewers: eadler, chad

Reviewed By: chad

Maniphest Tasks: T10560

Differential Revision: https://secure.phabricator.com/D15486
2016-03-16 14:58:56 -07:00
epriestley
ca4c0db2c1 Add a key to improve Diffusion's cache fill history query
Summary:
Ref T10560. I don't fully understand what MySQL is doing here, but it looks like this key improves the problematic dataset in practice.

(It makes sense that this key helps, I'm just not sure why the two separate keys and the UNION ALL are so bad.)

This key isn't hugely expensive to add, so we can try it and see if there are still issues.

Test Plan: Ran `bin/storage adjust`, saw key added to table. Used `SHOW CREATE TABLE ...` to verify the key exists. Used `EXPLAIN SELECT ...` to make sure MySQL actually uses it.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10560

Differential Revision: https://secure.phabricator.com/D15460
2016-03-10 17:38:36 -08:00
epriestley
abb4c03b47 Remove shouldShowSubscribersProperty() from SubscribableInterface
Summary:
Every caller returns `true`. This was added a long time ago for Projects, but projects are no longer subscribable.

I don't anticipate needing this in the future.

Test Plan: Grepped for this method.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D15409
2016-03-06 06:01:36 -08:00
epriestley
fdca684814 Slightly improve Buildable list in Harbormaster
Summary:
Ref T10457. This makes diffs/revisions show the revision as the buildable title, and commits show the commit as the title.

Previously, the title was "Buildable X".

Also makes icons/colors/labels more consitent.

Test Plan: {F1131885}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10457

Differential Revision: https://secure.phabricator.com/D15355
2016-02-27 07:11:30 -08:00
epriestley
7d4b323da2 Store Almanac "service types" instead of "service classes"
Summary:
Ref T10449. Currently, we store classes (like "AlmanacClusterRepositoryServiceType") in the database.

Instead, store types (like "cluster.repository").

This is a small change, but types are a little more flexible (they let us freely reanme classes), a little cleaner (fewer magic strings in the codebase), and a little better for API usage (they're more human readable).

Make this minor usability change now, before we unprototype.

Also make services searchable by type.

Also remove old Almanac API endpoints.

Test Plan:
  - Ran migration, verified all data migrated properly.
  - Created, edited, rebound, and changed properties of services.
  - Searched for services by service type.
  - Reviewed available Conduit methods.

Reviewers: chad

Reviewed By: chad

Subscribers: yelirekim

Maniphest Tasks: T10449

Differential Revision: https://secure.phabricator.com/D15346
2016-02-26 06:21:50 -08:00
epriestley
ee6070a984 Add a couple of missing needProperties() Almanac calls
Summary: Fixes T10432. I missed these in making properties non-default.

Test Plan: Diffusion now works again in a cluster configuration.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10432

Differential Revision: https://secure.phabricator.com/D15337
2016-02-23 13:50:49 -08:00
epriestley
dc7d0b4a56 Make repository callsigns optional
Summary:
Ref T4245. This could still use a little UI smoothing, but:

  - Don't require a callsign on the create flow (you can add one later in "Edit Basic Information" if you want).
  - Allow existing callsigns to be removed.

Test Plan:
  - Created a new repository with no callsign.
  - Cloned it; pushed to it.
  - Browsed around Diffusion a bunch.
  - Visited a commit URI.
  - Added a callsign to it.
  - Removed the callsign again.
  - Referenced it with `R22` in remarkup.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D15305
2016-02-18 10:36:13 -08:00
epriestley
dfc8f8bcb4 Make callsigns editable on repository basic information
Summary:
Ref T4245. This is a prelude to removing them from the "create" screen.

Currently, if you try to delete the callsign you get an unceremonious database error, but the next diff (or maybe two) will permit that, so I didn't put any "this is required yada yada" text in.

This could also maybe use some big flashing warning lights and a "if you edit this, all your URIs break" but I'll save that for later.

Test Plan: Changed the callsign for a repository.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D15304
2016-02-18 10:34:32 -08:00
epriestley
b63eb09cac Don't require a callsign to set a repository's local path
Summary: Ref T4245. When creating new repositories, set a default local path based on the repository ID instead of callsign.

Test Plan:
  - Created a new repository.
  - Saw it get a reasonable, ID-based local path.
  - Edited a repository to make sure the `applyFinalEffects()` wasn't doing anything whacky.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D15303
2016-02-18 10:33:10 -08:00
epriestley
74a79aa634 Make serving repositories work with alternate URIs
Summary: Ref T4245. Consolidates the URI parsing/rewriting logic so that repositories can be served from either `/diffusion/XYZ/` or `/diffusion/123/`, over both HTTP and SSH.

Test Plan:
  - Pulled a Git repository by ID and callsign over HTTP and SSH.
  - Pulled a Mercurial repository by ID and callsign over HTTP and SSH.
  - Pulled a Subversion repository by ID and callsign over SSH (no HTTP support for SVN).

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D15302
2016-02-18 10:06:03 -08:00
epriestley
c2b8dd28d8 Support ID-based repository URIs, and canonicalize repository URIs
Summary:
Ref T4245. Make `/diffusion/123/` work, but redirect the user to `/diffusion/XYZ/` if the repository has a callsign.

(Right now, every repository has a callsign, so this always redirects.)

Also redirect `/R123:abcdef` if the repository has a callsign.

Also also, move the Pull garbage collector somewhere more sensible.

Test Plan:
  - Added test coverage.
  - Visited `/diffusion/1/`, was redirected.
  - Visited `/diffusion/R1:abcdef`, was redirected.
  - Browsed Diffusion normally.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D15301
2016-02-18 09:56:28 -08:00
epriestley
973b8ace86 Remove dependence on callsigns from bin/commit-hook
Summary:
Ref T4245. Two effects:

  - First, let hooks work for future repositories without callsigns.
  - Second, provide a better error when users push directly to hosted repositories.

Test Plan: Ran `bin/commit-hook PHID-REPO-xxx`.

Reviewers: chad, avivey

Reviewed By: avivey

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D15293
2016-02-17 16:50:36 -08:00
epriestley
9d125b459e Use large text columns to store IP addresses
Summary: Fixes T10259. There was no real reason to do this `ip2long()` stuff in the first place -- it's very slightly smaller, but won't work with ipv6 and the savings are miniscule.

Test Plan:
  - Ran migration.
  - Viewed logs in web UI.
  - Pulled and pushed.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10259

Differential Revision: https://secure.phabricator.com/D15165
2016-02-02 10:13:14 -08:00
Chad Little
fe5cd4ca2c Move FontIcon calls to Icon
Summary: Normalizes all `setFontIcon` calls to `setIcon`.

Test Plan: UIExamples, Almanac, Apps list, etc.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, hach-que, yelirekim

Differential Revision: https://secure.phabricator.com/D15129
2016-01-28 08:48:45 -08:00
epriestley
30473549ac Add a basic pull event log for debugging repository cloning
Summary:
Ref T10228. This is currently quite limited:

  - No UI.
  - No SSH support.

My primary goal is to debug the issue in T10228. In the long run we can expand this to be a bit fancier.

Test Plan:
Made various valid and invalid clones, got sucess responses and not-so-successful responses, viewed the log table for general corresponding messages and broad sanity.

Ran GC via `bin/phd debug trigger`, no issues.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10228

Differential Revision: https://secure.phabricator.com/D15127
2016-01-28 08:18:34 -08:00
epriestley
5c2e49a812 Allow any user to watch any project they can see
Summary:
Ref T6183. Ref T10054. Historically, only members could watch projects because there were some weird special cases with policies. These policy issues have been resolved and Herald is generally powerful enough to do equivalent watches on most objects anyway.

Also puts a "Watch Project" button on the feed panel to make the behavior and meaning more obvious.

Test Plan:
  - Watched a project I was not a member of.
  - Clicked the feed watch/unwatch button.

{F1064909}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T6183, T10054

Differential Revision: https://secure.phabricator.com/D15063
2016-01-19 19:38:30 -08:00
epriestley
155cb1d2c5 Fix two issues with repository monogram regular expressions
Summary: Ref T4245. Fixes T10172. These regular expressions were simply incorrect: they intend `<start> (form one | form two) <end>` but were written as `(<start> form one) | (form two <end>)` which allowed stuff like "R2/R13" to be interpreted as a monogram because it matches `(<start> form one)`.

Test Plan: Parsed commit `ba46ffa6169c` from RTEMS repository, see T10172. Before patch, got an identical trace; after patch, clean import.

Reviewers: chad, avivey

Reviewed By: avivey

Maniphest Tasks: T4245, T10172

Differential Revision: https://secure.phabricator.com/D15049
2016-01-18 09:46:40 -08:00
epriestley
1eab16c395 Move repository URIs to a dedicated index
Summary:
Ref T4705 (there are also some other adjacent related tasks dealing with URIs).

Currently, we issue a "get repositories matching URIs: ..." query by loading every possible repository and then checking their URIs in PHP.

Instead, put URIs in a separate table. I plan for each repository to potentially have multiple URIs soon, so this prepares for that.

Test Plan:
  - Ran migrations.
  - Looked at index table, made sure it appeared sensible.
  - Ran some queries by `uri` to find repositories, found the repositories I expected.
  - Updated the remote URI of a repository, saw queries / index update appropriately.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4705

Differential Revision: https://secure.phabricator.com/D15005
2016-01-13 09:34:31 -08:00
epriestley
a061bd2d09 Parse and display commit authorship date in Git in Diffusion
Summary: Fixes T8826. Git tracks an "author date", which may be different from the "committed date". We don't currently extract/show this; do so.

Test Plan: {F1059235}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8826

Differential Revision: https://secure.phabricator.com/D14995
2016-01-11 09:32:37 -08:00
epriestley
9fb929dff3 Show repositories in global autocomplete dropdown
Summary: Fixes T9523.

Test Plan: {F1059225}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9523

Differential Revision: https://secure.phabricator.com/D14994
2016-01-11 09:32:23 -08:00
epriestley
b848ab87b3 Introduce "local names" for commits
Summary: Ref T4245. Full commit display names (like `rPaaaa`) are going to be obnoxious soon in some cases (e.g., `rPaaaa` becomes `R123:aaaa`, which is much uglier) so reduce how often we show the repository in cases where it isn't really necessary to include it.

Test Plan:
  - Saw no more `rX` on repository list view for Git/Mercurial (still present for Subversion).
  - Saw no more `rX` on various repository detail views, except when referencing other commits (e.g., mentions).
  - Grepped for removed `getShortName()`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D14990
2016-01-11 09:18:30 -08:00
epriestley
96ebd35824 Change repository "Clone/Checkout As" to "Short Name"
Summary:
Ref T4245.

  - Rename "Clone/Checkout As" to "Short Name" in the UI.
  - Allow any repository to have a short name, not just hosted repositories.

Test Plan:
  - Ran migration.
  - Reviewed old transactions, saw they looked good.
  - Edited an existing repository's short name.
  - Gave an imported repository a new short name.
  - Removed a repository's short name.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D14989
2016-01-11 09:17:32 -08:00
epriestley
a06715ccdd Allow repository short names to be used as identifiers
Summary:
Ref T4245. This allows `bin/repository update bread` to work, in addition to `rBREAD`, `R123`, `123`, `BREAD`, etc., if a repository has a short name set.

This primarily affects CLI commands (like `bin/repository`) and Conduit API calls. It has no normal user-facing impact.

Test Plan: Ran `bin/repository update bread` and such.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D14988
2016-01-11 09:17:13 -08:00
epriestley
0b3d10c3da Enforce sensible, unique clone/checkout names for repositories
Summary:
Fixes T7938.

  - Primarily, users can currently shoot themselves in the foot by putting `../../etc/passwd` and other similar nonsense in these fields (this is not dangerous, but also does not work). Require sensible names.
  - Enforce uniqueness so these names can be used in URIs and as identifiers in the future.
  - (This doesn't start actually using them for anything fancy yet.)

Test Plan:
  - Gave several repositories clone names: a valid name, two duplicate names, an invalid, name, some with no names.
  - Ran migrations.
  - Got clean conversion for valid names, appropriate errors for invalid/duplicate names.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7938

Differential Revision: https://secure.phabricator.com/D14986
2016-01-11 02:06:44 -08:00
epriestley
413ca12fda Move commit attachment to a separate CLI command
Summary:
Ref T9319. See D14967. As before, this is making a deeply-buried, complex operation easier to test by providing a CLI command.

This adds `bin/differential attach-commit rXnnnn Dnnnn` to pretend that `rXnnnn` was just committed and matched `Dnnnn`.

Test Plan:
  - Ran `bin/differential attach-commit X Y` for several different values, saw updates in the UI.
  - Faked the message parser to make sure stuff still worked there.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9319

Differential Revision: https://secure.phabricator.com/D14968
2016-01-08 09:27:57 -08:00
epriestley
5592e59f92 Improve error message if local Git working copy directory exists but isn't a working copy
Summary: Fixes T9701. I don't want to try to autofix this because destroying the directory could destroy important files, but we can improve the error message.

Test Plan: Faked a failure, ran `repository update X`, got a more tailored error message.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9701

Differential Revision: https://secure.phabricator.com/D14971
2016-01-08 09:26:40 -08:00
epriestley
0dd947cced Move diff extraction from commits to a separate test with a CLI command
Summary:
Ref T9319. When we discover a commit, we sometimes update the corresponding revision with a "this is the actual committed change" diff and send out a link to the changes between review and commit.

This is currently very difficult to test, because it only happens the first time and you have to either go set up a bunch of objects or add a bunch of special casing to the parser to hit the workflow.

I'm making some changes to how it pulls file content. To make those changes easier to test, first start extracting this stuff so the code can be run with `bin/differential extract ...` instead of needing to do a bunch of more complicated setup steps.

Test Plan:
  - Ran `bin/differential extract ...` to extract diffs from commits.
  - Forced my way through the daemon workflow by faking out a bunch of flags, got a clean extract + attach + update. After this patch, this should rarely be necessary.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9319

Differential Revision: https://secure.phabricator.com/D14967
2016-01-08 09:22:37 -08:00
Fabian Stelzer
e8d3071452 Implement a git blame cache
Summary: Ref T2450. Ref T2453. Add a repository_blamecache table and cache git blame information

Test Plan: View files in Diffusion with enabled blame

Reviewers: fabe, chad, #blessed_reviewers

Reviewed By: chad, #blessed_reviewers

Subscribers: joshuaspence, epriestley

Maniphest Tasks: T2453, T2450

Differential Revision: https://secure.phabricator.com/D10600
2016-01-06 18:43:30 -08:00
epriestley
741118a08f Improve Diffusion behavior for directories with impressive numbers of files
Summary:
Fixes T4366. Two years ago, Facebook put 16,000 files in a directory. Today, the page has nearly loaded.

Paginate large directories.

Test Plan:
  - Viewed home and browse views in Git, Mercurial and Subversion.

I put an artificially small page size (5) on home:

{F1055653}

I pushed 16,000 files to a directory and paged through them. Here's the last page, which rendered in about 200ms:

{F1055655}

Our behavior is a bit better than GitHub here, which shows only the first 1,000 files, disables pagination, and can't retrieve history for the files:

{F1055656}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4366

Differential Revision: https://secure.phabricator.com/D14956
2016-01-06 14:19:55 -08:00
epriestley
9728c65e93 Drive blame generation through diffusion.blame
Summary:
Ref T2450. Ref T9319. This is still a bit messy, but not quite so bad as it was: instead of using a single call to get both blame information and file content, use `diffusion.blame` for blame information.

This will make optimizations to both blame and file content easier.

Test Plan: Viewed a bunch of blame (color on/off, blame on/off).

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T2450, T9319

Differential Revision: https://secure.phabricator.com/D14958
2016-01-06 09:24:21 -08:00