1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-28 09:42:41 +01:00
Commit graph

10204 commits

Author SHA1 Message Date
epriestley
1582bb54f6 Move version numbers to a dedicated "Versions" panel
Summary:
Currently, Version numbers are sort of randomly shown on "All Settings" beacuse we didn't have any better place to put them.

Now that we have modules, expose them as a config module.

Test Plan:
{F906426}

Grepped for "all settings" to look for other references to the old location, but didn't get any relevant hits.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D14327
2015-10-24 08:13:22 -07:00
epriestley
32dc62955a Disable "Send Message" profile action if viewer is logged out
Summary: Fixes T9598.

Test Plan:
  - Used "Send Message" as a logged-in user.
  - Used "Send Message" as a logged-out user. The action was disabled and clicking it popped up a login dialog.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9598

Differential Revision: https://secure.phabricator.com/D14326
2015-10-24 08:13:14 -07:00
epriestley
ad53e7b878 Record how long storage patches took to apply
Summary:
It's hard for us to predict how long patches and migrations will take in the general case since it varies a lot from install to install, but we can give installs some kind of rough heads up about longer patches. I'm planning to just put a sort of hint for things in the changelog, something like this:

{F905579}

To make this easier, start storing how long stuff took. I'll write a little script to dump this into a table for the changelog.

Test Plan:
Ran `bin/storage status`:

{F905580}

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D14320
2015-10-24 05:58:44 -07:00
epriestley
b038041dc6 Prevent duplicate account links from being created by swapping logins and then refreshing the link
Summary:
Fixes T6707. Users can currently do this:

  - Log in to a service (like Facebook or Google) with account "A".
  - Link their Phabricator account to that account.
  - Log out of Facebook, log back in with account "B".
  - Refresh the account link from {nav Settings > External Accounts}.

When they do this, we write a second account link (between their Phabricator account and account "B"). However, the rest of the codebase assumes accounts are singly-linked, so this breaks down elsewhere.

For now, decline to link the second account. We'll permit this some day, but need to do more work to allow it, and the need is very rare.

Test Plan:
  - Followed the steps above, hit the new error.
  - Logged back in to the proper account and did a link refresh (which worked).

{F905562}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T6707

Differential Revision: https://secure.phabricator.com/D14319
2015-10-24 04:50:36 -07:00
epriestley
4afeebe834 Don't store IP addresses in content sources
Summary:
We don't use these for anything, we're inconsistent about recording them, and there's some mild interaction with privacy concerns and data retention. Every other log we store any kind of information in can be given a custom retention policy after recent GC changes.

If we did put this back eventually it would probably be better to store a session identifier anyway, since that's more granular and more detailed.

You can fetch this info out of access logs anyway, too.

Test Plan: Left a couple of comments.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D14315
2015-10-21 12:37:37 -07:00
epriestley
5b619862cb Show a more reasonable status element for pull requests
Summary:
Ref T182. Replace the total mess we had before with a sort-of-reasonable element.

This automatically updates using "javascript".

Test Plan:
{F901983}

{F901984}

Used "Land Revision", saw the land status go from "Waiting" -> "Working" -> "Landed" without having to mash reload over and over again.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T182

Differential Revision: https://secure.phabricator.com/D14314
2015-10-21 11:28:26 -07:00
Giedrius Dubinskas
421c2453e5 Truncate long source lines in Paste search result list snippets
Summary:
An attempt to resolve T9600.

- `PhabricatorPasteQuery` builds truncated snippet when requested using `needSnippet()`.
- `PhabricatorPasteSearchEngine` uses Paste snippet istead of content.
- `PhabricatorSourceCodeView` accepts truncated source and type instead of line limit.

Test Plan: Generated some content for Paste application and also added huge JSON oneliner. Checked Paste application pages in browser.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Maniphest Tasks: T9600

Differential Revision: https://secure.phabricator.com/D14313
2015-10-20 19:07:04 +00:00
epriestley
4c1463eb56 Probably fix bad URI construction for Diffusion symbols
Summary: Ref T9532.

Test Plan: I don't have this configured locally but this seems very likely to be the correct fix. This list should be a list of PHIDs, but is a list of PHIDs followed by one PhabricatorRepository object.

Reviewers: avivey, chad

Reviewed By: chad

Maniphest Tasks: T9532

Differential Revision: https://secure.phabricator.com/D14311
2015-10-20 09:03:47 -07:00
Chad Little
09ab82faef Update Search for handleRequest
Summary: Ref T8628. Updates Search.

Test Plan: Did various searches, saved new queries, reordered, ran new queries.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T8628

Differential Revision: https://secure.phabricator.com/D14268
2015-10-20 09:02:55 -07:00
epriestley
22b9b76079 Fix control state for custom application policies with template types
Summary:
Fixes T9118. When populating some policy controls like "Default Can View" for repositories, we do some special logic to add object policies which are valid for the target object type.

For example, it's OK to set the default policy for an object which has subscribers to "Subscribers".

However, this logic incorrectly //removed// custom policies, so the form input ended up blank.

Instead, provide both object policies and custom policies.

Test Plan:
  - Set default view policy to a custom policy.
  - Hit "Edit" again, saw control correctly reflect custom policy after change.
  - Set default edit policy to a different custom policy.
  - Saved, edited, verified both policies stuck.
  - Set both policies back.
  - Checked some other object types to make sure object policies still work properly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9118

Differential Revision: https://secure.phabricator.com/D14310
2015-10-20 06:56:11 -07:00
Chad Little
bbbda23678 In PHUIInfoView, only show list UI if more than 1 item
Summary: We often just setError as an array even if it's only one error. This just makes the UI a little cleaner in these cases.

Test Plan: Remove all reviewers from a diff, see status error without list styling.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D14308
2015-10-19 20:08:30 -07:00
epriestley
4cb2ec1120 Update support documentation for modern times
Summary:
Basically similar to D13941 but a little more extreme:

  - Really strongly emphasize reproducibility for bug reports, and set users up for rejection if they don't satisfy this.
  - Really strongly emphasize problem descriptions for feature requests, and set users up for rejection.
  - Get rid of various "please give us feedback"; we get plenty of feedback these days.
  - Some modernization tweaks.
  - Split the support document into:
    - Stuff we actually support for free (security / good bug reports / feature requests).
    - Stuff you can pay us for (hosting / consulting / prioritization).
    - A nebulous "community" section, with appropriate (low) expectations that better reflects reality.

My overall goals here are:

  - Set expectations better, so users don't show up in IRC expecting it to be a "great place to get amazing support" or whatever the docs said in 2011.
  - Possibly move the needle slightly on bug reports / feature request quality, maybe.

Test Plan: Read changes carefully.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D14305
2015-10-19 13:29:24 -07:00
epriestley
fbd365d571 Remove scattered links to "Support" document
Summary:
I'm going to do some version of D13941. Clean up extra links to the old document first.

These were just randomly links from various places that we no longer really want feedback on and/or are now better covered by other documents.

Test Plan:
- `grep`
- Reviewed Config/Welcome screen.
- Reviewed `uri.allowed-editor-protocols`.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D14303
2015-10-19 13:27:47 -07:00
Chad Little
ec485de8f9 Restrict Workboard initialization to CAN_EDIT
Summary: Make Workboard initialization more restrictive.

Test Plan: Log out, see "No Workboard", Log in with permissions, see "New Workboard", Log in with notchad, see "No Workboard".

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T7410

Differential Revision: https://secure.phabricator.com/D14306
2015-10-19 13:22:13 -07:00
Chad Little
267e718dfe Don't allow logged out users to initialize a Workboard
Summary: Right now logged out users can enable a workboard on a project.

Test Plan: Log out, view a public project, click on Workboard, get not set up dialog. Click Cancel, return to project details.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D14304
2015-10-19 12:12:52 -07:00
epriestley
d784bf1ea8 Make disk-based setup caches more correct (but slower)
Summary:
Fixes T9599. When APC/APCu are not available, we fall back to a disk-based cache.

We try to share this cache across webserver processes like APC/APCu would be shared in order to improve performance, but are just kind of guessing how to coordinate it. From T9599, it sounds like we don't always get this right in every configuration.

Since this is complicated and error prone, just stop trying to do this. This cache has bad performance anyway (no production install should be using it), and we have much better APC/APCu setup instructions now than we did when I wrote this. Just using the PID is simpler and more correct.

Test Plan:
  - Artificially disabled APC.
  - Reloaded the page, saw all the setup stuff run.
  - Reloaded the page, saw no setup stuff run (i.e., cache was hit).
  - Restarted the webserver.
  - Reloaded the page, saw all the setup stuff run.
  - Reloaded again, got a cache hit.

I don't really know how to reproduce the exact problem with the parent PID not working, but from T9599 it sounds like this fixed the issue and from my test plan we still appear to get correct behavior in the standard/common case.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9599

Differential Revision: https://secure.phabricator.com/D14302
2015-10-19 11:14:46 -07:00
Chad Little
057d62d570 Update Phlux for handleRequest
Summary: Ref T8628. Updates Phlux

Test Plan: New var, list vars, edit vars

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T8628

Differential Revision: https://secure.phabricator.com/D14267
2015-10-18 16:08:20 -07:00
Chad Little
a8e9da4a56 Update Conduit for handleRequest
Summary: Ref T8628. Updates Conduit for handleRequest

Test Plan: Use Conduit, test list, method calls, try a query, post this diff.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T8628

Differential Revision: https://secure.phabricator.com/D14265
2015-10-18 16:07:07 -07:00
Chad Little
4782491470 Fix fatal in Maniphest
Summary:
Fixes T9596.

Was unable to navigate to a task in Maniphest.

Test Plan: navigate to that task.

Reviewers: #blessed_reviewers, epriestley, avivey, tycho.tatitscheff

Reviewed By: avivey, tycho.tatitscheff

Subscribers: tycho.tatitscheff, avivey, Korvin

Maniphest Tasks: T9596

Differential Revision: https://secure.phabricator.com/D14300
2015-10-18 14:43:29 -07:00
tycho
0e8ed0c616 Desactivate subtask when logged out.
Summary: Fixes T9592.

Test Plan: Log out ! Navigates to a task. See the add button grey-ed out !

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin

Maniphest Tasks: T9592

Differential Revision: https://secure.phabricator.com/D14299
2015-10-18 04:22:05 -07:00
epriestley
92a626fc1c Add a basic list view for repository operations
Summary: Ref T182. Nothing fancy, just make these slightly easier to work with.

Test Plan: {F884754}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T182

Differential Revision: https://secure.phabricator.com/D14295
2015-10-16 18:47:05 -07:00
Aviv Eyal
c9e3dd98d1 Fix message about pygments being in $PATH
Test Plan: read it

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D14269
2015-10-16 09:51:39 -07:00
epriestley
cdd5e3f7dd Initialize $assign_phid properly in the "!assign" email action
Summary: If you `!assign cahd` when you meant to `!assign chad`, we'll hit an "Undefined variable: assign_phid" a little further down.

Test Plan: Eyeballed it. See IRC.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D14291
2015-10-16 06:39:31 -07:00
epriestley
4b43667086 Introduce PHUIRemarkupView, a sane way to work with Remarkup
Summary:
Fixes T9273. Remarkup has reasonably good fundamentals but the API is a giant pain to work with.

Provide a `PHUIRemarkupView` to make it easier. This object is way simpler to use by default.

It's not currently as powerful, but we can expand the power level later by adding more setters.

Eventually I'd expect to replace `PhabricatorRemarkupInterface` and `PhabricatorMarkupOneOff` with this, but no rush on those.

I converted a few callsites as a sanity check that it works OK.

Test Plan:
- Viewed remarkup in Passphrase.
- Viewed remarkup in Badges.
- Viewed a Conduit method.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9273

Differential Revision: https://secure.phabricator.com/D14289
2015-10-15 10:20:19 -07:00
epriestley
034ff3c870 Remove "_-_" -> "-" slug behavior
Summary: Fixes T9573. This incorrectly affected Phriction. I could restore it for only projects, but you didn't like the rule very much anyway and I don't feel strongly about it.

Test Plan: Unit tests.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9573

Differential Revision: https://secure.phabricator.com/D14287
2015-10-15 07:04:14 -07:00
Chad Little
f1552f54a0 Link Timeline image to profile
Summary: Ref T9336. Links the timeline photo to user profile. Presume this always exists?

Test Plan: Review a few timelines, click on heads.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T9336

Differential Revision: https://secure.phabricator.com/D14283
2015-10-14 16:28:10 -07:00
epriestley
f3f3d95702 When landing revisions via repository automation, use better metadata
Summary: Ref T182. Make a reasonable attempt to get the commit message, author, and committer data correct.

Test Plan: BEHOLD: rGITTEST810b7f17cd0c909256a45d29a5062fcf417d0489

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T182

Differential Revision: https://secure.phabricator.com/D14280
2015-10-14 10:50:53 -07:00
epriestley
3a91e64897 Preserve "Space" UI control value when editing Passphrase credentials
Summary: Fixes T9568. We just weren't setting this properly so it would default away from the proper value.

Test Plan:
  - Edited a credential in a non-default space, edit form populated properly.
  - Changed "Space", introduced an error, saved form, got error with sticky value for "Space" properly.
  - Saved form with new space value.
  - Created a new credential.

Reviewers: chad

Reviewed By: chad

Subscribers: revi

Maniphest Tasks: T9568

Differential Revision: https://secure.phabricator.com/D14278
2015-10-14 08:15:14 -07:00
epriestley
ac7edf54af Fix bad counting in SQL when enforcing Drydock allocator soft limits
Summary:
Ref T9252. This fixes a bug from D14236. D14272 discusses the observable effects of the bug, primarily that the window for racing is widened from ~a few milliseconds to several minutes under our configuration.

This SQL query is missing a `GROUP BY` clause, so all of the resources get counted as having the same status (specifically, the alphabetically earliest status any resource had, I think). For test cases this often gets the right result since the number of resources may be small and they may all have the same status, but in production this isn't true. In particular, the allocator would sometimes see "35 destroyed resources" (or whatever), when the real counts were "32 destroyed resources + 3 pending resources".

Since this allocator behavior is soft/advisory this didn't cause any actual problems, per se (we do expect races here occasionally), it just made the race very very easy to hit. For example, Drydock in production currently has three pending working copy resources. Although we do expect this to be //possible//, getting 4 resources when the configured limit is 1 should be hard (not lightning strike / cosmic radiaion hard, but "happens once a year" hard).

Also exclude destroyed resources since we never care about them.

Test Plan:
Followed the plan from D14272 and restarted two Harbormaster workers at the same time.

After this patch was applied, they no longer created two different resources (we expect it to be possible for this to happen, just very hard).

We should still be able to force this race by putting something like `sleep(10)` right before the query, then `sleep(10)` right after it. That would prevent the allocators from seeing one another (so they would both think there were no other resources) and push us down the pathway where we exceed the soft limit.

Reviewers: chad, hach-que

Reviewed By: hach-que

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14274
2015-10-14 06:18:10 -07:00
epriestley
083a321dad Fix an issue where newly created Drydock resources could be improperly acquired
Summary:
Ref T9252. This is mostly a fix for an edge case from D14236. Here's the setup:

  - There are no resources.
  - A request for a new resource arrives.
  - We build a new resource.

Now, if we were leasing an existing resource, we'd call `canAcquireLeaseOnResource()` before acquiring a lease on the new resource.

However, for new resources we don't do that: we just acquire a lease immediately. This is wrong, because we now allow and expect some resources to be unleasable when created.

In a more complex workflow, this can also produce the wrong result and leave the lease acquired sub-optimally (and, today, deadlocked).

Make the "can we acquire?" pathway consistent for new and existing resources, so we always do the same set of checks.

Test Plan:
  - Started daemons.
  - Deleted all working copy resources.
  - Ran two working-copy-using build plans at the same time.
  - Before this change, one would often [1] acquire a lease on a pending resource which never allocated, then deadlock.
  - After this change, the same thing happens except that the lease remains pending and the work completes.

[1] Although the race this implies is allowed (resource pool limits are soft/advisory, and it is expected that we may occasionally run over them), it's MUCH easier to hit right now than I would expect it to be, so I think there's probably at least one more small bug here somewhere. I'll see if I can root it out after this change.

Reviewers: chad, hach-que

Reviewed By: hach-que

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14272
2015-10-14 06:16:21 -07:00
epriestley
4169d7bfd5 Fix an issue where Harbormaster might cycle while saving
The way custom field interact with storage is a little odd, and can send us
down a bad path when applying external effect while saving changes.
2015-10-14 02:56:39 -07:00
epriestley
43bee4562c If the stars align, make "Land Revision" kind of work
Summary:
Ref T182. If 35 other things are configured completely correctly, make it remotely possible that this button may do something approximating the thing that the user wanted.

This primarily fleshes out the idea that "operations" (like landing, merging or cherry-picking) can have some beahavior, and when we run an operation we do whatever that behavior is instead of just running `git show`.

Broadly, this isn't too terrible because Drydock seems like it actually works properly for the most part (???!?!).

Test Plan: {F876431}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T182

Differential Revision: https://secure.phabricator.com/D14270
2015-10-13 15:46:30 -07:00
epriestley
b4af57ec51 Rough cut of DrydockRepositoryOperation
Summary:
Ref T182. This doesn't do anything interesting yet and is mostly scaffolding, but here's roughly the workflow. From previous revision, you can configure "Repository Automation" for a repository:

{F875741}

If it's configured, a new "Land Revision" button shows up:

{F875743}

Once you click it you get a big warning dialog that it won't work, and then this shows up at the top of the revision (completely temporary/placeholder UI, some day a nice progress bar or whatever):

{F875747}

If you're lucky, the operation eventually sort of works:

{F875750}

It only runs `git show` right now, doesn't actually do any writes or anything.

Test Plan:
  - Clicked "Land Revision".
  - Watched `phd debug task`.
  - Saw it log `git show` to output.
  - Verified operation success in UI (by fiddling URL, no way to get there normally yet).

Reviewers: chad

Reviewed By: chad

Subscribers: revi

Maniphest Tasks: T182

Differential Revision: https://secure.phabricator.com/D14266
2015-10-13 15:46:12 -07:00
epriestley
df5a031b54 Allow "Repository Automation" to be configured for repositories
Summary:
Ref T182. This allows you to assign blueprints that a repository can use to perform working copy operations. Eventually, this will support "merge this" in Differential, etc.

This is just UI for now, with no material effects.

Most of this diff is just taking logic that was in the existing "Blueprints" CustomField and putting it in more general places so Diffusion (which does not use CustomFields) can also access it.

Test Plan:
  - Configured repository automation for a repository.
  - Removed repository automation for a repository.

Reviewers: chad

Reviewed By: chad

Subscribers: avivey

Maniphest Tasks: T182

Differential Revision: https://secure.phabricator.com/D14259
2015-10-13 15:45:59 -07:00
Chad Little
6ff1354ac1 Fix errors when mentioning others in Ponder
Summary: Fixes T9552. We need to set a questionID and the question object (for policy) when initializing a new Answer.

Test Plan: Write an answer that mentions another user.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T9552

Differential Revision: https://secure.phabricator.com/D14263
2015-10-13 09:09:07 -07:00
epriestley
0b6c031042 Work around an issue with custom "users" fields in Maniphest
Summary:
Fixes T9558. The recent changes to validate PHID fields don't work cleanly with this gross hack.

This can probably be unwound now but it will definitely get fixed in T9132 so I may just wait for that.

Test Plan: Edited a custom "users" field in Maniphest. This should only affect Maniphest because it has a weird hack.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9558

Differential Revision: https://secure.phabricator.com/D14264
2015-10-13 08:41:49 -07:00
epriestley
3f3626c11a Write some documentation about Drydock security and repository automation
Summary: Ref T182. Ref T9519. Some of what this describes doesn't exist yet, but should soon.

Test Plan: Read documentation.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T182, T9519

Differential Revision: https://secure.phabricator.com/D14258
2015-10-12 17:54:11 -07:00
Christopher Speck
812c41a18a Conditionally use hg files vs. hg locate depending on version of Mercurial
Summary:
In Mercurial 3.2 the `locate` command was deprecated in favor of `files` command. This change updates the DiffusionLowLevelMercurialPathsQuery command to conditionally use `locate` or `files` based on the version of Mercurial used.

Closes T7375

Test Plan:
My test/develop Phabricator instance is setup to run Mercurial 3.5.1.

The test procedure to verify valid file listings are being returned:
 1. I navigated to `http://192.168.0.133/conduit/method/diffusion.querypaths/`
 2. I populated the following fields:
  - path: `"/"`
  - commit: `"d721d5b57fc9ef72e47ff9d4e0c583d74a46590c"`
  - callsign: `"HGTEST"`
 3. I submitted request and verified that result contained all files in the repository:
```
{
  "0": "README",
  "1": "alpha/beta/trifle",
  "2": "test/Chupacabra.cow",
  "3": "test/socket.ks"
}
```

I repeated the above steps after setting up Mercurial 2.6.2, which I installed in the following manner:
 1. I downloaded Mercurial 2.6.2 source and run `make local` which will only compile it to work from its own directory (`/opt/mercurial-2.6.2`)
 2. I linked `/usr/local/bin/hg -> /opt/mercurial-2.6.2/hg` (there's also a `/usr/bin/hg` which is a link to `/usr/local/bin/hg`)
 3. I navigated to my home directory and verify that `hg --version` returns 2.6.2.
 4. I restarted phabricator services (probably unnecessary).

With the Multimeter application active
 1. I verified that `/usr/local/bin/hg` referred to version 2.6
 2. I ran the same conduit call from the conduit application
 3. I verified that `http://192.168.0.133/multimeter/?type=2&group=label` incremented values for `bin.hg locate`.
 4. I swapped out mercurial versions for 3.5.1
 5. I ran the same conduit call from the conduit application
 6. I verified that `http://192.168.0.133/multimeter/?type=2&group=label` incremented values for `bin.hg files`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Maniphest Tasks: T7375

Differential Revision: https://secure.phabricator.com/D14253
2015-10-12 17:50:26 -07:00
epriestley
cd8be8106b Improve ruleset for generating project hashtags
Summary:
Ref T9551. We currently use the same logic for generating project hashtags and Phriction slugs, but should be a little more conservative with project hashtags.

Stop them from generating with stuff that won't parse in a "Reviewers:" field or generally in commments (commas, colons, etc).

Test Plan:
Created a bunch of projects with nonsense in them and saw them generate pretty reasonable hashtags.

{F873456}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9551

Differential Revision: https://secure.phabricator.com/D14261
2015-10-12 17:02:58 -07:00
epriestley
3ff5ca789a Fix /tag/aa%20bb project URIs
Summary:
Ref T9551. To set things up:

  - Name a project `aa bb`. This will have the tag `aa_bb`.
  - Try to visit `/tag/aa%20bb`.

Here's what happens now:

  - You get an Aphront redirect error as it tries to add the trailing `/`. Add `phutil_escape_uri()` so that works again.
  - Then, you 404, even though this tag is reasonably equivalent to the real project tag and could be redirected. Add a fallback to lookup, resolve, and redirect if we can find a hit for the tag.

This also fixes stuff like `/tag/AA_BB/`.

Test Plan: Visited URIs like `/tag/aa%20bb`, `/tag/aa%20bb/`, `/tag/Aa_bB/`, etc. None of them worked before and now they all do.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9551

Differential Revision: https://secure.phabricator.com/D14260
2015-10-12 17:02:42 -07:00
epriestley
1bdf225354 Use Drydock authorizations when acquiring leases
Summary:
Ref T9519. When acquiring leases on resources:

  - Only consider resources created by authorized blueprints.
  - Only consider authorized blueprints when creating new resources.
  - Fail with a tailored error if no blueprints are allowed.
  - Fail with a tailored error if missing authorizations are causing acquisition failure.

One somewhat-substantial issue with this is that it's pretty hard to figure out from the Harbormaster side. Specifically, the Build step UI does not show field value anywhere, so the presence of unapproved blueprints is not communicated. This is much more clear in Drydock. I'll plan to address this in future changes to Harbormaster, since there are other related/similar issues anyway.

Test Plan: {F872527}

Reviewers: hach-que, chad

Reviewed By: chad

Maniphest Tasks: T9519

Differential Revision: https://secure.phabricator.com/D14254
2015-10-12 17:02:35 -07:00
Chad Little
dac16264e4 Update metamta for handleRequest
Summary: Updates metamta for handleRequest

Test Plan: Unable to test this, but looks safe?

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D14256
2015-10-12 12:02:11 -07:00
Chad Little
44e61a2397 Update home for handleRequest
Summary: Updates /home/ for handleRequest

Test Plan: Visit /home/creat/

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D14257
2015-10-12 12:01:02 -07:00
Chad Little
02f42628c3 Update Harbormaster for handleRequest
Summary: Updates Harbormaster for handleRequest over processRequest

Test Plan: Went through various Harbormaster areas, buildables, actions.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D14255
2015-10-12 11:39:01 -07:00
Chad Little
bb37ad65a2 Update Differential for handleRequest
Summary: Moves from processRequest to handleRequest.

Test Plan: New diff, edit diff, leave comment, view list, browse revisions, etc.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D14252
2015-10-11 08:18:42 -07:00
epriestley
2f6d3119f5 Rough cut of "Blueprint Authorizations"
Summary:
Ref T9519. This is like 80% of the way there and doesn't fully work yet, but roughly shows the shape of things to come. Here's how it works:

First, there's a new custom field type for blueprints which works like a normal typeahead but has some extra logic. It's implemented this way to make it easy to add to Blueprints in Drydock and Build Plans in Harbormaster. Here, I've added a "Use Blueprints" field to the "WorkingCopy" blueprint, so you can control which hosts the working copies are permitted to allocate on:

{F869865}

This control has a bit of custom rendering logic. Instead of rendering a normal list of PHIDs, it renders an annotated list with icons:

{F869866}

These icons show whether the blueprint on the other size of the authorization has approved this object. Once you have a green checkmark, you're good to go.

On the blueprint side, things look like this:

{F869867}

This table shows all the objects which have asked for access to this blueprint. In this case it's showing that one object is approved to use the blueprint since I already approved it, but by default new requests come in here as "Authorization Requested" and someone has to go approve them.

You approve them from within the authorization detail screen:

{F869868}

You can use the "Approve" or "Decline" buttons to allow or prevent use of the blueprint.

This doesn't actually do anything yet -- objects don't need to be authorized in order to use blueprints quite yet. That will come in the next diff, I just wanted to get the UI in reasonable shape first.

The authorization also has a second piece of state, which is whether the request from the object is active or inactive. We use this to keep track of the authorization if the blueprint is (maybe temporarily) deleted.

For example, you might have a Build Plan that uses Blueprints A and B. For a couple days, you only want to use A, so you remove B from the "Use Blueprints: ..." field. Later, you can add B back and it will connect to its old authorization again, so you don't need to go re-approve things (and if you're declined, you stay declined instead of being able to request authorization over and over again). This should make working with authorizations a little easier and less labor intensive.

Stuff not in this diff:

  - Actually preventing any allocations (next diff).
  - Probably should have transactions for approve/decline, at least, at some point, so there's a log of who did approvals and when.
  - Maybe should have a more clear/loud error state when no blueprints are approved?
  - Should probably restrict the typeahead to specific blueprint types.

Test Plan:
  - Added the field.
  - Typed some stuff into it.
  - Saw the UI update properly.
  - Approved an authorization.
  - Declined an authorization.
  - Saw active authorizations on a blueprint page.
  - Didn't see any inactive authroizations there.
  - Clicked "View All Authorizations", saw all authorizations.

Reviewers: chad, hach-que

Reviewed By: chad

Maniphest Tasks: T9519

Differential Revision: https://secure.phabricator.com/D14251
2015-10-10 07:15:25 -07:00
Christopher Speck
32d4ae8cb2 Added an intercept to Mercurial's capabilities command to remove bundle2.
Summary:
If Mercurial 3.4+ is used to host repositories in Phabricator, any clients using 3.5+ will receive an exception after the bundle is pushed up. Clients will also fail to update phases for changesets pushed up.

Before directly responding to mercurial clients with all capabilities, this change filters out the 'bundle2' capability so the client negotiates using a legacy bundle wire format instead.

Test Plan:
Server: Mercurial 3.5
Client: Mercurial 3.4

Test with both HTTP and SSH protocols:
1. Create a local commit on client
2. Push commit to server
3. Verify the client emits something like:
```
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
```

Closes T9450

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Maniphest Tasks: T9450

Differential Revision: https://secure.phabricator.com/D14241
2015-10-10 07:14:48 -07:00
epriestley
5a874ba0a8 Put cows and figlet bannners in <pre> in HTML mail bodies
Summary: Fixes T9538. Ref T9408. `cowsay` and `figlet` Remarkup rules are being mangled in HTML mail right now. Put them in <pre> to unmangle them.

Test Plan:
Sent myself a cow + figlet in mail.

Used `bin/mail show-outbound --id ... --dump-html > dump.html` + open that HTML file in Safari to preview HTML mail.

Saw linebreaks and monospaced formatting.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9538, T9408

Differential Revision: https://secure.phabricator.com/D14248
2015-10-08 20:03:15 -07:00
Chad Little
4549afbdce Link Ponder Answer header to user
Summary: Fixes T9509

Test Plan: View a Ponder Question, hover over answerer name, get URL.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T9509

Differential Revision: https://secure.phabricator.com/D14238
2015-10-06 14:15:50 -07:00
epriestley
2bfa0e087e Improve consistency and Harbormaster integration of Diffusion
Summary:
Ref T9123. Two major Harbormaster-related UI changes in Diffusion:

  - Tags table now shows tag build status.
  - Branches table now shows branch build status.

Then some minor consistency / qualtiy of life changes:

  - Picked a nicer looking "history" icon?
  - Branches table now uses the same "history" icon as other tables.
  - Tags table now has a "history" link.
  - Browse table now has a "history" link.
  - Dates now use more consistent formatting.
  - Column order is now more consistent.
  - Use of style is now more consistent.

Test Plan:
{F865056}

{F865057}

{F865058}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9123

Differential Revision: https://secure.phabricator.com/D14242
2015-10-06 07:38:15 -07:00
epriestley
03fea70497 Fix some header formatting in bin/storage probe
Summary: Ref T9514. I missed these when I swapped out the console stuff recently.

Test Plan: Ran `bin/storage probe`, saw bold instead of escape sequences.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9514

Differential Revision: https://secure.phabricator.com/D14240
2015-10-06 07:38:09 -07:00
epriestley
4d5278af11 Put Drydock build steps into their own group in Harbormaster
Summary: Ref T9252. Move these into a new "Drydock" group.

Test Plan: Clicked "Add Build Step", saw Drydock steps in a Drydock group.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14237
2015-10-05 15:59:35 -07:00
epriestley
ee937e99fb Fix unbounded expansion of allocating resource pool
Summary:
Ref T9252. I think there's a more complex version of this problem discussed elsewhere, but here's what we hit today:

  - 5 commits land at the same time and trigger 5 builds.
  - All of them go to acquire a working copy.
  - Working copies have a limit of 1 right now, so 1 of them gets the lease on it.
  - The other 4 all trigger allocation of //new// working copies. So now we have: 1 active, leased working copy and 4 pending, leased working copies.
  - The 4 pending working copies will never activate without manual intervention, so these 4 builds are stuck forever.

To fix this, prevent WorkingCopies from giving out leases until they activate. So now the leases won't acquire until we know the working copy is good, which solves the first problem.

However, this creates a secondary problem:

  - As above, all 5 go to acquire a working copy.
  - One gets it.
  - The other 4 trigger allocations, but no longer acquire leases. This is an improvement.
  - Every time the leases update, they trigger another allocation, but never acquire. They trigger, say, a few thousand allocations.
  - Eventually the first build finishes up and the second lease acquires the working copy. After some time, all of the builds finish.
  - However, they generated an unboundedly large number of pending working copy resources during this time.

This is technically "okay-ish", in that it did work correctly, it just generated a gigantic mess as a side effect.

To solve this, at least for now, provide a mechanism to impose allocation rate limits and put a cap on the number of allocating resources of a given type. As hard-coded, this the greater of "1" or "25% of the active resources in the pool".

So if there are 40 working copies active, we'll start allocating up to 10 more and then cut new allocations off until those allocations get sorted out. This prevents us from getting runaway queues of limitless size.

This also imposes a total active working copy resource limit of 1, which incidentally also fixes the problem, although I expect to raise this soon.

These mechanisms will need refinement, but the basic idea is:

  - Resources which aren't sure if they can actually activate should wait until they do activate before allowing leases to acquire them. I'm fairly confident this rule is a reasonable one.
  - Then we limit how many bookkeeping side effects Drydock can generate once it starts encountering limits.

Broadly, some amount of mess is inevitable because Drydock is allowed to try things that might not work. In an extreme case we could prevent this mess by setting all these limits at "1" forever, which would degrade Drydock to effectively be a synchronous, blocking queue.

The idea here is to put some amount of slack in the system (more than zero, but less than infinity) so we get the performance benefits of having a parallel, asyncronous system without a finite, manageable amount of mess.

Numbers larger than 0 but less than infinity are pretty tricky, but I think rules like "X% of active resources" seem fairly reasonable, at least for resources like working copies.

Test Plan:
Ran something like this:

```
for i in `seq 1 5`; do sh -c '(./bin/harbormaster build --plan 10 rX... &) &'; done;
```

Saw 5 plans launch, acquire leases, proceed in an orderly fashion, and eventually finish successfully.

Reviewers: hach-que, chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14236
2015-10-05 15:59:16 -07:00
epriestley
b2e89a9e48 Fix several error handling issues with Subversion commits in Diffusion
Summary:
Ref T9513. I checked this briefly but didn't do a very thorough job of it.

  - Don't try to query merges for Subversion, since it doesn't support them.
  - Fix up "existsquery" to work properly (and efficiently) for both hosted and imported repositories.
  - Fix up "parentsquery" to have similar behavior on invalid commits to other VCSes (throw an exception).

Test Plan:
  - No more merges warning on SVN.
  - Hosted SVN gets the right exists result now.
  - Visiting "r23980283789287" now 404's instead of "not parsed yet".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9513

Differential Revision: https://secure.phabricator.com/D14239
2015-10-05 15:57:41 -07:00
epriestley
de2bbfef7d Allow PhabricatorWorker->queueTask() to take full $options
Summary:
Ref T9252. Currently, `queueTask()` accepts `$priority` as its third argument. Allow it to take a full range of `$options` instead. This API just never got updated after we expanded avialable options.

Arguably this whole API should be some kind of "TaskQueueRequest" object but I'll leave that for another day.

Test Plan:
  - Grepped for `queueTask()` and verified no other callsites are affected by this API change.
  - Ran some daemons.
  - See also next diff.

Reviewers: hach-que, chad

Reviewed By: hach-que, chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14235
2015-10-05 09:46:29 -07:00
epriestley
4cf1270ecd In Harbormaster, make sure artifacts are destroyed even if a build is aborted
Summary:
Ref T9252. Currently, Harbormaster and Drydock work like this in some cases:

  # Queue a lease for activation.
  # Then, a little later, save the lease PHID somewhere.
  # When the target/resource is destroyed, destroy the lease.

However, something can happen between (1) and (2). In Drydock this window is very short and the "something" would have to be a lighting strike or something similar, but in Harbormaster we wait until the resource activates to do (2) so the window can be many minutes long. In particular, a user can use "Abort Build" during those many minutes.

If they do, the target is destroyed but it doesn't yet have a record of the artifact, so the artifact isn't cleaned up.

Make these things work like this instead:

  # Create a new lease and pre-generate a PHID for it.
  # Save that PHID as something that needs to be cleaned up.
  # Queue the lease for activation.
  # When the target/resource is destroyed, destroy the lease if it exists.

This makes sure there's no step in the process where we might lose track of a lease/resource.

Also, clean up and standardize some other stuff I hit.

Test Plan:
  - Stopped daemons.
  - Restarted a build in Harbormaster.
  - Stepped through the build one stage at a time using `bin/worker execute ...`.
  - After the lease was queued, but before it activated, aborted the build.
  - Processed the Harbormaster side of things only.
  - Saw the lease get destroyed properly.

Reviewers: chad, hach-que

Reviewed By: hach-que

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14234
2015-10-05 05:58:53 -07:00
epriestley
0db86cce7d Improve Diffusion behavior for no-longer-existing commits
Summary:
Ref T9028. When users push a commit, then later delete it (e.g., by deleting the branch which contained it) we currently explode when trying to view it.

Instead, degrade gradually if some information is not available.

Test Plan:
  - Looked at valid commits with parents, refs, branches and merges.
  - Looked at invalid commits.
  - Looked at a previously valid, now-deleted + gc'd commit:

{F859273}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9028

Differential Revision: https://secure.phabricator.com/D14227
2015-10-02 16:11:03 -07:00
epriestley
14d6325394 Acccept any HTTP 2xx status as success in Harbormaster
Summary: Ref T9478. This should probably be configurable eventually, but for now treat any 200-block status as success. Also show the result code.

Test Plan:
  - Hit a bad URI, saw "HTTP 503" + failure.
  - Hit a good URI, saw "HTTP 200" + success.

Reviewers: chad, hach-que

Reviewed By: chad, hach-que

Maniphest Tasks: T9478

Differential Revision: https://secure.phabricator.com/D14226
2015-10-02 09:17:51 -07:00
epriestley
9c798e5cca Provide bin/garbage for interacting with garbage collection
Summary:
Fixes T9494. This:

  - Removes all the random GC.x.y.z config.
  - Puts it all in one place that's locked and which you use `bin/garbage set-policy ...` to adjust.
  - Makes every TTL-based GC configurable.
  - Simplifies the code in the actual GCs.

Test Plan:
  - Ran `bin/garbage collect` to collect some garbage, until it stopped collecting.
  - Ran `bin/garbage set-policy ...` to shorten policy. Saw change in web UI. Ran `bin/garbage collect` again and saw it collect more garbage.
  - Set policy to indefinite and saw it not collect garabge.
  - Set policy to default and saw it reflected in web UI / `collect`.
  - Ran `bin/phd debug trigger` and saw all GCs fire with reasonable looking queries.
  - Read new docs.

{F857928}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9494

Differential Revision: https://secure.phabricator.com/D14219
2015-10-02 09:17:24 -07:00
epriestley
bb4667cb84 Fix WorkingCopy step to read correct commit variables
Summary: Ref T9252. This variable was always wrong but we fell back to just resetting to `HEAD` before. Use the correct variable name.

Test Plan: Verified variable name.

Reviewers: chad, hach-que

Reviewed By: hach-que

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14224
2015-10-02 06:37:17 -07:00
epriestley
2728a9f964 Allow builds to have parameters
Summary: Ref T9352. See D13635. Build targets can have variables already, but let builds have them too. This mostly enables future use cases (sub-builds, more sophisticated build triggers).

Test Plan: With a custom Herald rule + action like the one in T9352, updated a revision and saw it generate multiple builds with varying parameters.

Reviewers: chad, hach-que

Reviewed By: hach-que

Maniphest Tasks: T9352

Differential Revision: https://secure.phabricator.com/D14222
2015-10-02 06:32:08 -07:00
epriestley
878a493301 Begin standardizing garbage collectors
Summary: Ref T9494. Improve support infrastructure for garbage collectors.

Test Plan:
  - Ran `bin/phd debug trigger`, saw collectors execute.

{F857852}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9494

Differential Revision: https://secure.phabricator.com/D14218
2015-10-01 16:58:43 -07:00
epriestley
e431ab2189 Use getPhobjectClassConstant() to access class constants
Summary: Ref T9494. Depends on D14216. Remove 10 copies of this code.

Test Plan: Ran `arc unit --everything`, browsed Config > Modules, clicked around Herald / etc.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9494

Differential Revision: https://secure.phabricator.com/D14217
2015-10-01 16:56:21 -07:00
epriestley
c95fcb8970 Add a little Drydock documentation
Summary: Ref T9252. Provide some general descriptions of Drydock in the docs.

Test Plan: Reading.

Reviewers: hach-que, chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14215
2015-10-01 16:55:24 -07:00
epriestley
4496176924 Add staging area support to Harbormaster/Drydock + various fixes
Summary:
Ref T9252. This primarily allows Harbormaster to request (and Drydock to fulfill) working copies with a patch from a staging area. Doing this means we can do builds on in-review changes from `arc diff`.

This is a little cobbled-together but should basically work.

Also fix some other issues:

  - Yielded, awakend workers are fine to update but could complain.
  - We can't log slot lock failures to resources if we don't end up saving them.
  - Killing the transaction would wipe out the log.
  - Fix some TODOs, etc.

Test Plan: Ran Harbormaster builds on a local revision.

Reviewers: hach-que, chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14214
2015-10-01 16:55:01 -07:00
epriestley
d4a0b1c870 Remove names from Drydock resources
Summary:
Ref T9252. Long ago you sometimes manually created resources, so they had human-enterable names. However, users never make resources manually any more, so this field isn't really useful any more.

In particular, it means we write a lot of untranslatable strings like "Working Copy" to the database in the default locale. Instead, do the call at runtime so resource names are translatable.

Also clean up a few minor things I hit while kicking the tires here.

It's possible we might eventually want to introduce a human-choosable label so you can rename your favorite resources and this would just be a default name. I don't really have much of a use case for that yet, though, and I'm not sure there will ever be one.

Test Plan:
  - Restarted a Harbormaster build, got a clean build.
  - Released all leases/resources, restarted build, got a clean build with proper resource names.

Reviewers: hach-que, chad

Reviewed By: hach-que, chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14213
2015-10-01 08:13:43 -07:00
epriestley
b219bcfb3d Improve error and exception handling for Drydock leases
Summary:
Ref T9252. See companion change in D14211. This does the same thing for leases.

Particularly, most of the TODOs about error handling can just be removed because they'll do the right things by default now.

This and D14211 also move slot lock release to after resource destruction. This feels cleaner than trying to release early at release/break.

Test Plan: Restarted a Harbormaster build, got a clean build result. This needs more vetting but I'll clean up any issues as I hit them.

Reviewers: chad, hach-que

Reviewed By: hach-que

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14212
2015-10-01 08:13:20 -07:00
epriestley
e589d15231 Improve error and exception handling for Drydock resources
Summary:
Ref T9252. Currently, error handling behavior isn't great and a lot of errors aren't dealt with properly. Try to improve this by making default behaviors better:

  - Yields, slot lock exceptions, and aggregate or proxy exceptions containing an excpetion of these types turn into yields.
  - All other exceptions are considered permanent failures. They break the resource and

This feels a little bit "magical" but I want to try to get the default behaviors to align reasonably well with expectations so that blueprints mostly don't need to have a ton of error handling. This will probably need at least some refinement down the road, but it's a reasonable rule for all exception/error conditions we currently have.

Test Plan: I did a clean build, but haven't vetted this super thoroughly. Next diff will do the same thing to leases, then I'll work on stabilizing this code better.

Reviewers: chad, hach-que

Reviewed By: hach-que

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14211
2015-10-01 08:12:51 -07:00
epriestley
6b775e6090 Add more Drydock log types and some additional logging
Summary: Ref T9252. Add a bit more logging and improve some behaviors.

Test Plan: Restarted a build, got a good result.

Reviewers: chad, hach-que

Reviewed By: hach-que

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14210
2015-10-01 08:11:42 -07:00
epriestley
4ac82be5ed Merge the DrydockLease workers into a single worker
Summary:
Ref T9252. This is the same as D14201, but for lease stuff instead of resource stuff.

This one is a little heavier but still feels pretty reasonable to me at the end of the day (worker is <1K lines and has a ton of comment stuff).

Also fixes a few random bugs I hit in the task queue.

Test Plan:
  - Restarted some Harbormaster builds, saw them go through cleanly.
  - Released pre-activation resources/leases.
  - Probably still kinda buggy but I'll iron the details out over time.

Logs are starting to look somewhat plausible:

{F855747}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14202
2015-10-01 08:11:02 -07:00
epriestley
91e5ca0ee2 Merge the DrydockResource workers into a single worker
Summary:
Ref T9252. Currently, Drydock Leases and Resources have several workers:

  - Resources: ResourceWorker, ResourceUpdateWorker, ResourceDestroyWorker
  - Leases: AllocatorWorker, LeaseWorker, LeaseUpdateWorker, LeaseDestroyWorker

This is kind of a lot of stuff, and it creates some problems.

In particular, leases and resources in early lifecycle phases (pending/allocating/acquiring) can't process commands yet, because that code is only in the "UpdateWorker" classes. If they aren't able to move forward because of a bug, they also can't be released because they can't react to the release command until later in their lifecycle. This creates a soft hang where I have to go wipe stuff out of the database since there's no other way to get rid of it.

Instead, I want leases and resources to be releasable from any (pre-release / pre-destroy) phase of their lifecycle. To support this, all the workers before the "UpdateWorker" need to be able to process commands.

A second, similar issue is that logging and exception handling behaviors are underpowered right now. Elsewhere I began improving this, but ran into issues where all of the workers needed to share very similar exception code. Merging them will make this future change simpler.

This diff fixes this for resources: it merges the Worker, UpdateWorker and DestroyWorker logic into UpdateWorker and throws away the other two workers.

Test Plan: Nothing substantive yet, see next diff. I'll do the same thing for Leases, then test both more thoroughly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14201
2015-10-01 08:10:40 -07:00
epriestley
8bf5905024 Add Drydock log types and more logging
Summary: Ref T9252. Make log types modular so they can be translated and have complicated rendering logic if necessary (currently, none have this).

Test Plan: {F855330}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14198
2015-10-01 08:10:07 -07:00
epriestley
06f9272502 Garbage collect Drydock logs after 30 days
Summary:
Ref T9252. Drydock logs are almost exclusively useful as a diagnostic tool for debugging immediate problems, so GC them fairly aggressively.

(I expect 99% of the usefulness of these logs to be within the first 24 hours, basically "why isn't my thing working". I can't really think of any cases where having old logs would be useful.)

Test Plan:
  - Ran GC, saw it hit the log table (with no effect).
  - Changed TTL from 30 days to 30 seconds, ran GC, saw it wipe recent logs.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14197
2015-10-01 08:09:27 -07:00
epriestley
2ef5b5321d Move Drydock logs to PHIDs and increased structure
Summary:
Ref T9252. Several general changes here:

  - Moves logs to use PHIDs instead of IDs. This generally improves flexibility (for example, it's a lot easier to render handles).
  - Adds `blueprintPHID` to logs. Although you can usually figure this out from the leasePHID or resourcePHID, it lets us query relevant logs on Blueprint views.
  - Instead of making logs a top-level object, make them strictly a sub-object of Blueprints, Resources and Leases. So you go Drydock > Lease > Logs, etc., to get to logs.
    - I might restore the "everything" view eventually, but it doesn't interact well with policies and I'm not sure it's very useful. A policy-violating `bin/drydock log` might be cleaner.
  - Policy-wise, we always show you that logs exist, we just don't show you log content if it's about something you can't see. This is similar to seeing restricted handles in other applications.
  - Instead of just having a message, give logs "type" + "data". This will let logs be more structured and translatable. This is similar to recent changes to Herald which seem to have worked well.

Test Plan:
Added some placeholder log writes, viewed those logs in the UI.

{F855199}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14196
2015-10-01 08:06:23 -07:00
epriestley
9d997df964 Reset Drydock git working copies better
Summary: Ref T9252. We're currently resetting to the local branch, but should be resetting to the origin branch.

Test Plan: Restarted a build, had it run `git show`, saw proper HEAD.

Reviewers: chad

Reviewed By: chad

Subscribers: hach-que

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14194
2015-09-30 07:45:02 -07:00
epriestley
8f2f841d17 Fix some links to "Adding New Classes" in docs
Summary: Fixes T9483. This bookname is `phabcontrib`, not `contributor`.

Test Plan: `grep` / clicked these links.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9483

Differential Revision: https://secure.phabricator.com/D14195
2015-09-30 07:44:54 -07:00
Chad Little
ae082c6033 Make Ponder Emails a little more consistently delivered
Summary: Ref T9271, maybe fixes it. This restores feed publishing for answers (broken in D13951) and sends the author of the question an email for new answers. Also, unsure how to pull all question subsribers to the answer email, or is it automagical?

Test Plan: notchad asks a question, chad answers, log into notchad and see that mail was delivered, see feed story.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: revi, Korvin

Maniphest Tasks: T9271

Differential Revision: https://secure.phabricator.com/D14171
2015-09-29 14:25:28 -07:00
Ray Lillywhite
1ac919c29c Improved example setting for differential.generated-paths
Summary: This is to make it more obvious how to ignore a folder that may be at the root, and it resolves https://secure.phabricator.com/T8894

Test Plan: Just a documentation change

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, nornagon

Differential Revision: https://secure.phabricator.com/D14193
2015-09-29 14:14:31 -07:00
epriestley
45a5ea7bf5 Show lease owner in Drydock UI
Summary: Replaces D13687. Leases track an owner but don't currently show it.

Test Plan:
Looked at a lease.

{F851223}

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D14191
2015-09-29 09:51:06 -07:00
epriestley
6d5c9e897d Hide "Revision" column in Diffusion history view if Differential is uninstalled
Summary: Fixes T9481. If the viewer does not have access to Differential (for example, because it is not installed), hide the "Revision" column in Diffusion.

Test Plan:
  - Viewed history, saw "Revision" column.
  - Uninstalled Differential, reloaded, no "Revision" column.

Reviewers: chad

Reviewed By: chad

Subscribers: revi

Maniphest Tasks: T9481

Differential Revision: https://secure.phabricator.com/D14188
2015-09-29 07:09:12 -07:00
epriestley
be83d62375 Fix button for "All Problem Commits" on Owners packages
Summary:
Ref T9482. This button goes to the wrong place and this table conditionally hides itself so I missed it.

Instead:

  - Always show the table, with an empty string if there are no relevant commits.
  - Link to a working UI.

Test Plan: Saw table. Clicked button.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9482

Differential Revision: https://secure.phabricator.com/D14189
2015-09-29 06:43:55 -07:00
epriestley
fa943f744b Stop all object mentions from matching after "@"
Summary:
Fixes T9479. Currently, `@aaaaaaaa` may try to match as a commit hash, and `@C123456` may try to match as a Countdown reference. These should only match as user mentions.

Prevent object mention rules from matching after `@`. We already prevent them after `-` and `#`, and already prevented the username rule after `@` (i.e., preventing `@@user`).

Test Plan:
Created some "interesting" users locally and `@mentioned` them:

{F850779}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9479

Differential Revision: https://secure.phabricator.com/D14186
2015-09-29 06:43:49 -07:00
epriestley
21021b55c4 Update configuring_file_domain.diviner's cloudflare link
Summary: [[ https://cloudflare.net | https://cloudflare.net ]] generates SSL certification error, and even if I click "ignore it", the page is 403 forbidden afterwards. Their domain has moved to .com, therefore update it to reflect this.

Test Plan:
Go to [[ https://secure.phabricator.com/book/phabricator/article/configuring_file_domain/ | Configuring a File Domain ]], click [[ https://cloudflare.net | CloudFlare]].
Then, apply the patch. Refresh the page, and the new link works.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: revi, epriestley

Differential Revision: https://secure.phabricator.com/D14172
2015-09-29 03:35:59 -07:00
epriestley
55767aac0f Fix an issue where followup tasks could fail to queue with string priorities
Auditors: chad
2015-09-28 19:46:41 -07:00
epriestley
a5c4177160 Fix an issue with BuildLogs and web UI during builds
Summary:
Ref T9123. After recent changes, viewing a live build log from the web UI throws a CSRF exception.

Check `start` ("this object is an active log"), not `live` ("this log is open somewhere").

Test Plan: Will push / etc.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9123

Differential Revision: https://secure.phabricator.com/D14185
2015-09-28 19:11:51 -07:00
epriestley
efaa8170c3 Simplify value decoding for PHID custom fields
Summary:
Ref T9123. The handling in D14183 didn't deal with new field values properly.

Make all this handling more consistent.

Test Plan: Created a new WorkignCopy build plan with some repos.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9123

Differential Revision: https://secure.phabricator.com/D14184
2015-09-28 18:44:40 -07:00
epriestley
bfaa93aa9b Allow Harbormaster build plans to request additional working copies
Summary:
Ref T9123. To run upstream builds in Harbormaster/Drydock, we need to be able to check out `libphutil`, `arcanist` and `phabricator` next to one another.

This adds an "Also Clone: ..." field to Harbormaster working copy build steps so I can type all three repos into it and get a proper clone with everything we need.

This is somewhat upstream-centric and a bit narrow, but I don't think it's totally unreasonable, and most of the underlying stuff is relatively general.

This adds some more typechecking and improves data/type handling for custom fields, too. In particular, it prevents users from entering an invalid/restricted value in a field (for example, you can't "Also Clone" a repository you don't have permission to see).

Test Plan: Restarted build, got a Drydock resource with multiple repositories in it.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9123

Differential Revision: https://secure.phabricator.com/D14183
2015-09-28 17:57:41 -07:00
epriestley
0438a481e1 Fix issue with "Publish/Notify" handling in repositories
Summary:
Fixes T8728. As far as I can tell, I simply got this wrong in D11826. This is not the proper name for the preference.

That change primarily focused on the "spammy junk during import" issue, and the code did get the importing flag right. It looks like my testing in D11827 focused on "during import" and just missed this case.

Test Plan: Grepped for `disable-herald`. Grepped for `herald-disable`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8728

Differential Revision: https://secure.phabricator.com/D14181
2015-09-28 11:17:04 -07:00
epriestley
33be8f719f Allow WorkingCopy resources to have multiple working copies
Summary:
Ref T9252. For building Phabricator itself, we need to have `libphutil/`, `arcanist/` and `phabricator/` next to one another on disk.

Expand the Drydock WorkingCopy resource so that it can have multiple repositories if the caller needs them.

I'm not sure if I'm going to put the actual config for this in Harbormaster or Drydock yet, but the WorkingCopy resource itself should work the same way in either case.

Test Plan: Restarted a Harbormaster build which leases a working copy, saw it build as expected.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14180
2015-09-28 09:35:58 -07:00
epriestley
9b29d46e60 Make Drydock lease infrastructure more nimble
Summary:
Ref T9252. Currently, Harbormaster does this when trying to acquire a working copy:

  - Ask for a working copy.
  - Yield for 15 seconds.
  - Check if we have a working copy yet.

That's OK, but Drydock takes ~1s to acquire a working copy lease if a resource is already available, so we end up doing this:

  - T+0: Ask for a working copy.
  - T+0: Yield for 15 seconds.
  - T+1: Working copy lease activates.
  - T+15: Working copy lease is used.
  - T+16: Build finishes.

So we end up spending about 2 seconds doing work and 14 seconds sleeping.

One way to fix this would be to fiddle with the yield duration, so we yield for 1, 2, 4, ... seconds or something. This probably isn't a bad idea for longer leases (i.e., wait for 15, 30, 45 ... seconds or similar) but it implies a lot of churn for short leases.

Instead, let tasks "awaken" other tasks when they complete. The "awaken" operation means: if a task is in a yielded state (no failures, no owner, explicitly yielded, future expires time), pretend it only yielded until right now instead of whenever it really yielded to.

Basically, this rewrites history so that even though Harbormaster did a `yield(15)`, we pretend it did a `yield(4)` after we activate the lease if lease activation took 4 seconds.

If this misses, it's fine: we fall back to the normal yield behavior and things move forward normally a few seconds later.

If it hits, we get a more nimble process pretty cleanly.

Test Plan:
  - Restarted a build plan (lease working copy + run `ls`) with this patch no-op'd, took about 16 seconds.
  - Restarted a build plan with this patch active, took about 1 second.

Reviewers: hach-que, chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14178
2015-09-28 09:35:40 -07:00
epriestley
cd2dd2a08f Give visual feedback when a Drydock resource or lease is releasing
Summary: Ref T9252. Show the user when a resource or lease has a pending release command in queue.

Test Plan: Released a resource and lease from the web UI. In both cases, saw a "releasing" tag and the action disable.

Reviewers: hach-que, chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14177
2015-09-28 09:35:26 -07:00
epriestley
ec6d69e74d Give Drydock resources a proper expiry mechanism
Summary:
Fixes T6569. This implements an expiry mechanism for Drydock resources which parallels the mechanism for leases.

A few things are missing that we'll probably need in the future:

  - An "EXPIRES" command to update the expiration time. This would let resources be permanent while leased, then expire after, say, 24 hours without any leases.
  - A callback like `shouldActuallyExpireRightNow()` for resources and leases that lets them decide not to expire at the last second.
  - A callback like `didAcquireLease()` for resource blueprints, to parallel `didReleaseLease()`, letting them clear or extend their timer.

However, this stuff would mostly just let us tune behaviors, not really open up new capabilities.

Test Plan: Changed host resources to expire after 60 seconds, leased one, saw it vanish 60 seconds later.

Reviewers: hach-que, chad

Reviewed By: chad

Maniphest Tasks: T6569

Differential Revision: https://secure.phabricator.com/D14176
2015-09-28 09:35:14 -07:00
Chad Little
a3b49053c0 Allow polls to be public
Summary: Fixes T9474

Test Plan: Make a poll, log out, still see it.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T9474

Differential Revision: https://secure.phabricator.com/D14179
2015-09-28 07:55:28 -07:00
epriestley
ffbcefb629 Fix a doc typo
Summary: chack spels

Test Plan: typey typey

Reviewers: chad, hach-que

Reviewed By: hach-que

Differential Revision: https://secure.phabricator.com/D14175
2015-09-28 04:13:36 -07:00
epriestley
24845c70b9 Refine error behavior of bin/search index
Summary: Fixes T5991. If //all requested documents// failed to index, consider this a catastrophic failure and exit with an error code.

Test Plan:
  - Ran `bin/search index --type TASK`, observed successful exit despite a small number of un-indexable documents.
  - Ran `bin/search index PHID-TASK-xxx` for an invalid task, observed exception on exit after complete failure.
  - Ran normal indexing through daemons.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T5991

Differential Revision: https://secure.phabricator.com/D14174
2015-09-27 13:11:11 -07:00
epriestley
c99508cfe2 Explain upstream attitudes toward CLI exit codes
Summary: Ref T5991. See D14116. We are consistent but nonstandard in our use of exit codes. This document explains what we use exit codes for and why we do this.

Test Plan: Read it.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T5991

Differential Revision: https://secure.phabricator.com/D14173
2015-09-27 13:10:44 -07:00
epriestley
99d972fc81 Fix Herald rule actions on empty custom PHID fields
Summary: Fixes T9260. That task has a good description of the issue.

Test Plan: Followed steps in T9260 to reproduce the issue. Applied patch; issue went away.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9260

Differential Revision: https://secure.phabricator.com/D14169
2015-09-25 15:00:55 -07:00
epriestley
3e60740c7c Slightly modernize transaction diff controller
Summary: Ref T9272. This doesn't fix anything, just a little cleanup while I was looking at it.

Test Plan: Clicked "Show Details" on a couple description changes, got the same effect for less code.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9272

Differential Revision: https://secure.phabricator.com/D14168
2015-09-25 11:15:57 -07:00
epriestley
b7ca5a2d29 Provide a stable URI for getting raw paste content
Summary: Fixes T9312. This is a bit fluff, but does simplify the view controller slightly and seems reasonable/useful in general.

Test Plan: Clicked "View Raw File" on a paste, got redirected to the raw file via a stable URI.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9312

Differential Revision: https://secure.phabricator.com/D14167
2015-09-25 10:45:01 -07:00
epriestley
d735c7adf2 Allow Harbormaster to run commands on Drydock working copies
Summary: Ref T9252. This mostly cleans up future and log handling, and edges us closer to being able to do useful work with Harbormaster / Drydock.

Test Plan:
  - Added a "Run `ls -alh`" step to my trivial build plan.
  - Ran it a bunch of times.
  - Worked great.
  - Also did an HTTP plan.

{F835227}

{F835228}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9252

Differential Revision: https://secure.phabricator.com/D14161
2015-09-25 10:43:32 -07:00