Summary:
Ref T9252. Right now, we have very strict limits on Drydock: one lease per host, and one working copy per working copy blueprint.
These are silly and getting in the way of using "Land Revision" more widely, since we need at least one working copy for each landable repository.
For now, just remove the host limit and put a simple limit on working copies. This might need to be fancier some day (e.g., limit working copies per-host) but it is generally reasonable for the use cases of today.
Also add a `--background` flag to make testing a little easier.
(Limits are also less important nowadays than they were in the past, because pools expand slowly now and we seem to have stamped out all the "runaway train" bugs where allocators go crazy and allocate a million things.)
Test Plan:
- With a limit of 5, ran 10 concurrent builds and saw them finish after allocating 5 total resources.
- Removed limit, raised taskmaster concurrency to 128, ran thousands of builds in blocks of 128 or 256.
- Saw Drydock gradually expand the pool, allocating a few more working copies at first and a lot of working copies later.
- Got ~256 builds in ~140 seconds, which isn't a breakneck pace or anything but isn't too bad.
- This stuff seems to be mostly bottlenecked on `sbuild` throttling inbound SSH connections. I haven't tweaked it.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9252
Differential Revision: https://secure.phabricator.com/D14334
Summary:
Fixes T9519. Right now, build steps go straight from the build to the edit screen.
This means that there's no way to see their edit history or review details without edit permission. In particular, this makes it a bit harder to catch the Drydock Blueprint authorization warnings from T9519.
- Add a standard view screen.
- Add a little warning callout to blueprint authorizations.
This also does a bit of a touchup on the weird dropshadow element from T9586. Maybe not totally design-approved now but it's less ugly, at least.
Test Plan:
{F906695}
{F906696}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9519
Differential Revision: https://secure.phabricator.com/D14330
Summary:
Ref T9614. Currently, a lot of Build Plan behavior is covered by a global "can manage" policy.
One install in particular is experiencing difficulty with warring factions within engineering aborting one another's builds.
As a first step to remedy this, and also generally make Harbormaster more flexible and bring it in line with other applications in terms of policy power:
- Give Build Plans normal view/edit policies.
- Require "Can Edit" to run a plan manually.
Having "Can View" on plans may be a little weird in some cases (the status of a Buildable might be bad because of a build you can't see) but we can cross that bridge when we come to it.
Next change here will require "Can Edit" to abort a build. This will reasonably allow installs to reserve pause/abort for administrators/adults. (I might let anyone restart a plan, though?)
Test Plan:
- Created a new build plan.
- Verified defaults were inherited from application defaults (swapped them around, too).
- Saved build plan.
- Edited policies.
- Verified autoplans get the right policies.
- Verified old plans got migrated properly.
- Tried to run a plan I couldn't edit (denied).
- Ran a plan from CLI with `bin/harbormaster`.
- Tried to create a plan with an unprivileged user.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9614
Differential Revision: https://secure.phabricator.com/D14321
Summary: Fixes T9631. Build steps created before I added this option may not have it specified, which could throw later. Make handling a little more robust.
Test Plan: Will ask @yelirekim to report back.
Reviewers: chad
Reviewed By: chad
Subscribers: yelirekim
Maniphest Tasks: T9631
Differential Revision: https://secure.phabricator.com/D14336
Summary: Ref T9628. The porting feature has been fairly stable for a while, so make some reasonable effort to document how it works and some of the tradeoffs it involves.
Test Plan: Generated and read documentation.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9628
Differential Revision: https://secure.phabricator.com/D14335
Summary: Brings this in line with other support docs and pushes users toward the modern stuff.
Test Plan: Read documentation.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14329
Summary:
Fixes T9610.
- We currently permit you to `bin/auth recover` users who can not establish web sessions (but this will never work). Prevent this.
- We don't emit a tailored error if you follow one of these links. Tailor the error.
Even with the first fix, you can still hit the second case by doing something like:
- Recover a normal user.
- Make them a mailing list in the DB.
- Follow the recovery link.
The original issue here was an install that did a large migration and set all users to be mailing lists. Normal installs should never encounter this, but it's not wholly unreasonable to have daemons or mailing lists with the administrator flag.
Test Plan:
- Tried to follow a recovery link for a mailing list.
- Tried to generate a recovery link for a mailing list.
- Generated and followed a recovery link for a normal administrator.
{F906342}
```
epriestley@orbital ~/dev/phabricator $ ./bin/auth recover tortise-list
Usage Exception: This account ("tortise-list") can not establish web sessions, so it is not possible to generate a functional recovery link. Special accounts like daemons and mailing lists can not log in via the web UI.
```
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9610
Differential Revision: https://secure.phabricator.com/D14325
Summary: Ref T9625. I want this to be fixed ASAP hence here's the patch.
Test Plan:
- ~~Apply D14323~~ (This patch was made before it was merged)
- Apply this patch
- voila! Now I see the Ponder answer has correct logo.
{F906357}
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, revi
Maniphest Tasks: T9625
Differential Revision: https://secure.phabricator.com/D14331
Summary: Ref T9625. This is an example of how to fill in the missing calls.
Test Plan:
- Verified that an icon is now shown for feed stories.
- Verified that an icon is now shown in the "PHID Types" module panel in Config.
{F906325}
{F906326}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9625
Differential Revision: https://secure.phabricator.com/D14324
Summary: Ref T9625. Some PHID types are missing application or icon specifications. This makes it easier to spot them.
Test Plan: {F906321}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9625
Differential Revision: https://secure.phabricator.com/D14323
Summary:
These are a little out of date:
- Link to Starmap since it explicitly exists now.
- Link to "Planning" instead of the old task.
- Link to "Prioritization" instead of telling anyone to build stuff themselves.
Test Plan: Read documentation.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14328
Summary:
Currently, Version numbers are sort of randomly shown on "All Settings" beacuse we didn't have any better place to put them.
Now that we have modules, expose them as a config module.
Test Plan:
{F906426}
Grepped for "all settings" to look for other references to the old location, but didn't get any relevant hits.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14327
Summary: Fixes T9598.
Test Plan:
- Used "Send Message" as a logged-in user.
- Used "Send Message" as a logged-out user. The action was disabled and clicking it popped up a login dialog.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9598
Differential Revision: https://secure.phabricator.com/D14326
Summary:
It's hard for us to predict how long patches and migrations will take in the general case since it varies a lot from install to install, but we can give installs some kind of rough heads up about longer patches. I'm planning to just put a sort of hint for things in the changelog, something like this:
{F905579}
To make this easier, start storing how long stuff took. I'll write a little script to dump this into a table for the changelog.
Test Plan:
Ran `bin/storage status`:
{F905580}
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14320
Summary:
Fixes T6707. Users can currently do this:
- Log in to a service (like Facebook or Google) with account "A".
- Link their Phabricator account to that account.
- Log out of Facebook, log back in with account "B".
- Refresh the account link from {nav Settings > External Accounts}.
When they do this, we write a second account link (between their Phabricator account and account "B"). However, the rest of the codebase assumes accounts are singly-linked, so this breaks down elsewhere.
For now, decline to link the second account. We'll permit this some day, but need to do more work to allow it, and the need is very rare.
Test Plan:
- Followed the steps above, hit the new error.
- Logged back in to the proper account and did a link refresh (which worked).
{F905562}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T6707
Differential Revision: https://secure.phabricator.com/D14319
Summary:
We don't use these for anything, we're inconsistent about recording them, and there's some mild interaction with privacy concerns and data retention. Every other log we store any kind of information in can be given a custom retention policy after recent GC changes.
If we did put this back eventually it would probably be better to store a session identifier anyway, since that's more granular and more detailed.
You can fetch this info out of access logs anyway, too.
Test Plan: Left a couple of comments.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14315
Summary:
Ref T182. Replace the total mess we had before with a sort-of-reasonable element.
This automatically updates using "javascript".
Test Plan:
{F901983}
{F901984}
Used "Land Revision", saw the land status go from "Waiting" -> "Working" -> "Landed" without having to mash reload over and over again.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T182
Differential Revision: https://secure.phabricator.com/D14314
Summary:
An attempt to resolve T9600.
- `PhabricatorPasteQuery` builds truncated snippet when requested using `needSnippet()`.
- `PhabricatorPasteSearchEngine` uses Paste snippet istead of content.
- `PhabricatorSourceCodeView` accepts truncated source and type instead of line limit.
Test Plan: Generated some content for Paste application and also added huge JSON oneliner. Checked Paste application pages in browser.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin
Maniphest Tasks: T9600
Differential Revision: https://secure.phabricator.com/D14313
Summary: Ref T9532.
Test Plan: I don't have this configured locally but this seems very likely to be the correct fix. This list should be a list of PHIDs, but is a list of PHIDs followed by one PhabricatorRepository object.
Reviewers: avivey, chad
Reviewed By: chad
Maniphest Tasks: T9532
Differential Revision: https://secure.phabricator.com/D14311
Summary: Ref T8628. Updates Search.
Test Plan: Did various searches, saved new queries, reordered, ran new queries.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T8628
Differential Revision: https://secure.phabricator.com/D14268
Summary:
Fixes T9118. When populating some policy controls like "Default Can View" for repositories, we do some special logic to add object policies which are valid for the target object type.
For example, it's OK to set the default policy for an object which has subscribers to "Subscribers".
However, this logic incorrectly //removed// custom policies, so the form input ended up blank.
Instead, provide both object policies and custom policies.
Test Plan:
- Set default view policy to a custom policy.
- Hit "Edit" again, saw control correctly reflect custom policy after change.
- Set default edit policy to a different custom policy.
- Saved, edited, verified both policies stuck.
- Set both policies back.
- Checked some other object types to make sure object policies still work properly.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9118
Differential Revision: https://secure.phabricator.com/D14310
Summary: We often just setError as an array even if it's only one error. This just makes the UI a little cleaner in these cases.
Test Plan: Remove all reviewers from a diff, see status error without list styling.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14308
Summary:
Basically similar to D13941 but a little more extreme:
- Really strongly emphasize reproducibility for bug reports, and set users up for rejection if they don't satisfy this.
- Really strongly emphasize problem descriptions for feature requests, and set users up for rejection.
- Get rid of various "please give us feedback"; we get plenty of feedback these days.
- Some modernization tweaks.
- Split the support document into:
- Stuff we actually support for free (security / good bug reports / feature requests).
- Stuff you can pay us for (hosting / consulting / prioritization).
- A nebulous "community" section, with appropriate (low) expectations that better reflects reality.
My overall goals here are:
- Set expectations better, so users don't show up in IRC expecting it to be a "great place to get amazing support" or whatever the docs said in 2011.
- Possibly move the needle slightly on bug reports / feature request quality, maybe.
Test Plan: Read changes carefully.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14305
Summary:
I'm going to do some version of D13941. Clean up extra links to the old document first.
These were just randomly links from various places that we no longer really want feedback on and/or are now better covered by other documents.
Test Plan:
- `grep`
- Reviewed Config/Welcome screen.
- Reviewed `uri.allowed-editor-protocols`.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14303
Summary: Make Workboard initialization more restrictive.
Test Plan: Log out, see "No Workboard", Log in with permissions, see "New Workboard", Log in with notchad, see "No Workboard".
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T7410
Differential Revision: https://secure.phabricator.com/D14306
Summary: Right now logged out users can enable a workboard on a project.
Test Plan: Log out, view a public project, click on Workboard, get not set up dialog. Click Cancel, return to project details.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Differential Revision: https://secure.phabricator.com/D14304
Summary:
Fixes T9599. When APC/APCu are not available, we fall back to a disk-based cache.
We try to share this cache across webserver processes like APC/APCu would be shared in order to improve performance, but are just kind of guessing how to coordinate it. From T9599, it sounds like we don't always get this right in every configuration.
Since this is complicated and error prone, just stop trying to do this. This cache has bad performance anyway (no production install should be using it), and we have much better APC/APCu setup instructions now than we did when I wrote this. Just using the PID is simpler and more correct.
Test Plan:
- Artificially disabled APC.
- Reloaded the page, saw all the setup stuff run.
- Reloaded the page, saw no setup stuff run (i.e., cache was hit).
- Restarted the webserver.
- Reloaded the page, saw all the setup stuff run.
- Reloaded again, got a cache hit.
I don't really know how to reproduce the exact problem with the parent PID not working, but from T9599 it sounds like this fixed the issue and from my test plan we still appear to get correct behavior in the standard/common case.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9599
Differential Revision: https://secure.phabricator.com/D14302
Summary: Ref T8628. Updates Conduit for handleRequest
Test Plan: Use Conduit, test list, method calls, try a query, post this diff.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T8628
Differential Revision: https://secure.phabricator.com/D14265
Summary:
Fixes T9596.
Was unable to navigate to a task in Maniphest.
Test Plan: navigate to that task.
Reviewers: #blessed_reviewers, epriestley, avivey, tycho.tatitscheff
Reviewed By: avivey, tycho.tatitscheff
Subscribers: tycho.tatitscheff, avivey, Korvin
Maniphest Tasks: T9596
Differential Revision: https://secure.phabricator.com/D14300
Summary: Fixes T9592.
Test Plan: Log out ! Navigates to a task. See the add button grey-ed out !
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin
Maniphest Tasks: T9592
Differential Revision: https://secure.phabricator.com/D14299
Summary: Ref T182. Nothing fancy, just make these slightly easier to work with.
Test Plan: {F884754}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T182
Differential Revision: https://secure.phabricator.com/D14295
Summary: If you `!assign cahd` when you meant to `!assign chad`, we'll hit an "Undefined variable: assign_phid" a little further down.
Test Plan: Eyeballed it. See IRC.
Reviewers: chad
Reviewed By: chad
Differential Revision: https://secure.phabricator.com/D14291
Summary:
Fixes T9273. Remarkup has reasonably good fundamentals but the API is a giant pain to work with.
Provide a `PHUIRemarkupView` to make it easier. This object is way simpler to use by default.
It's not currently as powerful, but we can expand the power level later by adding more setters.
Eventually I'd expect to replace `PhabricatorRemarkupInterface` and `PhabricatorMarkupOneOff` with this, but no rush on those.
I converted a few callsites as a sanity check that it works OK.
Test Plan:
- Viewed remarkup in Passphrase.
- Viewed remarkup in Badges.
- Viewed a Conduit method.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9273
Differential Revision: https://secure.phabricator.com/D14289
Summary: Fixes T9573. This incorrectly affected Phriction. I could restore it for only projects, but you didn't like the rule very much anyway and I don't feel strongly about it.
Test Plan: Unit tests.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9573
Differential Revision: https://secure.phabricator.com/D14287
Summary: Ref T9336. Links the timeline photo to user profile. Presume this always exists?
Test Plan: Review a few timelines, click on heads.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T9336
Differential Revision: https://secure.phabricator.com/D14283
Summary: Ref T182. Make a reasonable attempt to get the commit message, author, and committer data correct.
Test Plan: BEHOLD: rGITTEST810b7f17cd0c909256a45d29a5062fcf417d0489
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T182
Differential Revision: https://secure.phabricator.com/D14280
Summary: Fixes T9568. We just weren't setting this properly so it would default away from the proper value.
Test Plan:
- Edited a credential in a non-default space, edit form populated properly.
- Changed "Space", introduced an error, saved form, got error with sticky value for "Space" properly.
- Saved form with new space value.
- Created a new credential.
Reviewers: chad
Reviewed By: chad
Subscribers: revi
Maniphest Tasks: T9568
Differential Revision: https://secure.phabricator.com/D14278
Summary:
Ref T9252. This fixes a bug from D14236. D14272 discusses the observable effects of the bug, primarily that the window for racing is widened from ~a few milliseconds to several minutes under our configuration.
This SQL query is missing a `GROUP BY` clause, so all of the resources get counted as having the same status (specifically, the alphabetically earliest status any resource had, I think). For test cases this often gets the right result since the number of resources may be small and they may all have the same status, but in production this isn't true. In particular, the allocator would sometimes see "35 destroyed resources" (or whatever), when the real counts were "32 destroyed resources + 3 pending resources".
Since this allocator behavior is soft/advisory this didn't cause any actual problems, per se (we do expect races here occasionally), it just made the race very very easy to hit. For example, Drydock in production currently has three pending working copy resources. Although we do expect this to be //possible//, getting 4 resources when the configured limit is 1 should be hard (not lightning strike / cosmic radiaion hard, but "happens once a year" hard).
Also exclude destroyed resources since we never care about them.
Test Plan:
Followed the plan from D14272 and restarted two Harbormaster workers at the same time.
After this patch was applied, they no longer created two different resources (we expect it to be possible for this to happen, just very hard).
We should still be able to force this race by putting something like `sleep(10)` right before the query, then `sleep(10)` right after it. That would prevent the allocators from seeing one another (so they would both think there were no other resources) and push us down the pathway where we exceed the soft limit.
Reviewers: chad, hach-que
Reviewed By: hach-que
Maniphest Tasks: T9252
Differential Revision: https://secure.phabricator.com/D14274
Summary:
Ref T9252. This is mostly a fix for an edge case from D14236. Here's the setup:
- There are no resources.
- A request for a new resource arrives.
- We build a new resource.
Now, if we were leasing an existing resource, we'd call `canAcquireLeaseOnResource()` before acquiring a lease on the new resource.
However, for new resources we don't do that: we just acquire a lease immediately. This is wrong, because we now allow and expect some resources to be unleasable when created.
In a more complex workflow, this can also produce the wrong result and leave the lease acquired sub-optimally (and, today, deadlocked).
Make the "can we acquire?" pathway consistent for new and existing resources, so we always do the same set of checks.
Test Plan:
- Started daemons.
- Deleted all working copy resources.
- Ran two working-copy-using build plans at the same time.
- Before this change, one would often [1] acquire a lease on a pending resource which never allocated, then deadlock.
- After this change, the same thing happens except that the lease remains pending and the work completes.
[1] Although the race this implies is allowed (resource pool limits are soft/advisory, and it is expected that we may occasionally run over them), it's MUCH easier to hit right now than I would expect it to be, so I think there's probably at least one more small bug here somewhere. I'll see if I can root it out after this change.
Reviewers: chad, hach-que
Reviewed By: hach-que
Maniphest Tasks: T9252
Differential Revision: https://secure.phabricator.com/D14272
Summary:
Ref T182. If 35 other things are configured completely correctly, make it remotely possible that this button may do something approximating the thing that the user wanted.
This primarily fleshes out the idea that "operations" (like landing, merging or cherry-picking) can have some beahavior, and when we run an operation we do whatever that behavior is instead of just running `git show`.
Broadly, this isn't too terrible because Drydock seems like it actually works properly for the most part (???!?!).
Test Plan: {F876431}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T182
Differential Revision: https://secure.phabricator.com/D14270
Summary:
Ref T182. This doesn't do anything interesting yet and is mostly scaffolding, but here's roughly the workflow. From previous revision, you can configure "Repository Automation" for a repository:
{F875741}
If it's configured, a new "Land Revision" button shows up:
{F875743}
Once you click it you get a big warning dialog that it won't work, and then this shows up at the top of the revision (completely temporary/placeholder UI, some day a nice progress bar or whatever):
{F875747}
If you're lucky, the operation eventually sort of works:
{F875750}
It only runs `git show` right now, doesn't actually do any writes or anything.
Test Plan:
- Clicked "Land Revision".
- Watched `phd debug task`.
- Saw it log `git show` to output.
- Verified operation success in UI (by fiddling URL, no way to get there normally yet).
Reviewers: chad
Reviewed By: chad
Subscribers: revi
Maniphest Tasks: T182
Differential Revision: https://secure.phabricator.com/D14266
Summary:
Ref T182. This allows you to assign blueprints that a repository can use to perform working copy operations. Eventually, this will support "merge this" in Differential, etc.
This is just UI for now, with no material effects.
Most of this diff is just taking logic that was in the existing "Blueprints" CustomField and putting it in more general places so Diffusion (which does not use CustomFields) can also access it.
Test Plan:
- Configured repository automation for a repository.
- Removed repository automation for a repository.
Reviewers: chad
Reviewed By: chad
Subscribers: avivey
Maniphest Tasks: T182
Differential Revision: https://secure.phabricator.com/D14259
Summary: Fixes T9552. We need to set a questionID and the question object (for policy) when initializing a new Answer.
Test Plan: Write an answer that mentions another user.
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin
Maniphest Tasks: T9552
Differential Revision: https://secure.phabricator.com/D14263
Summary:
Fixes T9558. The recent changes to validate PHID fields don't work cleanly with this gross hack.
This can probably be unwound now but it will definitely get fixed in T9132 so I may just wait for that.
Test Plan: Edited a custom "users" field in Maniphest. This should only affect Maniphest because it has a weird hack.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9558
Differential Revision: https://secure.phabricator.com/D14264
