1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-25 08:12:40 +01:00
Commit graph

13260 commits

Author SHA1 Message Date
epriestley
05f7227329 Document how to manually close revisions
Summary: Fixes T11484. These mechanisms aren't necessarily obvious and make sense to document here.

Test Plan: Read documentation.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11484

Differential Revision: https://secure.phabricator.com/D16404
2016-08-16 16:04:38 -07:00
epriestley
95cf83f14e Convert some whiny exceptions into quiet MalformedRequest exceptions
Summary:
Fixes T11480. This cleans up the error logs a little by quieting three common errors which are really malformed requests:

  - The CSRF error happens when bots hit anything which does write checks.
  - The "wrong cookie domain" errors happen when bots try to use the `security.alternate-file-domain` to browse stuff like `/auth/start/`.
  - The "no phcid" errors happen when bots try to go through the login flow.

All of these are clearly communicated to human users, commonly encountered by bots, and not useful to log.

I collapsed the `CSRFException` type into a standard malformed request exception, since nothing catches it and I can't really come up with a reason why anything would ever care.

Test Plan:
Hit each error through some level of `curl -H ...` and/or fakery. Verified that they showed to users before/after, but no longer log.

Hit some other real errors, verified that they log.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11480

Differential Revision: https://secure.phabricator.com/D16402
2016-08-16 15:50:21 -07:00
Chad Little
f50e550c9e Correct various spelling errors
Summary: Fixes T11477.

Test Plan: Grep for Mulitple

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11477

Differential Revision: https://secure.phabricator.com/D16399
2016-08-15 10:47:51 -07:00
epriestley
15021a0bcc Fix bad array index test in Differential package code
Summary: This needs an `isset()` for cases when authority and packages don't completely overlap.

Test Plan:
  - With a package set to trigger autoreview, created a revision.
  - Observed error log, saw no more error.
  - Saw package trigger autoreview properly.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16398
2016-08-14 13:10:32 -07:00
epriestley
8c41226624 Survive receipt of too much workboard column position information
Summary:
Fixes T11468. I was able to find //a// reproduction case for the issue, at least, described in my comments there.

When the server sends back position information for columns we aren't displaying, for whatever reason, just ignore that information.

Future work in T4900 to do more synchronization may revisit this.

Test Plan: See T11468#190177 for steps. Did that stuff with this patch, got a clean edit.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11468

Differential Revision: https://secure.phabricator.com/D16397
2016-08-14 09:13:08 -07:00
epriestley
07082d2867 Don't allow empty list constraints in Conduit calls
Summary:
Ref T11473. If you write a method like `get_stuff(ids)` and then call it with an empty list of IDs, you can end up passing an empty constraint to Conduit.

If you run a `*.search` method with such a constraint, like this one:

```
{
  "ids": []
}
```

...we have three possible beahviors:

  # Treat it like the user passed no constraint (basically, ignore the constraint).
  # Respect the constraint (return no results).
  # Error.

Currently, we do (1). However, this is pretty confusing and I think clearly the worst option, since it means `get_stuff(array())` in client code will often tend to return a ton of results.

We could do (2) instead, but this is also sort of confusing (it may not be obvious why nothing matched, even though it's an application bug) and I think most reasonable client code should be doing an `if ($ids)` test: this test makes clients a little more complicated, but they can save a network call, and I think they often need to do this test anyway (for example, to show the user a different message).

This implements (3), and just considers these to be errors: this is the least tricky behavior, it's consistent with what we do in PHP, makes fairly good sense, and the only cost for this is that client code may need to be slightly more complex, but this slightly more complex code is usually better code.

Test Plan: Ran Conduit `*.search` queries with `"ids":[]` and `"phids":[]`, got sensible errors instead of runaway result sets.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11473

Differential Revision: https://secure.phabricator.com/D16396
2016-08-14 08:31:13 -07:00
epriestley
99889a6321 Execute Harbormaster buildable filtering properly from HarbormasterBuildSearchEngine
Summary:
Ref T11473. When running `harbormaster.build.search` with a `buildables` constraint, the constraint doesn't get passed to the Query and so currently has no effect.

This piece of logic was just accidentally omitted from D16356. It is probably not used anywhere today and doesn't show up in the UI, so it's easy to overlook (I missed it in review, too).

Test Plan: Ran `harbormaster.build.search` with a `buildables` constraint, got expected filtering.

Reviewers: yelirekim, chad

Reviewed By: chad

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11473

Differential Revision: https://secure.phabricator.com/D16395
2016-08-14 08:31:02 -07:00
Chad Little
65e964fca1 Make "Core Applications" more reasonable
Summary: Ref T11132, cleaning up what "Core Applications" means.

Test Plan: Visit `/applications/`, see less poseurs.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11132

Differential Revision: https://secure.phabricator.com/D16394
2016-08-12 07:57:59 -07:00
Chad Little
07ea4e6f74 Fix scrollbar in filetree sidenav
Summary: Fixes T11457, I guess. It feels wierd when you drag it a bit, and Im not positive if it's the calc css or javascript.

Test Plan: Enable filetree, shorten browser, see scrolling appear if files too long.

Reviewers: avivey, epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11457

Differential Revision: https://secure.phabricator.com/D16393
2016-08-11 17:27:55 -07:00
epriestley
67861ec197 List both hosted and observed repositories in "Cluster Repository Status" configuration console
Summary:
When I wrote this the first time, only hosted repositories could be clustered.

This check wasn't removed when I allowed observed repositories to be clustered in D15986.

Test Plan:
Reloaded {nav Config > Repository Servers} page, saw more stuff locally.

Reviewed the cardinal digits between 1 and 17, inclusive.

Reviewers: chad, avivey

Reviewed By: avivey

Differential Revision: https://secure.phabricator.com/D16392
2016-08-11 16:42:30 -07:00
epriestley
ca78c1825a When already running as the daemon user, don't "sudo" daemon commands
Summary:
The cluster synchronization code runs either actively (before returning a response to `git clone`, for example) or passively (routinely, as the daemons update reposiories).

The active sync runs as the web user (if running `git clone http://...`) or the VCS user (if running `git clone ssh://...`). But the passive sync runs as the daemon user.

All of these sync processes need to run actual commands as the daemon user (`git fetch ...`).

For the active ones, we must `sudo`.

For the passive ones, we're already the right user. We run the same code, and end up trying to sudo to ourselves, which `sudo` isn't happy about by default.

Depending on how `sudo` is configured and which users things are running as this might work anyway, but it's silly and if it doesn't work it requires you to go make non-obvious, weird config changes that are unintuitive and somewhat nonsensical. This is probably worse on the balance than adding a bit of complexity to the code.

Instead, test which user we're running as. If it's already the right user, don't sudo.

Test Plan:
  - Ran `bin/repository update --trace` as daemon user, saw no more `sudo`.
  - Ran a `git clone` to make sure that didn't break.

Reviewers: chad, avivey

Reviewed By: avivey

Differential Revision: https://secure.phabricator.com/D16391
2016-08-11 16:41:19 -07:00
epriestley
39d4e21eec Fix a bad DiffusionCommandEngine parameter from HTTPEngine conversion
Summary:
I converted this call incorrectly in D16092. We should pass the `PhutilURI` object, not the string version of it.

Specifically, this resulted in hitting an error like this if a replica needed synchronization:

```
[2016-08-11 21:22:37] EXCEPTION: (InvalidArgumentException) Argument 1 passed to DiffusionCommandEngine::setURI() must be an instance of PhutilURI, string given, called in...
#0 PhutilErrorHandler::handleError(integer, string, string, integer, array) called at [<phabricator>/src/applications/diffusion/protocol/DiffusionCommandEngine.php:52]
#1 DiffusionCommandEngine::setURI(string) called at [<phabricator>/src/applications/diffusion/protocol/DiffusionRepositoryClusterEngine.php:601]
...
```

Test Plan: Clusterized an observed repository, demoted a node, ran `bin/repository update Rxxx` to update, saw no typehint fatal.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16390
2016-08-11 16:41:09 -07:00
epriestley
5e3efca08a In taskmaster daemons, only close connections which were not used recently
Summary:
Ref T11458. Depends on D16388. Currently, we're very aggressive about closing connections in the taskmaster daemons.

This can end up taking up a lot of resources. In particular, because the outgoing port for outbound connections normally can not be reused for 60 seconds after a connection closes, we may exhaust outbound ports on the host if there's a big queue full of stuff that's being processed very quickly.

At a minimum, we //always// are holding open a `worker` connection, which we always need again right away. So even in the best case we end up opening/closing this about once per second and each daemon takes up about ~60 outbound ports when it should take up ~1.

So, make two adjustments:

  - First, only close connections which we haven't issued a query on in the last 60 seconds. This should prevent us from closing connections that we'll need again immediately in most cases. In the worst case, we shouldn't be eating up any extra ports under default TCP behavior.
  - Second, explicitly close connections. We were relying on implicit/GC behavior (maybe as a holdover from very long ago, before we got connection wrappers in place?), which probably did about the same thing but isn't as predictable and can't be profiled or instrumented.

Test Plan:
This is somewhat difficult to test completely convincingly in isolation since the problem behavior depends on production scales and the workload, and to some degree on configuration.

I tested that this stuff baiscally works by adding logging to connect/close and running the daemons, verifying that they churned connections a lot before this change (e.g., ~1/s even at no load) and churn rarely afterward (e.g., almost never at no load).

I ran some workload through them to make sure I didn't completely break anything.

The best real test is just seeing how production responds. Current inbound/outbound connections on `secure001` are 1,200:

```
secure001 $ netstat -t | grep :mysql | wc -l
1164
```

Current outbound from `repo001` are 18,600:

```
repo001 $ netstat -t | grep :mysql | wc -l
18663
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11458

Differential Revision: https://secure.phabricator.com/D16389
2016-08-11 12:03:56 -07:00
epriestley
3b45608c78 Fix misleading error message when only cluster database masters are configured
Summary:
Fixes T11446. We can raise the misleading error:

> No valid databases are configured!

...when a valid master is configured but unreachable.

Instead, more carefully raise either "nothing is configured" or "nothing is reachable".

Test Plan: Configured only a master, artificially severed it, got "nothing is reachable" instead of "nothing is configured".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11446

Differential Revision: https://secure.phabricator.com/D16386
2016-08-10 13:06:12 -07:00
epriestley
e8083ad63a Increase the storage size for commit summaries
Summary:
Fixes T11453. Currently, commit message summaries are limited to 80 bytes. This may only be 20-40 characters for CJK languages or langauges with Cyrillic script.

Increase storage size to 255, then truncate to the shorter of 255 bytes or 80 glyphs. This preserves the same behavior for latin languages, but is less tight for Russian, etc.

Some minor additional changes:

  - Provide a way to ask "how much data fits in this column?" so we don't have to duplicate column lengths across summary checks or UI errors like "title too long".
  - Remove the `text80` datatype, since no other columns use it and we have no use cases (or likely use cases) for it.

Test Plan:
  - Made a commit with a Cyrillic title, saw reasonable summarization in UI:

{F1757522}

  - Added and ran unit tests.
  - Grepped for removed `SUMMARY_MAX_LENGTH` constant.
  - Grepped for removed `text80` data type.

Reviewers: avivey, chad

Reviewed By: avivey

Subscribers: avivey

Maniphest Tasks: T11453

Differential Revision: https://secure.phabricator.com/D16385
2016-08-10 11:12:45 -07:00
epriestley
38403b12be Update Herald documentation for modern policies and beahvior
Summary: Ref T11428. This documentation was a bit misleading and out of date. Update it to reflect modern reality.

Test Plan: Read documentation.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11428

Differential Revision: https://secure.phabricator.com/D16384
2016-08-10 08:55:43 -07:00
epriestley
7de2fae156 Link Herald rules to rule detail pages in Herald transcripts
Summary: Fixes T9410. Depends on D16382. Since all users can now view all Herald rules, we can link them in the transcripts.

Test Plan: Viewed a transcript, clicked rule names, reviewed rules.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9410

Differential Revision: https://secure.phabricator.com/D16383
2016-08-10 08:53:03 -07:00
epriestley
78ea6641a2 Let everyone view Herald rules
Summary:
Ref T9410. This changes the view policy for all Herald rules to the most public policy ("All Users" for private installs, "Public" for public installs).

See T11428 for discussion of this change in greater detail. In practice, this is //approximately// how things work today anyway, since you can almost always see almost all of this information in transcripts.

I believe this narrower view policy is helpful in zero cases and slightly confusing or harmful in a number of reasonable cases.

Test Plan: Viewed personal, object and global rules as users who could and could not edit the rules.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9410

Differential Revision: https://secure.phabricator.com/D16382
2016-08-10 08:52:36 -07:00
Eitan Adler
f032146591 Convert bin/storage workflow to presume utf8mb4 as the default encoding
Summary: see comments on rP68904d941c54

Test Plan: run ,/bin/storage upgrade

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16370
2016-08-10 05:56:27 -07:00
Chad Little
325e3b3ff3 Remove slight wash on transparent background on logo
Summary: Very slight gradient behind the eye I didn't catch (extra layer).

Test Plan: Remove all layers except logo, re-export.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16381
2016-08-08 13:30:07 -07:00
Chad Little
c80933be2f Force logotext to nowrap
Summary: When using spaces in the logo header, nowrap the text so it properly displays when too long on mobile, tablet.

Test Plan: Snowmen

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16380
2016-08-08 09:47:26 -07:00
Chad Little
679fbada44 Remove PHUIDocumentView
Summary: Converts final call site to PHUIDocumentViewPro.

Test Plan: grep for PHUIDocumentView, view new Welcome Page

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16379
2016-08-08 09:39:48 -07:00
epriestley
3a002b6b83 Make new logo and wordmark more reasonably configurable by human users
Summary: Fixes T11437. Provides a normal form for configuring this, instead of weird "look up the PHID and adjust things in the database" stuff.

Test Plan:
{F1753651}

{F1753652}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11437

Differential Revision: https://secure.phabricator.com/D16377
2016-08-07 12:00:21 -07:00
Chad Little
52c0ec2700 Update Phabricator logo
Summary:
This updates the eye logo and removes the formal wordmark "Phabricator" as an image. Instead we'll use the new updated eye logo and plain text for "Phabricator", both of which are more friendly and less industrial.

Installs that already use the `header-logo` customization setting will need to rebuild their logo to 80px x 80px. They will then also get to use plain text to whitebox their install as they see fit.

Test Plan:
Tested new logo at desktop, tablet, and mobile sizes. Set a random instance name, saw new wordmark. Created a really long wordmark of MMMMMMMMMMMM, saw text cut off so UI doesn't break. May need some additional tweaking, but I think we covered the most edge cases here.

{F1751791, size=full}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: edibiase, bjshively, yelirekim, Korvin

Maniphest Tasks: T4214, T11096

Differential Revision: https://secure.phabricator.com/D16373
2016-08-07 11:35:21 -07:00
epriestley
87f663ef77 Add basic Herald support to Calendar
Summary: Fixes T7939. This doesn't get too fancy, but allows you to write Herald rules against Calendar events.

Test Plan:
  - Wrote an "add red flag to events with party in the name" rule.
  - Created a "mundane meeting", didn't get flagged.
  - Created a "cool party", got flagged.
  - Ran rules from the Herald test console.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7939

Differential Revision: https://secure.phabricator.com/D16368
2016-08-06 14:37:33 -07:00
Evaldas Alexander
0cb9ca5500 Explain reasoning behind kitty gifts
Summary: I find this fact very useful for understanding my feline companion

Test Plan: Added "Motivator: Cat Facts" to the project, nothing broke

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D16374
2016-08-06 06:26:15 -07:00
Chad Little
1dd9c37fdf Clean up some tablet issues with new nav layouts
Summary: Makes sidenav disappear again on projects/profiles, but shows it on home again (tablet views).

Test Plan: Visit Profile/Projects/Home on mobile, desktop, and tablet. See nav disappear correctly.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16369
2016-08-04 20:49:16 +00:00
epriestley
8f4a63d708 Use consistent tag order in Maniphest list views and workboard cards
Summary: Fixes T11420. These are selected in newest-to-oldest order from the database, but we should show them in oldest-to-newest order in the UI.

Test Plan:
Tagged a couple tasks with "A, B, C" projects, saw correct order in UI:

{F1749351}

{F1749352}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11420

Differential Revision: https://secure.phabricator.com/D16367
2016-08-03 16:46:40 -07:00
epriestley
4d68c0ae04 Make Herald test workflow modular and more clear
Summary:
Fixes T9719. Currently, the Herald "Test Console" has a big `instanceof` thing, so new adapters (like a Calendar adapter, or third-party adapters) aren't available automatically. Instead, do a standard modular thing: load the available adapters, ask which ones can test the object the user selected, then let the user pick which one they want to move forward with.

Additionally, it isn't very clear that you can't test "commit hook" rules because they rely on push state which we don't really have a good way to simulate. When the user picks a commit, we now show them the "Hook" events, but the options are disabled and explain why they can not be selected.

Test Plan:
 - Ran test rules for revisions, commits, mocks, tasks, wiki documents, questions, and outbound mail.
 - Plugged in a commit, got a more-helpful choice screen explaining why you do a test run of hook rules.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9719

Differential Revision: https://secure.phabricator.com/D16360
2016-08-03 16:12:33 -07:00
epriestley
182a146280 Set "history.immutable" to "false" explicitly in .arcconfig
Summary:
If contributors use "history.immutable" by default (for example, in user/global config) and then try to contribute to Phabricator, they might land in a way that creates merge commits, which we'd prefer to avoid.

To make this a bit harder to do unintentionally, set "history.immutable" to "false" explicitly in `.arcconfig`. This will override any user or system setting and keep `land` in squash mode in this project.

This might also be a problem in general, and maybe in the long term we need something like one of these warnings:

> (when arc landing) You haven't landed changes to this project before, are you sure you want to create a merge commit / are you sure you want to squash?

...or:

> (when doing anything) This project doesn't have a history.immutable setting; you should probably set one so user settings don't win out.

...or just not let this be set at the user/system level, although that's annoying in some cases. Also maybe this:

> (when arc landing) You're about to create a merge commit, but the last Differential change in history squashed instead. Merge anyway?

However, we haven't seen this as a widespread issue outside of this project yet (and even in this project I think it has only happened 2-3 times), so just put up a guard rail in our own configuration for now.

Test Plan:
  - Ran `arc set-config history.immutable true` to set the flag in my user settings.
  - Ran `arc get-config` to verify that the setting overrode system/user settings:

```
epriestley@orbital ~/dev/phabricator $ arc get-config history.immutable --verbose
history.immutable

    If true, arc will never change repository history (e.g., through amending
    or rebasing). Defaults to true in Mercurial and false in Git. This
    setting has no effect in Subversion.

       Example Value: false

       Current Value: false
      Current Source: project

         local Value: -
       project Value: false
          user Value: true
        system Value: -
       default Value: -
```

Reviewers: chad, fooishbar

Reviewed By: fooishbar

Differential Revision: https://secure.phabricator.com/D16364
2016-08-03 08:12:49 -07:00
Daniel Stone
eecf6b8eca Fix broken link to PHPExcel site
Summary:
phpexcel.net currently serves a 500 page, and the top Google hit for
PHPExcel (on codeplex) gives you a site warning you that it is 3 years
out of date, and to see GitHub instead.

Update the link from Maniphest's 'please install PHPExcel to enable
export' prompt.

Test Plan:
- Click 'Export to Excel' from Maniphest query result page without PHPExcel installed
- Follow suggested link to install it
- See code rather than a HTTP 500 error

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D16363
2016-08-03 15:52:31 +01:00
Daniel Stone
518479a916 Fix broken link to PHPExcel site
phpexcel.net currently serves a 500 page, and the top Google hit for
PHPExcel (on codeplex) gives you a site warning you that it is 3 years
out of date, and to see GitHub instead.

Update the link from Maniphest's 'please install PHPExcel to enable
export' prompt.
2016-08-03 14:06:53 +01:00
epriestley
24a28dd1f3 Fix a documentation typo ("repositorie")
Summary: This isn't a word!

Test Plan: Read carefully.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16362
2016-08-02 18:02:34 -07:00
Chad Little
0bb5dd88c8 Always pre-wrap code blocks
Summary: Fixes T11416. Unclear what the side-effects of this would be, so bark if you find something. Previously, we'd have to overflow and scroll, which is kind of a pain since you're hiding content on long code blocks. This just wraps long lines, and preserves line breaks globally.

Test Plan: Test feed, profile, comments, inline comments, triple backticks.

Reviewers: avivey, epriestley

Reviewed By: avivey, epriestley

Subscribers: Korvin

Maniphest Tasks: T11416

Differential Revision: https://secure.phabricator.com/D16361
2016-08-02 11:46:43 -07:00
Chad Little
b53d38315f Fix Differental Filetree browser hotkey
Summary: Adds a class for explicitly hiding the sidenav.

Test Plan: Set Config to Enable Filetree. View a diff, see tree. Press `f`, see it go away. Reload page, see persistence.

Reviewers: avivey, epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16359
2016-08-01 14:36:48 -07:00
Chad Little
11e84c166a Redesign Application Search
Summary: This moves aphront-side-nav to use same table css display as profile nav. Slightly less code to support. Cleans up AppSearch UI, think I've gotten all the edge cases here, but bang on it, can hold until after release cut.

Test Plan: Config, Maniphest, Differential, Diffusion, Home.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16346
2016-08-01 12:23:36 -07:00
epriestley
5380d87792 Use long array syntax for compatibility instead of short array syntax
Summary: Fixes T11409. This syntax isn't compatible with older PHP.

Test Plan: Ran `arc lint` on the file.

Reviewers: yelirekim, chad

Reviewed By: chad

Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T11409

Differential Revision: https://secure.phabricator.com/D16358
2016-08-01 10:45:19 -07:00
Mike Riley
98492765d3 Subsume 'harbormaster.querybuilds' with a modern search API method
Summary: We deprecate the existing API method used to access build information from the API, but preserve its response structure after calling through to the new method.  I've cordoned off the fields I needed to define in order to meet the output structure by putting those fields in a search attachment.

Test Plan:
Used the API console and looked at the list view controller for builds.

Old output structure:

```lang=json
{
  "data": [
    {
      "id": "16823",
      "phid": "PHID-HMBD-xghrwfz6luoye5rgc2hq",
      "uri": "https://secure.phabricator.com/harbormaster/build/16823/",
      "name": "Run Core Tests",
      "buildablePHID": "PHID-HMBB-s6ykzm2jzxz4ymduztq3",
      "buildPlanPHID": "PHID-HMCP-pcfxcgyoif67l3buc4zt",
      "buildStatus": "passed",
      "buildStatusName": "Passed"
    }
  ],
  "cursor": {
    "limit": 100,
    "after": "16823",
    "before": null
  }
}
```

New output structure:

```lang=json
{
  "data": [
    {
      "id": 1,
      "type": "HMBD",
      "phid": "PHID-HMBD-qpgcmv67tzaauzayzit5",
      "uri": "http://ec2-54-165-244-168.compute-1.amazonaws.com/harbormaster/build/1/",
      "name": "arc lint + arc unit",
      "buildStatusName": "Passed",
      "buildablePHID": "PHID-HMBB-qdefith5uakkepqpjr2g",
      "buildPlanPHID": "PHID-HMCP-zswbhazb7ipmaf4plygg",
      "buildStatus": "passed",
      "initiatorPHID": "PHID-USER-rihx4366f3aczsvc2wtb",
      "dateCreated": 1450295643,
      "dateModified": 1450295644,
      "policy": {
        "view": "users",
        "edit": "users"
      }
    }
  ],
  "maps": {},
  "query": {
    "queryKey": null
  },
  "cursor": {
    "limit": 100,
    "after": null,
    "before": null,
    "order": null
  }
}
```

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D16356
2016-07-31 21:44:22 +00:00
Mike Riley
4865dbdff1 Search builds based on who kicked them off
Summary:
It's only natural for users to be interested their own builds. We are also building in support for other sources of builds, the only formally supported way to run a build right now is via Herald.

In our third party codebase, we designate an application as the "thing" that started builds which are scheduled and managed automatically by phabricator. I believe this is a common practice elsewhere in the codebase when you're at a loss for a real human identity and you need to apply some transactions.

Test Plan: Ran some builds manually and saw them show up under the list of things I've run.  Looking up builds based on those that had been started by a herald rule.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D16353
2016-07-31 20:54:44 +00:00
epriestley
cd8a9fd61e Resolve an issue with differential.query if no results are matched
Fixes T11406.

Auditors: chad
2016-07-31 13:07:52 -07:00
epriestley
1b192f746a Improve performance when constructing custom fields for objects
Summary:
Ref T11404. This improves things by about 10%:

  - Use `PhutilClassMapQuery`, which has slightly better caching.
  - Do a little less work to generate pretty error messages.
  - Make the "disabled" code a little faster (and sort of clearer, too?) by doing less fancy stuff.

These are pretty minor adjustments and not the sort of optimizations I'd make normally, but this code gets called ~100x (once per revision) and generates ~10 fields normally, so even small savings can amount to something.

(I also want to try to make `arc` faster in the next update, and improving Conduit performance helps with that.)

Test Plan: Ran `differential.revision.search`, saw cost drop from ~195ms to ~170ms locally.

Reviewers: yelirekim, chad

Reviewed By: chad

Maniphest Tasks: T11404

Differential Revision: https://secure.phabricator.com/D16355
2016-07-31 11:25:58 -07:00
epriestley
64886b11d8 Remove expensive, pointless typeachecking in custom fields
Summary:
Ref T11404. On my system, this improves performance by 10-15% for `differential.revision.search`.

`PhutilTypeSpec` provides high quality typechecking and is great for user-facing things that need good error messages.

However, it's also a bit slow, and pointless here (the API is internal and it only has one possible option).

I think I added this after writing `checkMap` just because I wanted to use it more often. My desire is sated after finding many reasonable ways to use it to give users high-quality error messages about things like configuration files.

Test Plan: Profiled `differential.revision.search` before and after change, saw wall time drop from ~220ms to ~195ms.

Reviewers: yelirekim, chad

Reviewed By: chad

Maniphest Tasks: T11404

Differential Revision: https://secure.phabricator.com/D16354
2016-07-31 11:25:28 -07:00
epriestley
8fd20e82fc Improve Conduit performance of special edge-based custom Revision fields
Summary:
Ref T11404. Depends on D16351. Currently, both `differential.query` and `differential.revision.search` issue `2N` queries to fetch:

  - dependencies for each revision; and
  - projects for each revision.

Fix this:

  - Take these custom fields out of Conduit so they don't load this data by default.
  - For `differential.query`, put this data back in by hard coding it.
  - For `differential.revision.search`, just leave it out. You can already optionally get projects efficiently, and this endpoint is a work in progress. I would tentatively be inclined to expose graph data as a "graph" extension once we need it.

This makes both methods execute in `O(1)` time (which is still 20-30 queries, but at least it's not 320 queries anymore).

Test Plan:
  - Ran `differential.query`, observed no change in results but 199 fewer internal queries.
  - Ran `differential.revision.search`, observed data gone from results and 200 fewer internal queries.

Reviewers: yelirekim, chad

Reviewed By: chad

Maniphest Tasks: T11404

Differential Revision: https://secure.phabricator.com/D16352
2016-07-31 11:24:58 -07:00
epriestley
b8f75f9511 Improve Conduit performance for custom fields
Summary:
Ref T11404. Depends on D16350.

Currently, custom fields can issue "N+1" queries in some cases, so querying 100 revisions issues 100 extra queries.

This affects all `*.search` endpoints for objects with custom fields, and some older endpoints (notably `differential.query`).

This change bulk loads "normal" custom fields, which gets rid of some of these queries. Instead of loading fields for each object, we build a big list of all fields and load them all at once.

The next change will tackle the remaining inefficient edge queries.

Test Plan:
  - Configured a custom field with normal database storage in Differential.
  - Ran `differential.query`, looking at custom fields in results for correctness.
  - Ran `differential.revision.search`, looking at custom fields in results for correctness.
  - In both cases, observed queries drop from `3N` to `2N` (all the "normal" custom field stuff got bulk loaded).

Reviewers: yelirekim, chad

Reviewed By: chad

Maniphest Tasks: T11404

Differential Revision: https://secure.phabricator.com/D16351
2016-07-31 11:15:58 -07:00
epriestley
6e57582aff Allow *.search Conduit API methods to have data bulk-loaded by extensions
Summary:
Ref T11404. Currently, SearchEngineAttachments can bulk-load data but SearchEngineExtensions can not.

This leads to poor performance of custom fields. See T11404 for discussion.

This changes the API to support a bulk load + format pattern like the one Attachments use. The next change will use it to bulk-load custom field data.

Test Plan:

  - Ran `differential.query`, `differential.revision.search` as a sanity check.
  - No behavioral changes are expected
  - See next revision.

Reviewers: yelirekim, chad

Reviewed By: chad

Maniphest Tasks: T11404

Differential Revision: https://secure.phabricator.com/D16350
2016-07-31 11:15:18 -07:00
Mike Riley
33fca12816 Pick some preset build statuses
Summary:
We're picking three useful groups of build statuses to provide as default queries:

 - Stuff not yet building
 - Stuff building
 - Stuff which has finished building

These are reasonable buckets for builds since (unlike most objects in phabricatorland) users are generally waiting impatiently for the machine to do something for them, rather than being responsible for doing something with the machine.

Test Plan: clicked around the search engine and enjoyed my defaults

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D16349
2016-07-31 15:35:18 +00:00
Mike Riley
42b81a8090 Move build statuses to a constants class
Summary: No functional changes here, just lifting this out to make room for activities, heeding lint warnings along the way.

Test Plan:
before:
```lang=bash
$ grep -Rn "HarbormasterBuild::" *
src/applications/harbormaster/storage/HarbormasterBuildable.php:169:    $build = HarbormasterBuild::initializeNewBuild($viewer)
src/applications/harbormaster/storage/HarbormasterBuildable.php:173:      ->setBuildStatus(HarbormasterBuild::STATUS_PENDING);
src/applications/harbormaster/controller/HarbormasterStepEditController.php:242:    $variables = HarbormasterBuild::getAvailableBuildVariables();
src/applications/harbormaster/controller/HarbormasterBuildableViewController.php:200:        'fa-dot-circle-o '.HarbormasterBuild::getBuildStatusColor($status),
src/applications/harbormaster/controller/HarbormasterBuildableViewController.php:201:        HarbormasterBuild::getBuildStatusName($status));
src/applications/harbormaster/controller/HarbormasterBuildableViewController.php:203:      $item->addAttribute(HarbormasterBuild::getBuildStatusName($status));
src/applications/harbormaster/controller/HarbormasterBuildViewController.php:584:        HarbormasterBuild::getBuildStatusName($status);
src/applications/harbormaster/controller/HarbormasterBuildViewController.php:585:      $icon = HarbormasterBuild::getBuildStatusIcon($status);
src/applications/harbormaster/controller/HarbormasterBuildViewController.php:586:      $color = HarbormasterBuild::getBuildStatusColor($status);
src/applications/harbormaster/event/HarbormasterUIEventListener.php:135:      $status_name = HarbormasterBuild::getBuildStatusName($status);
src/applications/harbormaster/event/HarbormasterUIEventListener.php:136:      $icon = HarbormasterBuild::getBuildStatusIcon($status);
src/applications/harbormaster/event/HarbormasterUIEventListener.php:137:      $color = HarbormasterBuild::getBuildStatusColor($status);
src/applications/harbormaster/conduit/HarbormasterQueryBuildsConduitAPIMethod.php:78:        'buildStatusName' => HarbormasterBuild::getBuildStatusName($status),
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:66:      $build->setBuildStatus(HarbormasterBuild::STATUS_ERROR);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:109:      $build->setBuildStatus(HarbormasterBuild::STATUS_ABORTED);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:113:    if (($build->getBuildStatus() == HarbormasterBuild::STATUS_PENDING) ||
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:116:      $build->setBuildStatus(HarbormasterBuild::STATUS_BUILDING);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:121:      $build->setBuildStatus(HarbormasterBuild::STATUS_BUILDING);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:126:      $build->setBuildStatus(HarbormasterBuild::STATUS_PAUSED);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:132:    if ($build->getBuildStatus() == HarbormasterBuild::STATUS_BUILDING) {
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:246:      $build->setBuildStatus(HarbormasterBuild::STATUS_FAILED);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:254:      $build->setBuildStatus(HarbormasterBuild::STATUS_PASSED);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:290:      $build->setBuildStatus(HarbormasterBuild::STATUS_DEADLOCKED);
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:446:      if ($build->getBuildStatus() != HarbormasterBuild::STATUS_PASSED) {
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:449:      if ($build->getBuildStatus() == HarbormasterBuild::STATUS_FAILED ||
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:450:          $build->getBuildStatus() == HarbormasterBuild::STATUS_ERROR ||
src/applications/harbormaster/engine/HarbormasterBuildEngine.php:451:          $build->getBuildStatus() == HarbormasterBuild::STATUS_DEADLOCKED) {
```

after:

```lang=bash
$ grep -Rn "HarbormasterBuild::" *
src/applications/harbormaster/storage/HarbormasterBuildable.php:169:    $build = HarbormasterBuild::initializeNewBuild($viewer)
src/applications/harbormaster/controller/HarbormasterStepEditController.php:242:    $variables = HarbormasterBuild::getAvailableBuildVariables();
```

ran a manual build as a sanity check

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D16348
2016-07-31 14:56:31 +00:00
Mike Riley
2c55a4ad72 Provide a basic search engine for builds
Summary:
This supports a few basic use cases that aren't served by the buildable search engine:

 - I'm trying to discover when the last time that this particular build plan failed was.
 - I want to know if any builds have deadlocked.
 - At a glance, I'm more interested in what build plans are running, not which buildables are being built. This is more often than not the case.

Test Plan: {F1744003}

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D16347
2016-07-31 13:35:31 +00:00
epriestley
bbc2ae7858 Fix task graph fatal for graphs containing restricted tasks
Summary: Fixes T11392. If some tasks are restricted, we only have PHIDs for them, not objects. Just use the PHIDs instead.

Test Plan: {F1741335}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11392

Differential Revision: https://secure.phabricator.com/D16345
2016-07-29 07:21:45 -07:00
epriestley
15c7eb1425 When a task graph has too much stuff, only show adjacent nodes (direct parents/children)
Summary:
Ref T4788. This gives us a new level of graceful degradation, so now we show:

  - Zero through 100 connected tasks: whole graph.
  - More than 100 connnected tasks, but fewer than 100 direct parents/subtasks: just parents and subtasks, with "..." to hint that the graph is cut off.
  - More than 100 parents and children: just the "sorry, too much stuff" error message.

Test Plan: {F1740882}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4788

Differential Revision: https://secure.phabricator.com/D16344
2016-07-28 15:38:27 -07:00