1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-24 15:52:41 +01:00
Commit graph

135 commits

Author SHA1 Message Date
epriestley
19be32656f Implement clock/trigger infrastructure for scheduling actions
Summary:
Ref T6881. Hopefully, this is the hard part.

This adds a new daemon (the "trigger" daemon) which processes triggers, schedules them, and then executes them at the scheduled time. The design is a little complicated, but has these goals:

  - High resistance to race conditions: only the application writes to the trigger table; only the daemon writes to the event table. We won't lose events if someone saves a meeting at the same time as we're sending a reminder out for it.
  - Execution guarantees: scheduled events are guaranteed to execute exactly once.
  - Support for arbitrarily large queues: the daemon will make progress even if there are millions of triggers in queue. The cost to update the queue is proportional to the number of changes in it; the cost to process the queue is proportional to the number of events to execute.
  - Relatively good observability: you can monitor the state of the trigger queue reasonably well from the web UI.
  - Modular Infrastructure: this is a very low-level construct that Calendar, Phortune, etc., should be able to build on top of.

It doesn't have this stuff yet:

  - Not very robust to bad actions: a misbehaving trigger can stop the queue fairly easily. This is OK for now since we aren't planning to make it part of any other applications for a while. We do still get execute-exaclty-once, but it might not happen for a long time (until someone goes and fixes the queue), when we could theoretically continue executing other events.
  - Doesn't start automatically: normal users don't need to run this thing yet so I'm not starting it by default.
  - Not super well tested: I've vetted the basics but haven't run real workloads through this yet.
  - No sophisticated tooling: I added some basic stuff but it's missing some pieces we'll have to build sooner or later, e.g. `bin/trigger cancel` or whatever.
  - Intentionally not realtime: This design puts execution guarantees far above realtime concerns, and will not give you precise event execution at 1-second resolution. I think this is the correct goal to pursue architecturally, and certainly correct for subscriptions and meeting reminders. Events which execute after they have become irrelevant can simply decline to do anything (like a meeting reminder which executes after the meeting is over).

In general, the expectation for applications is:

  - When creating an object (like a calendar event) that needs to trigger a scheduled action, write a trigger (and save the PHID if you plan to update it later).
  - The daemon will process the event and schedule the action efficiently, in a race-free way.
  - If you want to move the action, update the trigger and the daemon will take care of it.
  - Your action will eventually dump a task into the task queue, and the task daemons will actually perform it.

Test Plan:
Using a test script like this:

```
<?php

require_once 'scripts/__init_script__.php';

$trigger = id(new PhabricatorWorkerTrigger())
  ->setAction(
    new PhabricatorLogTriggerAction(
      array(
        'message' => 'test',
      )))
  ->setClock(
    new PhabricatorMetronomicTriggerClock(
      array(
        'period' => 33,
      )))
  ->save();

var_dump($trigger);
```

...I queued triggers and ran the daemon:

  - Verified triggers fire;
  - verified triggers reschedule;
  - verified trigger events show up in the web UI;
  - tried different periods;
  - added some triggers while the daemon was running;
  - examined `phd debug` output for anything suspicious.

It seems to work in trivial use case, at least.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6881

Differential Revision: https://secure.phabricator.com/D11419
2015-01-16 12:13:31 -08:00
Joshua Spence
daadf95537 Fix visibility of PhutilArgumentWorkflow::didConstruct methods
Summary: Ref T6822.

Test Plan: `grep`. This method is only called from within `PhutilArgumentWorkflow::__construct`.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Maniphest Tasks: T6822

Differential Revision: https://secure.phabricator.com/D11415
2015-01-16 07:42:07 +11:00
Joshua Spence
463d094f96 Fix method visibility for PhabricatorPolicyAwareQuery subclasses
Summary: Ref T6822.

Test Plan:
`grep` for the following:

  - `->willFilterPage(`
  - `->loadPage(`
  - `->didFilterPage(`
  - `->getReversePaging(`
  - `->didFilterPage(`
  - `->willExecute(`
  - `->nextPage(`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: hach-que, Korvin, epriestley

Maniphest Tasks: T6822

Differential Revision: https://secure.phabricator.com/D11367
2015-01-14 07:01:16 +11:00
Joshua Spence
d6b882a804 Fix visiblity of LiskDAO::getConfiguration()
Summary: Ref T6822.

Test Plan: `grep`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: hach-que, Korvin, epriestley

Maniphest Tasks: T6822

Differential Revision: https://secure.phabricator.com/D11370
2015-01-14 06:54:13 +11:00
epriestley
ba4ebf28ad Allow archived tasks to be queried by object PHID and order by id
Summary: Ref T5402.

Test Plan:
  - Queried archived tasks.
  - Grepped for use sites and verified no other callsites are order-sensitive.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5402

Differential Revision: https://secure.phabricator.com/D11089
2014-12-30 15:54:56 -08:00
epriestley
c2f4ae3502 Separate daemon task table rendering into a standalone class
Summary: Ref T5402, T6238. Pull this out into a class so the Instances app can embed task views.

Test Plan: Loaded `/daemon/` and examined the content in the tables.

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T6238, T5402

Differential Revision: https://secure.phabricator.com/D11090
2014-12-30 10:00:06 -08:00
Bob Trahan
a4474a4975 Daemons - introduce PhabricatorWorkerArchiveTaskQuery
Summary: Ref T5402. This cleans up some code and sets us up to use this sort of data more easily later.

Test Plan: viewed the daemon console from the web and the log of a specific archived daemon. both looked good. for other callsites looked really, really carefully.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T5402

Differential Revision: https://secure.phabricator.com/D11042
2014-12-23 15:45:42 -08:00
epriestley
b3394c53d8 Detect goofy sudo -n output under OSX
Summary: See rP2fedb6f941d8. We might need a more general version of this since we do some `sudo` stuff elsewhere, but at least on my machine `sudo -n` exits with code 0 when the target user exists but needs a password.

Test Plan:
  - Tried to run daemons as root, with no automatic sudo to root. Got a bad result before (phd believed it had executed the daemons) and a good result afterward (phd recognized that sudo failed).
  - Tried to run daemons from root, as a non-root user. Got a good result in both cases.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: fabe, epriestley

Differential Revision: https://secure.phabricator.com/D11041
2014-12-23 14:45:07 -08:00
Fabian Stelzer
2fedb6f941 Start phd daemons as the correctly configured user and refuse otherwise
Summary:
Fixes T5196
If no phd.user is configured the behaviour is unchanged besides printing a warning when run as root (Usually i would add an exit(1) here but that would break existing installs who do that).
If phd.user is set and the current user is root it will run the daemon as: su USER -c "command" (I'm not sure if this works for every platform needed)
Otherwise it will refuse to start if configured and current user mismatch.

Test Plan: Stopped & Started phd daemon with various users and different phd.user settings including root

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: vinzent, epriestley

Maniphest Tasks: T5196

Differential Revision: https://secure.phabricator.com/D11036
2014-12-23 08:15:51 -08:00
epriestley
9a7383121d Move cancel/retry/free task queue actions to bin/worker
Summary:
Fixes T6702. Ref T3554. Currently, tasks can be cancelled, retried and freed from the web UI by any logged in user.

This isn't appreciably dangerous (I can't come up with a way that a user could do anything security-affecting), but I think I probably intended this to be admin-only, but these actions should move to the CLI anyway.

Move them to the CLI. Lay some groundwork for some future `bin/worker cancel --class SomeTaskClass`, but don't implement that yet.

Test Plan: Used `cancel`, `retry` and `free` from the CLI. Hit all the error/success states.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T3554, T6702

Differential Revision: https://secure.phabricator.com/D10939
2014-12-06 09:14:16 -08:00
epriestley
914b8bb32c Fix daemon task queue to respect task priority
Summary:
Fixes an issue with T5336 / D9871. We did 99% of the work here but didn't actually turn on the priority sorting. The unit test passed by default, which didn't catch this.

  - Fix the unit test (it failed).
  - Fix the query (test now passes).
  - Add a "Next in Queue" element to the UI to make this kind of thing easier to spot/understand.

Test Plan: Ran unit test. Viewed "Next in Queue". Queued some tasks, flushed the queue. Web UI tracked the state sensibly.

Reviewers: joshuaspence, btrahan

Reviewed By: btrahan

Subscribers: cburroughs, epriestley

Differential Revision: https://secure.phabricator.com/D10766
2014-10-31 09:27:04 -07:00
epriestley
8fa8415c07 Automatically build all Lisk schemata
Summary:
Ref T1191. Now that the whole database is covered, we don't need to do as much work to build expected schemata. Doing them database-by-database was helpful in converting, but is just reudndant work now.

Instead of requiring every application to build its Lisk objects, just build all Lisk objects.

I removed `harbormaster.lisk_counter` because it is unused.

It would be nice to autogenerate edge schemata, too, but that's a little trickier.

Test Plan: Database setup issues are all green.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley, hach-que

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10620
2014-10-02 09:51:20 -07:00
epriestley
943c62d1e9 Add missing expected keys and uniqueness
Summary:
Ref T1191.

  - Adds definitions for missing keys and keys with wrong uniqueness. Generally, I defined these before fixing the key query to actually pull all keys and support uniqueness.
  - Moves "key uniqueness" to note severity; this is fixable (probably?) and there are no remaining issues.
  - Moves "Missing Key" to note severity; missing keys are fixable and all remaining missing keys are really missing (either missing edge keys, or missing PHID keys):

{F210089}

  - Moves "Surplus Key" to note seveirty; surplus keys are fixable all remaining surplus keys are really surplus (duplicate key in Harbormaster, key on unused column in Worker):

{F210090}

Test Plan:
  - Vetted missing/surplus/unique messages.
  - 146 issues remaining.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10590
2014-10-01 07:53:50 -07:00
epriestley
1ead50c2cc Generate reasonable expected schemata for Chatlog, Conduit, Config, Countdown, Daemons
Summary: Ref T1191. Fills in some more of the databases. Nothing very notable here. I didn't encounter any issues or overlong keys.

Test Plan: Used web UI to click around and verify expected schemata match up against actual schemata well.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T1191

Differential Revision: https://secure.phabricator.com/D10516
2014-09-18 11:15:29 -07:00
Bob Trahan
7ecbc70784 Daemons - handle daemons that can't be killed a bit better
Summary:
Ref T2374. Fixes T5988.

Keep track of what's been killed and not been killed, and surface that maybe you need sudo if things don't get killed with --force

...also basically make this force thing work. I managed to convinced myself stuff was getting killed with --force when it mostly wasn't. Make sure the --force parameter gets pushed as low as it needs to go to have things get killed.

Test Plan:
 - `sudo ./bin/phd restart`
 - `rm -rf /var/tmp/phd/pid/*`
 - `./bin/phd stop` --> get warning about rogue daemons
 - `./bin/phd stop X` --> get warning about no running daemons
 - `./bin/phd stop --force` --> get warning about not being able to kill daemons
 - `sudo ./bin/phd stop --force` --> kill daemons successfully

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T2374, T5988

Differential Revision: https://secure.phabricator.com/D10386
2014-09-03 15:19:02 -07:00
epriestley
df3ddd5de4 Fix bin/phd log instructions
Summary: The command takes `--id` (since recently?), fix command prompt.

Test Plan: `bin/phd help log`

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D10404
2014-09-02 17:11:36 -07:00
epriestley
340b40172f Tweak --limit for bin/phd log
Summary:
Ref T5405.

  - `--limit` wasn't actually used anywhere.
  - Make it mean "the N newest lines".

Test Plan: Ran `bin/phd log`, `bin/phd log --limit 3`.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5405

Differential Revision: https://secure.phabricator.com/D10385
2014-08-29 10:02:19 -07:00
Bob Trahan
06882a99cf Daemons - move combined log to console
Summary: Fixes T5405.

Test Plan: ran a few commands (log, log --id X --id Y, log --id BADX, log --id BADX --id BADY) and verified good output

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T5405

Differential Revision: https://secure.phabricator.com/D10371
2014-08-27 14:53:38 -07:00
Bob Trahan
2fdd7f0f3d Daemons - better handle rogue daemons from phd
Summary: Ref T2374. While building D10367 I noticed that phd was finding rogue daemons way more than it should be. Re-jigger this code path so rogue daemons are checked for *after* we've dealt with known daemons. This keeps the logic pretty simple overall.

Test Plan: phd start; kill pid files; phd stop and get the right warning; phd stop --force and it kills the rogue demons. phd stop in normal conditions no longer reporting rogue daemons erroneously

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T2374

Differential Revision: https://secure.phabricator.com/D10368
2014-08-27 12:24:06 -07:00
Bob Trahan
c2874945c8 Daemons - add status of environment to daemon console, etc
Summary: Shows the UI everywhere. Also asort() the keys before calculating the environment hash as that is probably an issue for someone at some point we just don't need to have. Ref T5968.

Test Plan: Viewed the setup check and saw a link to the daemon console. Viewed the daemon console and saw the various stale config daemons. Clicked a daemon and saw a "stale config" header icon where expected. Restarted daemons and all of this went away.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T5968

Differential Revision: https://secure.phabricator.com/D10367
2014-08-27 11:35:21 -07:00
Bob Trahan
d13d6963dd phd - make stop / restart savvy to daemons without pid files
Summary: Ref T2374. This currently doesn't work in that while the daemons are stopped, they are restarted. How do I stop them for good? (If it worked I'd also give it a little pass for variable names, etc quality stuff)

Test Plan:
```
14:09:20 ~/Dropbox/code/phalanx (T2374)
~> sudo ./bin/phd restart
There are no running Phabricator daemons.
Freeing active task leases...
Freed 0 task lease(s).
Preparing to launch daemons.
NOTE: Logs will appear in '/var/tmp/phd/log/daemons.log'.

Launching daemon "PhabricatorRepositoryPullLocalDaemon".
Launching daemon "PhabricatorGarbageCollectorDaemon".
Launching daemon "PhabricatorTaskmasterDaemon".
Launching daemon "PhabricatorTaskmasterDaemon".
Launching daemon "PhabricatorTaskmasterDaemon".
Launching daemon "PhabricatorTaskmasterDaemon".
Done.
14:09:30 ~/Dropbox/code/phalanx (T2374)
~> rm -rf /var/tmp/phd/pid/*
14:09:42 ~/Dropbox/code/phalanx (T2374)
~> sudo ./bin/phd stop
There are no running Phabricator daemons.
There are processes running that look like Phabricator daemons but have no corresponding PID files:

php /Users/btrahan/Dropbox/code/libphutil/scripts/daemon/exec/exec_daemon.php PhabricatorRepositoryPullLocalDaemon --load-phutil-library=/Users/btrahan/Dropbox/code/arcanist/src --load-phutil-library=/Users/btrahan/Dropbox/code/phalanx/src --log=/var/tmp/phd/log/daemons.log --
php /Users/btrahan/Dropbox/code/libphutil/scripts/daemon/exec/exec_daemon.php PhabricatorGarbageCollectorDaemon --load-phutil-library=/Users/btrahan/Dropbox/code/arcanist/src --load-phutil-library=/Users/btrahan/Dropbox/code/phalanx/src --log=/var/tmp/phd/log/daemons.log --
php /Users/btrahan/Dropbox/code/libphutil/scripts/daemon/exec/exec_daemon.php PhabricatorTaskmasterDaemon --load-phutil-library=/Users/btrahan/Dropbox/code/arcanist/src --load-phutil-library=/Users/btrahan/Dropbox/code/phalanx/src --log=/var/tmp/phd/log/daemons.log --
php /Users/btrahan/Dropbox/code/libphutil/scripts/daemon/exec/exec_daemon.php PhabricatorTaskmasterDaemon --load-phutil-library=/Users/btrahan/Dropbox/code/arcanist/src --load-phutil-library=/Users/btrahan/Dropbox/code/phalanx/src --log=/var/tmp/phd/log/daemons.log --
php /Users/btrahan/Dropbox/code/libphutil/scripts/daemon/exec/exec_daemon.php PhabricatorTaskmasterDaemon --load-phutil-library=/Users/btrahan/Dropbox/code/arcanist/src --load-phutil-library=/Users/btrahan/Dropbox/code/phalanx/src --log=/var/tmp/phd/log/daemons.log --
php /Users/btrahan/Dropbox/code/libphutil/scripts/daemon/exec/exec_daemon.php PhabricatorTaskmasterDaemon --load-phutil-library=/Users/btrahan/Dropbox/code/arcanist/src --load-phutil-library=/Users/btrahan/Dropbox/code/phalanx/src --log=/var/tmp/phd/log/daemons.log --
php /Users/btrahan/Dropbox/code/phalanx/scripts/daemon/phd-daemon PhabricatorRepositoryPullLocalDaemon --daemonize --log=/var/tmp/phd/log/daemons.log --phd=/var/tmp/phd/pid
php /Users/btrahan/Dropbox/code/phalanx/scripts/daemon/phd-daemon PhabricatorGarbageCollectorDaemon --daemonize --log=/var/tmp/phd/log/daemons.log --phd=/var/tmp/phd/pid
php /Users/btrahan/Dropbox/code/phalanx/scripts/daemon/phd-daemon PhabricatorTaskmasterDaemon --daemonize --log=/var/tmp/phd/log/daemons.log --phd=/var/tmp/phd/pid
php /Users/btrahan/Dropbox/code/phalanx/scripts/daemon/phd-daemon PhabricatorTaskmasterDaemon --daemonize --log=/var/tmp/phd/log/daemons.log --phd=/var/tmp/phd/pid
php /Users/btrahan/Dropbox/code/phalanx/scripts/daemon/phd-daemon PhabricatorTaskmasterDaemon --daemonize --log=/var/tmp/phd/log/daemons.log --phd=/var/tmp/phd/pid
php /Users/btrahan/Dropbox/code/phalanx/scripts/daemon/phd-daemon PhabricatorTaskmasterDaemon --daemonize --log=/var/tmp/phd/log/daemons.log --phd=/var/tmp/phd/pid

Stop these processes by re-running this command with the --force parameter.
14:09:47 ~/Dropbox/code/phalanx (T2374)
~> sudo ./bin/phd stop --force
Interrupting daemon 'Rogue daemon' (66167)...
Interrupting daemon 'Rogue daemon' (66174)...
Interrupting daemon 'Rogue daemon' (66177)...
Interrupting daemon 'Rogue daemon' (66191)...
Interrupting daemon 'Rogue daemon' (66193)...
Interrupting daemon 'Rogue daemon' (66196)...
Interrupting daemon 'Rogue overseer' (66166)...
Interrupting daemon 'Rogue overseer' (66169)...
Interrupting daemon 'Rogue overseer' (66175)...
Interrupting daemon 'Rogue overseer' (66189)...
Interrupting daemon 'Rogue overseer' (66192)...
Interrupting daemon 'Rogue overseer' (66195)...
Daemon 66167 exited.
Daemon 66174 exited.
Daemon 66177 exited.
Daemon 66191 exited.
Daemon 66193 exited.
Daemon 66196 exited.
Daemon 66166 exited.
Daemon 66169 exited.
Daemon 66175 exited.
Daemon 66189 exited.
Daemon 66192 exited.
Daemon 66195 exited.
```

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T2374

Differential Revision: https://secure.phabricator.com/D10354
2014-08-26 14:12:31 -07:00
Bob Trahan
6f246bd351 Daemons - add a config check for out of date daemon environment
Summary: Fixes T4881.

Test Plan: made a config change, saw the issue, restarted daemons and it went away

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T4881

Differential Revision: https://secure.phabricator.com/D10339
2014-08-22 14:52:36 -07:00
epriestley
15385e1fe9 Don't SIGINT ourselves on bad daemon data
Summary:
If daemon data is mangled, `bin/phd restart` will SIGINT process `0`, which kills it.

uh oh T.T so sad

Test Plan: Used `bin/phd start` to start daemons; removed PID information from one; saw `bin/phd stop` shut down cleanly and not kill itself.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: mholden, epriestley

Differential Revision: https://secure.phabricator.com/D10308
2014-08-20 13:18:17 -07:00
epriestley
9309723ac4 Send graceful shutdown signals to daemons in Phabricator
Summary:
Fixes T5855. Adds a `--graceful N` flag to `phd stop` and `phd restart`.

`phd` will send SIGINT, wait `N` seconds, SIGTERM, wait 15 seconds, and SIGKILL. By default, `N` is 15.

Test Plan:
  - Ran `bin/phd debug ...` and used `^C` to interrupt daemons. Saw graceful shutdown behavior, and abrupt termination on multiple `^C`.
  - Ran `bin/phd start`, `bin/phd stop` and `bin/phd restart` with `--graceful` set to various things, notably `0`. Saw graceful shutdowns on the CLI and in the web UI. With `0`, abrupt shutdowns.

Reviewers: btrahan, hach-que

Reviewed By: hach-que

Subscribers: epriestley

Maniphest Tasks: T5855

Differential Revision: https://secure.phabricator.com/D10228
2014-08-11 20:18:31 -07:00
Joshua Spence
86c399b657 Rename PhabricatorApplication subclasses
Summary: Ref T5655. Some discussion in D9839. Generally speaking, `Phabricator{$name}Application` is clearer than `PhabricatorApplication{$name}`.

Test Plan:
# Pinned and uninstalled some applications.
# Applied patch and performed migrations.
# Verified that the pinned applications were still pinned and that the uninstalled applications were still uninstalled.
# Performed a sanity check on the database contents.

Reviewers: btrahan, epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: hach-que, epriestley, Korvin

Maniphest Tasks: T5655

Differential Revision: https://secure.phabricator.com/D9982
2014-07-23 10:03:09 +10:00
James Rhodes
9cb6b2cfcc Remove user-independent date and time functions from Phabricator
Summary: These have been moved into libphutil.

Test Plan: Browsed Phabricator, didn't see a crash.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9907
2014-07-13 12:03:17 +10:00
Joshua Spence
9a679bf374 Allow worker tasks to have priorities
Summary: Fixes T5336. Currently, `PhabricatorWorkerLeaseQuery` is basically FIFO. It makes more sense for the queue to be a priority-queue, and to assign higher priorities to alerts (email and SMS).

Test Plan: Created dummy tasks in the queue (with different priorities). Verified that the priority field was set correctly in the DB and that the priority was shown on the `/daemon/` page. Started a `PhabricatorTaskmasterDaemon` and verified that the higher priority tasks were executed before lower priority tasks.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Maniphest Tasks: T5336

Differential Revision: https://secure.phabricator.com/D9871
2014-07-12 03:02:06 +10:00
epriestley
950d81dac0 Continue on bad daemon pid data
Summary: The rest of this code works if we hand off `array()`, and fataling here, while more correct, is harder for users to get out of (they have to go manually remove files) and not obvious.

Test Plan: Corrupted pid file and ran `phd stop`.

Reviewers: joshuaspence

Reviewed By: joshuaspence

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D9749
2014-06-26 15:23:22 -07:00
epriestley
46d9bebc84 Remove all device = true from page construction
Summary: Fixes T5446. Depends on D9687.

Test Plan: Mostly regexp'd this. Lint doesn't complain.

Reviewers: chad

Reviewed By: chad

Subscribers: epriestley, hach-que

Maniphest Tasks: T5446

Differential Revision: https://secure.phabricator.com/D9690
2014-06-23 15:18:14 -07:00
epriestley
ca6bd26475 Set device to false for all pages which don't specify device readiness
Summary:
Ref T5446.

  - For all callsites which do not specify a value, set `false` explicitly.
  - Make `true` the default.

Test Plan: Used `grep`, then manually went through everything.

Reviewers: chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T5446

Differential Revision: https://secure.phabricator.com/D9687
2014-06-23 15:15:11 -07:00
Joshua Spence
ddd8c9c567 Add a --local flag for the ./bin/phd status workflow.
Summary: It is sometimes useful to use `./bin/phd status` as a means to determine if daemons //are// actually running on the current host. For example, a common practice in upstart scripts is something similar to `./bin/phd status || ./bin/phd status`.

Test Plan:
```
> ./bin/phd status
ID   Host            PID  Started                 Daemon                               Arguments
1162 ip-10-127-58-93 4046 Jun 20 2014, 3:17:43 AM PhabricatorFactDaemon
1161 ip-10-127-58-93 3984 Jun 20 2014, 3:17:43 AM PhabricatorTaskmasterDaemon
1160 ip-10-127-58-93 3973 Jun 20 2014, 3:17:42 AM PhabricatorTaskmasterDaemon
1159 ip-10-127-58-93 3968 Jun 20 2014, 3:17:42 AM PhabricatorTaskmasterDaemon
1158 ip-10-127-58-93 3943 Jun 20 2014, 3:17:42 AM PhabricatorTaskmasterDaemon
1157 ip-10-127-58-93 3914 Jun 20 2014, 3:17:41 AM PhabricatorGarbageCollectorDaemon
1156 ip-10-127-58-93 3909 Jun 20 2014, 3:17:41 AM PhabricatorRepositoryPullLocalDaemon

> ./bin/phd status --local
There are no running Phabricator daemons.
```

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9645
2014-06-22 06:09:53 +10:00
Joshua Spence
f52fbf6117 Unify the local and global view for ./bin/phd status.
Summary:
Ref T4209. Unifies the local (`./bin/phd status`) and global (`./bin/phd status --all`) view into a single table. This generally makes it easy to administer daemons running across multiple hosts.

Depends on D9606.

Test Plan:
```
> sudo ./bin/phd status
ID Host      PID  Started                 Daemon                               Arguments
38 localhost 2282 Jun 18 2014, 7:52:56 AM PhabricatorRepositoryPullLocalDaemon
39 localhost 2289 Jun 18 2014, 7:52:57 AM PhabricatorGarbageCollectorDaemon
40 localhost 2294 Jun 18 2014, 7:52:57 AM PhabricatorTaskmasterDaemon
41 localhost 2314 Jun 18 2014, 7:52:58 AM PhabricatorTaskmasterDaemon
42 localhost 2319 Jun 18 2014, 7:52:59 AM PhabricatorTaskmasterDaemon
43 localhost 2328 Jun 18 2014, 7:53:00 AM PhabricatorTaskmasterDaemon
44 localhost 2354 Jun 18 2014, 7:53:08 AM PhabricatorRepositoryPullLocalDaemon X --not Y
```

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T4209

Differential Revision: https://secure.phabricator.com/D9607
2014-06-18 11:44:52 +10:00
epriestley
b93fe30008 Update daemon status icons to FontAwesome
Summary: Fixes T5400. Couple of these were missed.

Test Plan: Forced daemons into all statuses, viewed icons.

Reviewers: chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T5400

Differential Revision: https://secure.phabricator.com/D9612
2014-06-17 15:44:42 -07:00
epriestley
df3736e81e Garbage collect daemon logs
Summary:
We already have GC for daemon log events, but not for daemon logs themselves.

Collect old daemon logs which aren't still running.

Test Plan: Ran `phd debug garbage`, observed old logs get cleaned up. Started some daemons, re-ran garbage, made sure they stuck around.

Reviewers: joshuaspence

Reviewed By: joshuaspence

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D9610
2014-06-17 15:33:08 -07:00
Joshua Spence
7892627ccc Add a withoutIDs method to PhabricatorDaemonLogQuery.
Summary: Add a method to `PhabricatorDaemonLogQuery` to exclude IDs from the results.

Test Plan: Thought long and hard.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9606
2014-06-18 08:07:20 +10:00
Joshua Spence
dde6d2491c Query daemons across all hosts with ./bin/phd status --all.
Summary: This was previously submitted as D9497, but I had accidentally `arc land`ed some not-reviewed not-yet-complete changes in addition to the accepted diff.

Test Plan: Same as D9497.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Maniphest Tasks: T5388, T4209

Differential Revision: https://secure.phabricator.com/D9589
2014-06-17 11:13:38 +10:00
epriestley
a10f969919 Revert "Query daemons across all hosts with ./bin/phd status --all."
This reverts commit 0ccebbe4b1.
2014-06-16 17:43:45 -07:00
Joshua Spence
0ccebbe4b1 Query daemons across all hosts with ./bin/phd status --all.
Summary: Ref T4209. Currently, `./bin/phd status` prints a table showing the daemons that are executing on the current host. It would be useful to be able to conventiently query the daemons running across all hosts. This would also (theoretically) make it possible to conditionally start daemons on a host depending upon the current state and on the daemons running on other hosts.

Test Plan:
```
> ./bin/phd status --all
ID      Host                    PID     Started                     Daemon                                            Arguments
   18          phabricator     6969   Jun 12 2014, 4:44:22 PM     PhabricatorTaskmasterDaemon
   17          phabricator     6961   Jun 12 2014, 4:44:19 PM     PhabricatorTaskmasterDaemon
   16          phabricator     6955   Jun 12 2014, 4:44:15 PM     PhabricatorTaskmasterDaemon
   15          phabricator     6950   Jun 12 2014, 4:44:14 PM     PhabricatorTaskmasterDaemon
   14          phabricator     6936   Jun 12 2014, 4:44:13 PM     PhabricatorGarbageCollectorDaemon
   13          phabricator     6931   Jun 12 2014, 4:44:12 PM     PhabricatorRepositoryPullLocalDaemon
```

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T4209

Differential Revision: https://secure.phabricator.com/D9497
2014-06-17 08:41:51 +10:00
Joshua Spence
a3c8b1e746 Fix a typo.
Summary: Self-explanatory.

Test Plan: Eyeball it.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9563
2014-06-16 08:24:35 +10:00
Joshua Spence
0a62f13464 Change double quotes to single quotes.
Summary: Ran `arc lint --apply-patches --everything` over rP, mainly to change double quotes to single quotes where appropriate. These changes also validate that the `ArcanistXHPASTLinter::LINT_DOUBLE_QUOTE` rule is working as expected.

Test Plan: Eyeballed it.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin, hach-que

Differential Revision: https://secure.phabricator.com/D9431
2014-06-09 11:36:50 -07:00
Chad Little
41ef6824be Make ObjectItem default as "Card"
Summary: This went smoother than expeced. Makes the rounded Card the default, also tweaked selected state a little.

Test Plan:
Test UIExamples, Maniphest, Home, Differential, Harbormaster, Audit. Everything seems normal

{F163971}

{F163973}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D9408
2014-06-07 12:12:11 -07:00
epriestley
81d95cf682 Make default view of "Applications" app a full-page launcher
Summary:
This probably needs some tweaks, but the idea is to make it easier to browse and access applications without necessarily needing them to be on the homepage.

Open to feedback.

Test Plan:
(This screenshot merges "Organization", "Communication" and "Core" into a single "Core" group. We can't actually do this yet because it wrecks the homepage.)

{F160052}

Reviewers: btrahan, chad

Reviewed By: chad

Subscribers: epriestley

Maniphest Tasks: T5176

Differential Revision: https://secure.phabricator.com/D9297
2014-05-29 12:17:54 -07:00
epriestley
5e7b316fbe Free task leases on "phd start"
Summary:
Fixes T5154. Currently, "phd stop" terminates daemons relatively abruptly (and other things do too, like killing them). This can leave them with long leases that won't expire any time soon. Normally this isn't a big deal, since it just means an email or an import takes a bit longer (often 2 hours, but up to 24 hours) to run. However:

  - We've increased default lease durations a lot fairly recently -- the 2 hours used to be 15 minutes.
  - Harbormaster and Drydock add new types of tasks which are more dependent on other tasks, so waiting 2 hours for something to free up can hold up more stuff in queue.

When `phd start` is run, we can be confident (at least, in normal circumstances) that leases are safe to free, since we do a check. This undoes any damage done by abrupt stops in "phd stop" or by users or systems killing stuff.

(It would be nice to make "phd stop" more graceful at some point, but we always have to deal with abrupt termination in some cases no matter how gentle "phd stop" is.)

One sort-of-questionable thing here is that we don't distinguish between tasks which had an active lease and tasks which had been released, since the system itself does not make a distiction. So, for example, if you have a task that retries 5 times and waits an hour between retries, you'll get a retry on every `phd start` now, and could exhaust them all in a few minutes if you cycle `phd start` aggressively. I think this is OK. In the future, we could try to distinguish between these types of tasks, and only free the ones with active leases.

Test Plan:
  - Used `phd start` normally, saw it free leases.
  - Used `phd start`, killed it real quick so no taskmasters spawned, ran it again an saw no leases freed.
  - Used `phd start --keep-leases`.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T5154

Differential Revision: https://secure.phabricator.com/D9256
2014-05-22 10:40:45 -07:00
Joshua Spence
eba11238b9 Show daemon arguments with ./bin/phd status.
Summary: Fixes T4735. When running `./bin/phd`, show daemon arguments.

Test Plan:
```
./bin/phd status
PID  	Started                 	Daemon                                            Arguments
12711	May 20 2014, 9:02:52 AM 	PhabricatorRepositoryPullLocalDaemon              []
12716	May 20 2014, 9:02:52 AM 	PhabricatorGarbageCollectorDaemon                 []
12733	May 20 2014, 9:02:53 AM 	PhabricatorTaskmasterDaemon                       []
12768	May 20 2014, 9:02:53 AM 	PhabricatorTaskmasterDaemon                       []
12775	May 20 2014, 9:02:53 AM 	PhabricatorTaskmasterDaemon                       []
12780	May 20 2014, 9:02:54 AM 	PhabricatorTaskmasterDaemon                       []
12838	May 20 2014, 9:02:54 AM 	PhabricatorFactDaemon                             []
13436	May 20 2014, 9:03:23 AM 	PhabricatorRepositoryPullLocalDaemon              ["X","--not","Y"]
```

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Maniphest Tasks: T4735

Differential Revision: https://secure.phabricator.com/D9208
2014-05-20 16:47:47 -07:00
Joshua Spence
b6d15377dd Show daemon arguments when launching a daemon with ./bin/phd.
Summary: Ref T4735. When launching daemons (with `./bin/phd start` or `./bin/phd launch`), print the arguments that are specified for the daemon.

Test Plan:
Ran `./bin/phd launch repo -- --not X`.

```
> sudo ./bin/phd launch repo -- --not X
Preparing to launch daemons.
NOTE: Logs will appear in '/mnt/logs/phd/daemons.log'.

Launching daemon "PhabricatorRepositoryPullLocalDaemon" with arguments ["--not","X"].
```

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Maniphest Tasks: T4735

Differential Revision: https://secure.phabricator.com/D9205
2014-05-20 16:45:22 -07:00
Chad Little
0120388a75 Found some missing icons
Summary: Did a more exhaustive grep on setIcon and found 99.9% of the icons.

Test Plan: I verified icon names on UIExamples, but unable to test some of the more complex flows visually. Mostly a read and replace.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: epriestley, Korvin, hach-que

Differential Revision: https://secure.phabricator.com/D9088
2014-05-13 07:45:39 -07:00
epriestley
136af8d2ab Do not perform write in PhabricatorDaemonLogQuery by default
Summary:
See <http://github.com/facebook/phabricator/issues/487>. By default, we perform a write in this query to moved daemons to "dead" status after a timeout. This is normally reasonable, but after D7964 we do a setup check against the daemons, which means this query is invoked very early in the stack, before we have a write guard.

Since doing this write unconditionally is unnecessarily, surprising, and overly ambitious, make the write conditional and do not attempt to perform it from the setup check.

(We could also move this to a GC/cron sort of thing eventually, maybe -- it's a bit awkward here, but we don't have other infrastructure which is a great fit right now.)

Test Plan: Hit setup issues and daemon pages. Will confirm with user that this fixes things.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D8023
2014-01-21 14:04:12 -08:00
Chad Little
75e7224c8a Use PHUIObjectBoxView on Daemons
Summary: Consistent headers.

Test Plan: Reviewed my running daemons

Reviewers: epriestley

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D8013
2014-01-20 12:08:09 -08:00
epriestley
9f35c7cc26 Complete modularization of the GC daemon
Summary: This modularizes the rest of the GC submethods. Turned out there was nothing tricky.

Test Plan: Ran `bin/phd debug garbage` and got reasonable looking behavior and output.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D7971
2014-01-15 10:02:31 -08:00
Chad Little
31a2bebf63 Move PhabricatorTagView to PHUITagView
Summary: For consistency and great justice.

Test Plan: tested audit, uiexamples, action headers

Reviewers: epriestley, btrahan

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7967
2014-01-14 14:09:52 -08:00