1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-24 15:52:41 +01:00
Commit graph

143 commits

Author SHA1 Message Date
epriestley
5d1f94ac8a Fix some Phabricator lint warnings
Summary: Lint.

Test Plan: Lint.

Reviewers: chad

Reviewed By: chad

CC: aran

Differential Revision: https://secure.phabricator.com/D6127
2013-06-04 15:28:24 -07:00
deedydas
d028790dd7 Close all DB connections when Daemons sleep
Summary: Fixes T2933

Test Plan:
As guided by Evan - by setting $tasks = array(); in PhabricatorTaskmasterDaemon.php
and running 'phd debug taskmaster' and 'show full processlist' on mysql as root. No extra connections
detected.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2933

Differential Revision: https://secure.phabricator.com/D5654
2013-04-10 14:52:29 -07:00
Jakub Vrana
3391e3d34b Use (a = ? AND b = ?) instead of (a, b) IN (?, ?)
Summary: MySQL is not able to use indexes with searching for tuples.

Test Plan:
Explained the query before and after, saw `key_len` 16 instead of 8.
Also saw time 0.0 s instead of 2.9 s (but that was probably caused by warming up).

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D5580
2013-04-05 23:02:06 -07:00
epriestley
bd6bb8a3ea In debug mode, log all phabricator bot data to the log
Summary: Dump everything to the debug log for `phd debug phabricatorbot ...`

Test Plan: Ran `phd debug phabricatorbot ...`

Reviewers: chad, AnhNhan

Reviewed By: chad

CC: aran

Differential Revision: https://secure.phabricator.com/D5239
2013-03-05 12:56:36 -08:00
epriestley
7d771b4ff7 Support "M" in phid.lookup and ircbot
Summary: Fixes T2651. This could be futher generalized but it's a bit out of the way.

Test Plan: See chatlog.

Reviewers: chad

Reviewed By: chad

CC: aran

Maniphest Tasks: T2651

Differential Revision: https://secure.phabricator.com/D5236
2013-03-05 12:31:52 -08:00
kwadwo
762ace810d Allow files to TTL and and be garbage collected
Summary: Added ttl field to files. Gabage collect files with expired ttl

Test Plan: created file with a ttl. Let garbage collector run

Reviewers: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D4987
2013-02-20 13:38:36 -08:00
John Watson
204d6481e4 Fix PhabricatorBot ignore messages from senders
Summary:
PhabricatorBotMessage->getSender returns a PhabricatorBotUser object (which potentially can be null)
So check null and then use getName to get actual name of the sender

Test Plan: Run phabot and add myself to ignore list

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D5036
2013-02-20 12:30:54 -08:00
epriestley
a22bea2a74 Apply lint rules to Phabricator
Summary: Mostly applies a new call spacing rule; also a few things that have slipped through via pull requests and such

Test Plan: `find src/ -type f -name '*.php' | xargs -n16 arc lint --output summary --apply-patches`

Reviewers: chad

Reviewed By: chad

CC: aran

Differential Revision: https://secure.phabricator.com/D5002
2013-02-19 13:33:10 -08:00
epriestley
4f2aa99248 Add "serviceName" and "serviceType" to bot and chat logger
Summary:
Make each adapter provide a "serviceType" (campfire, flowdock, IRC) and "serviceName" (irc network / chat server) so that we can disambiguate between, e.g., "#phabricator on EFNet" and "#phabricator on FreeNode".

Make the chatlog handler ship them over Conduit.

Also fix some "policy can not be null" bugs with chatlog recording.

Test Plan:
Verified data inserted correctly:

  mysql> select * from chatlog_channel;
  +----+------------------+-------------+--------------+------------+------------+-------------+--------------+
  | id | serviceName      | serviceType | channelName  | viewPolicy | editPolicy | dateCreated | dateModified |
  +----+------------------+-------------+--------------+------------+------------+-------------+--------------+
  |  1 | irc.freenode.net | IRC         | #phabricator | users      | users      |  1361201689 |   1361201689 |
  +----+------------------+-------------+--------------+------------+------------+-------------+--------------+
  1 row in set (0.00 sec)

  mysql> select * from chatlog_event where channelID = 1;
  +----+--------------+------------+------------+------+---------------+--------------------------------+-----------+
  | id | channel      | epoch      | author     | type | message       | loggedByPHID                   | channelID |
  +----+--------------+------------+------------+------+---------------+--------------------------------+-----------+
  | 45 | #phabricator | 1361201689 | epriestley | mesg | blip blip     | PHID-USER-5bt2phfepag4cdvjtzg5 |         1 |
  | 46 | #phabricator | 1361201700 | epriestley | mesg | boop boop bip | PHID-USER-5bt2phfepag4cdvjtzg5 |         1 |
  +----+--------------+------------+------------+------+---------------+--------------------------------+-----------+
  2 rows in set (0.00 sec)

Reviewers: Afaque_Hussain, indiefan

Reviewed By: Afaque_Hussain

CC: aran

Maniphest Tasks: T837

Differential Revision: https://secure.phabricator.com/D4996
2013-02-18 07:50:41 -08:00
epriestley
26aac16346 Garbage collect TTL'd cache entries from the general cache
Summary: We currently garbage collect general cache entries after a set period of time (30 days by default), but the recent changes to DarkConsole have left us writing a lot of large, short-TTL data to the cache. In addition to a maximum age, GC cache entires after they TTL out.

Test Plan: Ran GC daemon, saw TTL'd entries get collected. Inserted a TTL'd entry, saw it get collected by GC. Saw non-ttl'd entries not get collected.

Reviewers: chad

Reviewed By: chad

CC: aran

Differential Revision: https://secure.phabricator.com/D4990
2013-02-17 09:13:49 -08:00
indiefan
5fb56f859c Added Flowdock protocol adapter for the bot. Refactored campfire bot into a base streaming protocol adapter for common functionality.
Summary: First pass. Flowdock supports interesting message types (like replies to messages), but for now implementing a standard messaging interface.

Test Plan: Ran both a Flowdock bot and a Campfire bot. Made sure both still connected and responded properly to the Object Handler.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D4983
2013-02-15 20:24:24 -08:00
epriestley
dad7c65bf5 Fix some bot issues
Summary:
  - Deprecate differentialnotification in favor of feednotification; it's strictly better.
  - Fix feed notification + channels.
  - Fix rendering of new-style stories (pholio, macro), which currently fatal.

Test Plan: See chatlog.

Reviewers: codeblock, indiefan

Reviewed By: codeblock

CC: aran

Differential Revision: https://secure.phabricator.com/D4981
2013-02-15 17:10:51 -08:00
epriestley
0a8b0d1392 Merge IRCProtocolHandler into IRCAdapter
Summary:
Clearly silly to have a separate handler for this. I also made most of the protocol stuff direct writes so we don't need to ship them through handlers, and made the adapter ignore message it does not understand by default instead of sending them to IRC, and added PASTE "support".

We could still let handlers react to these messages by emitting them all as 'RAWIRC' or similar, but there's currently no need for that so I didn't bother.

Also fix an issue in D4924 with nickpass.

Test Plan: Had bot join IRC, talked to it.

Reviewers: indiefan

Reviewed By: indiefan

CC: aran

Differential Revision: https://secure.phabricator.com/D4925
2013-02-14 05:13:55 -08:00
epriestley
d5995d574d Formalize targets (users and channel) into objects
Summary:
Make users/channels/rooms into objects, so we can later sort out stuff like Campfire user IDs, Phabricator vs chat accounts, etc.

The only change here is that I removed output buffering from the macro handler. We should move throttling/buffering to adapters instead and have it apply globally.

Test Plan: Ran IRC and Campfire bots and interacted with them.

Reviewers: indiefan

Reviewed By: indiefan

CC: aran

Differential Revision: https://secure.phabricator.com/D4924
2013-02-14 05:13:38 -08:00
epriestley
ec306497f5 Lock down bot adapter API slightly
Summary:
  - Reduce visibiliy of config.
  - Add a typehint.

Test Plan: Ran campfire/irc bots and chatted with them.

Reviewers: indiefan

Reviewed By: indiefan

CC: aran, amerigomasini

Differential Revision: https://secure.phabricator.com/D4923
2013-02-14 05:07:50 -08:00
indiefan
eb942f3e1e Updated Campfire adapter to be able to post sound messages and paste messages.
Test Plan: Ran the bot with a handler that sends sound commands.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D4922
2013-02-12 11:30:21 -08:00
indiefan
eaa72c6155 Fixing a bug in the bot ignore logic to use sender.
Summary: Also added sender to the campfire adapter. This isn't extremely useful as it's just a numeric id, but it allows us to add ignores (specifically having the bot ignore itself).

Test Plan: Ran the bot, ignored itself.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D4893
2013-02-09 16:10:33 -08:00
epriestley
7ec8e885e1 Merge branch 'master' into phutil_tag
(Final final sync.)
2013-02-08 17:29:32 -08:00
epriestley
5f9a063333 Use some HTTPSFuture in CampfireBot
Summary:
  - Use PhutilURI to correct for specifying "https://yourname.campfire.com/" instead of "https://yourname.campfire.com".
  - Use HTTPSFuture to get logging via `--trace` and error detection (CA stuff should be OK since 37signals has real certs).
  - On destruction, only try to leave rooms we've actually joined.

Test Plan: Setup a bot, had it join a room, talked to it.

Reviewers: indiefan

Reviewed By: indiefan

CC: aran

Differential Revision: https://secure.phabricator.com/D4849
2013-02-07 10:32:33 -08:00
epriestley
11bb8db970 Merge branch 'master' into phutil_tag
(Sync.)
2013-02-07 08:08:01 -08:00
indiefan
431e2bee6e First (rough) pass at campfire protocol adapter for bot.
Summary:
Decided the best approach for refactoring the message/command stuff would be to actually start implementing the campfire adapter to get a better idea of what the abstractions should look like. It feels awkward and unwieldy trying to maintain the irc command interface (notice the message instantiation in the `processReadBuffer()` method. However, i'm still not clear what the best approach is without requiring a re-write of nearly all the existing handlers and defining essentially a custom dsl on top of irc's.

I suppose given that alternative, implementing to irc's dsl doesn't sound all that bad. Just feels like poor coupling.

Also, I know that there is some http stuff in libphutil's futures library, but the https future is shit and I need to do some custom curlopt stuff I wasn't sure how to do with that. But if you think this should be refactored, let me know.

I tested this with the ObjectHandler (messages with DXXX initiate the bot to respond with the title/link just as with irc), but beyond that, I haven't tried any of the other handlers, so if there are complications you think i'm going to run into, just let me know (this is one of the reasons for requesting review early on).

Also, this diff is against my last one, even though that hasn't been merged down yet. It was starting to get large and I'd prefer to keep to two conversations separate.

Fixing some lint issues.

Test Plan: Ran the bot with the Object Handler in campfire and observed it behaving properly.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2462

Differential Revision: https://secure.phabricator.com/D4830
2013-02-07 06:34:06 -08:00
vrana
f864d9e611 Fix double escaping in phutil_tag
Summary:
I wasn't able to reproduce the "recursion detected" in real web request but I saw lots of 1073741824 refcounts in `debug_zval_dump()` of $object.
I'm not sure how that happens.

Test Plan: D4807#4

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2432

Differential Revision: https://secure.phabricator.com/D4839
2013-02-06 15:21:05 -08:00
indiefan
574bc3ba31 First pass at decoupling Phabricator bot behavior from the protocol it's running on, this pulls the connection, reading, and writing functionalities out of the bot itself and into the adapter.
Summary:
Ugh, just wrote out a huge message, only to lose it with a fat-fingered ctrl-c. Le sigh.

First pass at decoupling the bot from the protocol. Noticeably absent is the command/message coupling. After this design pass I'll give that a go. Could use some advice, thinking that handlers should only create messages (which can be public or private) and not open ended, undefined 'commands'. The problem being that there needs to be some consistant api if we want handlers to be protocol agnostic. Perhaps that's a pipedream, what are your thoughts?

Secondly, a few notes, design review requests on the changes i did make:
 # Config. For now i'm passing config through to the adapter. This was mainly to remain backwards compatible on the config. I was thinking it should probably be namespaced into it's own subobject though to distinguish the adapter config from the bot config.
 # Adapter selection. This flavor is the one-bot-daemon, config specified protocol version. The upside is that in the future they won't have to run different daemons for this stuff, just have different config, and the door is open for multiple protocol adapters down the road if need be. The downside is that I had to rename the daemon (non-backwards compatible change) and there will need to be some sort of runtime evaluation for instatiation of the adapter. For now I just have a crude switch, but I was thinking of just taking the string they supply as the class name (ala `try { new $clasName(); } catch...`) so as to allow for homegrown adapters, but I wasn't sure how such runtime magic would go over. Also, an alternative would be to make the PhabricatorBot class a non-abstract non-final base class and have the adapters be accompanied by a bot class that just defines their adapter as a property. The upside of which is backwards compatibility (welcome back PhabricatorIRCBot) and perhaps a little bit clearer plugin path for homegrowners.
 # Logging. You'll notice I commented out two very important logging lines in the irc adapter. This isn't intended to remain commented out, but I'm not sure what the best way is to get logging at this layer. I'm wary of just composing the daemon back down into the adapter (bi-directional object composition makes my skin crawl), but something needs to happen, obviously. Advice?

That's it. After the feedback on the above, you can either merge down, or wait until i finish the command/message refactor if you don't think the diff will grow too large. Up to you, this all functions as is.

Test Plan: Ran an irc bot, connected, read input, and wrote output including handler integration.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2462

Differential Revision: https://secure.phabricator.com/D4757
2013-02-05 18:46:54 -08:00
epriestley
f6622f43e6 Replace all array_combine(x, x) with array_fuse(x) in Phabricator
Summary: Fixes various array_combine() warnings for PHP < 5.4

Test Plan: lint/unit/grep

Reviewers: btrahan, vrana, chad

Reviewed By: chad

CC: aran

Differential Revision: https://secure.phabricator.com/D4660
2013-01-25 17:06:55 -08:00
epriestley
22c64c67ff Fix performance problem for large task queues
Summary:
Some time ago, we added `ORDER BY id ASC` to the worker `UPDATE ...` query, because someone reported that their MySQL read slaves were complaining about the query (I can't find the exact error message, but something to the effect of the rows the query affected not being deterministic). This seemed harmless since it should be the same as the query's implicit order (I guess?), but actually made the query dramatically slower for large numbers of rows.

On my local machine, this query takes about 2 seconds with ~1M rows. If I run `SELECT`, or run `UPDATE` without ORDER BY, the query takes < 0.01s. I don't understand exactly what's happening -- my guess is something to do with the ORDER BY implying that a lot of rows need to be locked?

In T2372, a user is seeing 20-60s rumtimes on this query.

I solved this by doing a SELECT, followed by an UPDATE. Each query runs quickly. This introduces the possibility of a race (two processes SELECT the same rows, then try to UPDATE), which we currently recover from by having the second UPDATE fail and then having that daemon try again 1 second later. This seems generally reasonable. Some alternatives I considered:

  - We could SELECT ... LOCK FOR UPDATE, but failing and retrying a little later seems at least as good as blocking.
  - We could select more rows than we need, and then try to lock some of them randomly. I think this would work well, but it's a bit more complex than what we're doing now so I left it until we have a clearer need.

Test Plan:
Inserted ~1M tasks into the queue. Ran `phd debug taskmaster`, saw ~2s task updates. Applied patch. Ran `phd debug taskmaster`, saw <1ms updates. Ran `phd launch 8 taskmaster`, saw rapid completion of tasks.

This stuff also has fairly thorough unit test coverage.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Maniphest Tasks: T2372

Differential Revision: https://secure.phabricator.com/D4576
2013-01-22 12:27:18 -08:00
epriestley
35d73414f8 Remove legacy support for 'phd repository-launch' and 'phd repository-launch-readonly'
Summary: These have been marked as deprecated since May 2012. Clean them up.

Test Plan: Grepped for `repository-launch`, `phd_load_tracked_repositories`: no hits.

Reviewers: btrahan, vrana

Reviewed By: vrana

CC: aran

Maniphest Tasks: T2372

Differential Revision: https://secure.phabricator.com/D4575
2013-01-22 12:26:08 -08:00
John Watson
e948073107 Global ignore list for IRCBot
Summary: ignore - array - Array of nicks to ignore all mesages from

Test Plan: run phabot with ignore set

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D4582
2013-01-22 12:00:26 -08:00
John Watson
ff53b7942a Initial PhabricatorIRCFeedNotificationHandler
Summary:
Follows Phabricator's feed and puts notifications into channels
that are configured.

~~notification.all - bool - 1:1 stories to messages~~
notification.types - array - Specific story types to notify for - ["differential", "maniphest"]
notification.verbosity - int - Range of 0-3 for verbosity
notification.max_pages - int - Maximum number of pages to go back per poll
notification.page_size - int - Size of pages (limit) to poll
~~notification.channels - array - Array of channels to send messages to~~
~~notification.sleep - int - Seconds to sleep between polls~~

Test Plan: Run phabot with various configuration options

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin, asherkin

Differential Revision: https://secure.phabricator.com/D4418
2013-01-19 05:45:17 -08:00
vrana
9f549ba75e Fix whitespace around else 2013-01-16 12:16:37 -08:00
John Watson
ec19c3332a Break IRCSymbolHandler from IRCObjectNameHandler
Summary: Allows to easily disable responding to "where is..."

Test Plan: Run ircbot with and without the handler

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D4444
2013-01-15 13:17:40 -08:00
epriestley
cbe9aea876 Enable log discard modes for all scripts
Summary:
Fixes T2273. We currently discard logs, service calls, etc., for daemons, but not for other scripts. However, other scripts may be long-running or issue a large body of service calls (e.g., `bin/search index --all`). We never retrieve this information from scripts (it is used to build darkconsole; in scripts, we echo it immediately under --trace), so discard it immediately to prevent these scripts from requiring a large amount of memory.

(When the daemons load `__init_script__.php` they end up calling this code, so this doesn't change anything for them. They hit another ServiceProfiler discard along the daemon pathways in libphutil, but the call is idempotent.)

Test Plan: Ran `bin/search index --all` and saw increasing memory usage before this patch, but steady memory usage after this patch.

Reviewers: btrahan, vrana, codeblock

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2273

Differential Revision: https://secure.phabricator.com/D4364
2013-01-08 15:54:08 -08:00
Ricky Elrod
f12af03836 Make phabot never private message anyone.
Summary:
This replaces D4175 and makes it so phabot doesn't message anyone.

The reasons for this are twofold:

- It was possible to get information from the bot, by private messaging it, even
  if the bot was only in a +i channel (on a public network) -- meaning that if
  someone knew the nickname of the bot, they could obtain e.g. ticket names
  or diff titles.
- The other time it messaged people was when you typed e.g. "somenick: T123".
  Most times when this is triggered, it's done so on accident.

See discussion on the old revision (D4175).

Test Plan:
  15:29:33 ::: Irssi: Starting query in quartz with cb-phabot
  15:29:38 <relrod> T2
  (nothing back)

and

  15:29:21 <@relrod> rublets: T1
  15:29:21 < cb-phabot> T1: asdfasdf (Priority: Needs Triage) - http://local.elrod.me/T1

Reviewers: epriestley, btrahan, vrana

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D4339
2013-01-03 12:44:36 -08:00
epriestley
a3fdb20a8e Move GC into PHP and simplify it
Summary:
  - Move GC options into PHP.
  - Remove the "run at" and "run for" options. The GC daemon doesn't actually do any table scans, is very gentle, and runs for like 3 seconds per day in any normal install. Just limit it to running once every 4 hours when it's caught up and call it a day.

Test Plan: Edited GC options.

Reviewers: btrahan, codeblock

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4321
2013-01-02 14:03:08 -08:00
epriestley
aae5f9efd3 Implement a more compact, general database-backed key-value cache
Summary:
See discussion in D4204. Facebook currently has a 314MB remarkup cache with a 55MB index, which is slow to access. Under the theory that this is an index size/quality problem (the current index is on a potentially-384-byte field, with many keys sharing prefixes), provide a more general index with fancy new features:

  - It implements PhutilKeyValueCache, so it can be a component in cache stacks and supports TTL.
  - It has a 12-byte hash-based key.
  - It automatically compresses large blocks of data (most of what we store is highly-compressible HTML).

Test Plan:
  - Basics:
    - Loaded /paste/, saw caches generate and save.
    - Reloaded /paste/, saw the page hit cache.
  - GC:
    - Ran GC daemon, saw nothing.
    - Set maximum lifetime to 1 second, ran GC daemon, saw it collect the entire cache.
  - Deflate:
    - Selected row formats from the database, saw a mixture of 'raw' and 'deflate' storage.
    - Used profiler to verify that 'deflate' is fast (12 calls @ 220us on my paste list).
  - Ran unit tests

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D4259
2012-12-21 14:17:56 -08:00
Ricky Elrod
93c4030732 Fix IRC bot Dxxx Conduit call.
Summary: Some fallout from D4191.

Test Plan: Sent "D12" in IRC and got a response.

Reviewers: epriestley, vrana, zeeg

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D4235
2012-12-18 20:41:52 -08:00
Nick Harper
4a81ae6d6d Add data information to daemon task view
Summary:
Load the data for daemon worker tasks when viewing them, and present
the information in a useful way. This defaults to printing the json data,
but for some classes of worker it will also link to the corresponding
object, to make debugging problems with workers easier.

Test Plan:
load /daemon/task/NNN for a CommitParserWorker and a MetaMTAWorker, and
see the addition of a data field with useful content and link.

Reviewers: epriestley, vrana

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D4226
2012-12-17 17:12:55 -08:00
epriestley
4c7c518c63 Throw a richer exception when updating tasks with expired leases
Summary: Include task ID and class when raising this exception. I took a brief stab at doing this generically, but (a) we specifically raise this exception outside of normal try/catch because we can't follow normal recovery rules for it and (b) we don't have a reasonable PhutilProxyException or similar right now which would preserve stack traces, and don't have builtin exception nesting support until PHP 5.3.

Test Plan: Faked this exception, verified we get more information in the logs.

Reviewers: btrahan, vrana

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2193

Differential Revision: https://secure.phabricator.com/D4205
2012-12-16 16:27:32 -08:00
epriestley
ee2e85a0bb Fix several migration issues with the Task/Counter patch
Summary:
People hit three issues with D3914:

  - As per T2059, we applied a schema change from a `.php` patch, which currently does not work if you use a different user to make schema changes than for normal use.
    - Since the change in question is idempotent, just move it to a `.sql` patch. We'll follow up in T2059 and fix it properly.
  - Rogue daemons at several installs used old code (expecting autoincrement) to insert into the new table (no autoincrement), thereby creating tasks with ID 0.
    - Rename the table so they'll fail.
    - This also makes the code a little more consistent.
  - Some installs now have tasks with ID 0.
    - Use checks against null rather than against 0 so we can process these tasks.

The major issues this fixes are the schema upgrade failure in T2059, and the infinite loops in T2072 and elsewhere.

This isn't really a fully statisfactory fix. I'll discuss some next steps in T2072.

Test Plan: Created new tasks via MetaMTA/Differential. Ran tasks with `phd debug taskmaster`. Inserted a task 0 and verified it ran and archived correctly.

Reviewers: btrahan, vrana, nh

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2072, T2059

Differential Revision: https://secure.phabricator.com/D3973
2012-11-16 10:19:22 -08:00
epriestley
7332599e03 Provide an IDS_COUNTER mechanism for ID assignment
Summary: See D3912 for discussion. InnoDB may reuse autoincrement IDs after restart; provide a way to avoid it.

Test Plan: Unit tests. Scheduled and executed tasks through `drydock lease --type host` and `phd debug taskmaster`.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D3914
2012-11-07 13:33:07 -08:00
epriestley
7e0ce08154 Make various Drydock CLI/Allocator improvements
Summary:
  - Remove EC2, RemoteHost, Application, etc., blueprints for now. They're very proof-of-concept and Blueprints are getting API changes I don't want to bother propagating for now. Leave the abstract base class and the LocalHost blueprint. I'll restore the more complicated ones once better foundations are in place.
  - Remove the Allocate controller from the web UI. The original vision here was that you'd manually allocate resources in some cases, but it no longer makes sense to do so as all allocations come from leases now. This simplifies allocations and makes the rule for when we can clean up resources clear-cut (if a resource has no more active leases, it can be cleaned up). Instead, we'll build resources like the localhost and remote hosts lazily, when leases come in for them.
  - Add some configuration to manage the localhost blueprint.
  - Refactor `canAllocateResources()` into `isEnabled()` (for config checks) and `canAllocateMoreResources()` (for quota checks, e.g. too many resources are allocated already).
  - Juggle some signatures to align better with a world where blueprints generally do allocate.
  - Add some more logging and error handling.
  - Fix an issue with log ordering.

Test Plan: Allocated some localhost leases.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3902
2012-11-06 15:30:11 -08:00
vrana
ef85f49adc Delete license headers from files
Summary:
This commit doesn't change license of any file. It just makes the license implicit (inherited from LICENSE file in the root directory).

We are removing the headers for these reasons:

- It wastes space in editors, less code is visible in editor upon opening a file.
- It brings noise to diff of the first change of any file every year.
- It confuses Git file copy detection when creating small files.
- We don't have an explicit license header in other files (JS, CSS, images, documentation).
- Using license header in every file is not obligatory: http://www.apache.org/dev/apply-license.html#new.

This change is approved by Alma Chao (Lead Open Source and IP Counsel at Facebook).

Test Plan: Verified that the license survived only in LICENSE file and that it didn't modify externals.

Reviewers: epriestley, davidrecordon

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2035

Differential Revision: https://secure.phabricator.com/D3886
2012-11-05 11:16:51 -08:00
epriestley
0364ccdd5f Fix an issue where PhabricatorWorkerLeaseQuery may lease different tasks than it selects
Summary:
We lock tasks by setting `leaseOwner` to a unique value, but the value is currently unique-to-the-process rather than unique-to-the-query. This means that if a process leases a task, then leases another task, both tasks will have the same `leaseOwner`. This can cause an issue where we go to select the task we just leased and get the other task instead, if we aren't careful about the select construction.

We can avoid this by being clever and making sure the select is constructed correctly, but making the `leaseOwner` unique to the query is much simpler and more foolproof. This guarantees we always select only the rows we just leased.

Also remove `PhabricatorGoodForNothingWorker` since `PhabricatorTestWorker` fills its role of allowing things to be tested, and simplify the unit tests since we don't need to be clever about avoiding this issue any more.

Test Plan: Ran unit tests.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3862
2012-11-01 11:30:49 -07:00
epriestley
f0fdcf1a51 Undumb the Drydock resource allocator pipeline
Summary:
This was the major goal of D3859/D3855, and to a lesser degree D3854/D3852.

As Drydock is allocating a resource, it may need to allocate other resources first. For example, if it's allocating a working copy, it may need to allocate a host first.

Currently, we have the process basically queue up the allocation (insert a task into the queue) and sleep() until it finishes. This is problematic for a bunch of reasons, but the major one is that if allocation takes more resources (host, port, machine, DNS) than you have daemons, they could all end up sleeping and waiting for some other daemon to do their work. This is really stupid. Even if you only take up some of them, you're spending slots sleeping when you could be doing useful work.

To partially get around this and make the CLI experience less dumb, there's this goofy `synchronous` flag that gets passed around everywhere and pushes the workflow through a pile of special cases. Basically the `synchronous` flag causes us to do everything in-process. But this is dumb too because we'd rather do things in parallel if we can, and we have to have a lot of special case code to make it work at all.

Get rid of all of this. Instead of sleep()ing, try to work on the tasks that need to be worked on. If another daemon grabbed them already that's fine, but in the worst case we just gracefully degrade and do everything in process. So we get the best of both worlds: if we have parallelizable tasks and free daemons, things will execute in parallel. If we have nonparallelizable tasks or no free daemons, things will execute in process.

Test Plan: Ran `drydock_control.php --trace` and saw it perform cascading allocations without sleeping or special casing.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3861
2012-11-01 11:30:42 -07:00
epriestley
84ee4cd9f6 Factor out task execution and formalize permanent failures
Summary:
  - Clean up a TODO about permanent failures.
  - Clean up a TODO about failing tasks after too many retries.
  - Clean up a TODO about testing for bad leases.
  - Make the lease/retry implementation more flexible and natural.
  - Make completely bogus tasks fail permanently.
  - Make PhabricatorMetaMTAWorker use new `getWaitBeforeRetry()` (as intended), not hackily implement logic in `getRequiredLeaseTime()`.
  - Document worker hooks for failures and retries.
  - Provide coverage on everything.

Test Plan: Ran unit tests. Ran `bin/phd debug taskmaster`.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3859
2012-11-01 11:30:23 -07:00
epriestley
88fad90c1c Move task leasing to a dedicated query
Summary: This simplifies the fairly thorny logic of leasing tasks a bit. I'm planning to introduce another callsite shortly for Drydock.

Test Plan: Ran `bin/phd debug taskmaster`, observed sensible queries and correct operation.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3855
2012-11-01 11:30:16 -07:00
epriestley
fe329b9738 Modernize worker task detail view
Summary: Make mobile-friendly and provide UI to cancel/retry tasks. Remove display of task data to arbitrary users, as it may be sensitive.

Test Plan:
{F22502}
{F22503}
{F22504}
{F22505}
{F22506}

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3854
2012-10-31 15:22:32 -07:00
epriestley
5903ed650c Move completed tasks to an "archive" table and delete them in the GC
Summary:
Currently, when taskmasters complete a task it is immediately deleted. This prevents us from doing some general things, like:

  - Supporting the idea of permanent failure (e.g., after N failures just stop trying).
  - Showing the user how fast taskmasters are completing tasks.
  - Showing the user how long tasks took to complete.

Having better visibility into this is important to Drydock, which builds on the task system. Also, generally buff debug output for task execution.

Test Plan: Ran `bin/phd debug taskmaster`. Ran `bin/phd debug garbage`. Queued some tasks via various systems.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3852
2012-10-31 15:22:16 -07:00
epriestley
8798d2189e Prevent notice for missing configuration in PhabricatorIRCDifferentialNotificationHandler
Summary: Provide array() default so we don't foreach() over null in the case of a missing config (from @dctrwatson).

Test Plan: Will verify with @dctrwatson.

Reviewers: vrana, btrahan, dctrwatson

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D3686
2012-10-11 15:00:25 -07:00
KorvinSzanto
da2fc57d77 Fix irc server login
Summary: Previously, the identification string was thrown at the server long before you were connected, I've moved this to the end of the motd raw, and now errthangz gud

Test Plan: Register an account for your bot to use, give your bot the correct nick and password, then watch

Reviewers: epriestley

CC: aran

Differential Revision: https://secure.phabricator.com/D3410
2012-08-30 13:52:33 -07:00
KorvinSzanto
6c44587717 Fix "where is symbol" ircbot handler
Summary: In my haste, I forgot a trailing ?

Test Plan: Try both "Where is Derp?" and "Where in the world is Derp?"

Reviewers: epriestley

CC: aran

Differential Revision: https://secure.phabricator.com/D3387
2012-08-25 16:54:48 -07:00