1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-25 00:02:41 +01:00
Commit graph

14 commits

Author SHA1 Message Date
epriestley
9c798e5cca Provide bin/garbage for interacting with garbage collection
Summary:
Fixes T9494. This:

  - Removes all the random GC.x.y.z config.
  - Puts it all in one place that's locked and which you use `bin/garbage set-policy ...` to adjust.
  - Makes every TTL-based GC configurable.
  - Simplifies the code in the actual GCs.

Test Plan:
  - Ran `bin/garbage collect` to collect some garbage, until it stopped collecting.
  - Ran `bin/garbage set-policy ...` to shorten policy. Saw change in web UI. Ran `bin/garbage collect` again and saw it collect more garbage.
  - Set policy to indefinite and saw it not collect garabge.
  - Set policy to default and saw it reflected in web UI / `collect`.
  - Ran `bin/phd debug trigger` and saw all GCs fire with reasonable looking queries.
  - Read new docs.

{F857928}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9494

Differential Revision: https://secure.phabricator.com/D14219
2015-10-02 09:17:24 -07:00
epriestley
878a493301 Begin standardizing garbage collectors
Summary: Ref T9494. Improve support infrastructure for garbage collectors.

Test Plan:
  - Ran `bin/phd debug trigger`, saw collectors execute.

{F857852}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9494

Differential Revision: https://secure.phabricator.com/D14218
2015-10-01 16:58:43 -07:00
epriestley
a3518e19a5 Merge GC daemon into Trigger daemon
Summary:
Fixes T7352. This reduces the memory footprint for instances by combining these two similar daemons into one daemon which handles the responsibilities of both.

The fit isn't 100% perfect here but it's pretty close, and the GC daemon is fairly trivial.

Test Plan:
  - Adjusted all the numbers to small numbers (5 second sleep, 120 second GC length).
  - Added a ton of logging.
  - Started trigger daemon.
    - Saw it run a GC cycle.
    - Saw it reschedule another cycle after 120 seconds (adjusted down from 4 hours).
  - Reverted all the logging/small numbers.
  - Ran `bin/phd start`, saw stable trigger daemon running.
  - Grepped for removed daemon class name.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7352

Differential Revision: https://secure.phabricator.com/D11872
2015-02-24 14:50:39 -08:00
Joshua Spence
62dfcd1e55 Fix the visibility of PhutilDaemon::run methods
Summary: Ref T6822. This method is only called from `PhutilDaemon::execute()` and can be made `protected`.

Test Plan: See D11404.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6822

Differential Revision: https://secure.phabricator.com/D11405
2015-01-16 06:59:29 +11:00
epriestley
9309723ac4 Send graceful shutdown signals to daemons in Phabricator
Summary:
Fixes T5855. Adds a `--graceful N` flag to `phd stop` and `phd restart`.

`phd` will send SIGINT, wait `N` seconds, SIGTERM, wait 15 seconds, and SIGKILL. By default, `N` is 15.

Test Plan:
  - Ran `bin/phd debug ...` and used `^C` to interrupt daemons. Saw graceful shutdown behavior, and abrupt termination on multiple `^C`.
  - Ran `bin/phd start`, `bin/phd stop` and `bin/phd restart` with `--graceful` set to various things, notably `0`. Saw graceful shutdowns on the CLI and in the web UI. With `0`, abrupt shutdowns.

Reviewers: btrahan, hach-que

Reviewed By: hach-que

Subscribers: epriestley

Maniphest Tasks: T5855

Differential Revision: https://secure.phabricator.com/D10228
2014-08-11 20:18:31 -07:00
epriestley
9f35c7cc26 Complete modularization of the GC daemon
Summary: This modularizes the rest of the GC submethods. Turned out there was nothing tricky.

Test Plan: Ran `bin/phd debug garbage` and got reasonable looking behavior and output.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D7971
2014-01-15 10:02:31 -08:00
epriestley
56d44f1503 Modularize the Garbage Collector
Summary:
The GC is a big block of hard-coded application GCs right now. Among other things, this means third parties can't tap into the infrastructure.

Modularize it into `GarbageCollector` classes. This implements only one to prove the new stuff works; I'll followup with the rest in the next diff or few depending on how much mess I run into.

Test Plan: Used `bin/phd debug garbage` to run the collector in debug mode, observed reasonable output and behavior.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D7970
2014-01-15 10:02:24 -08:00
vrana
6cc196a2e5 Move files in Phabricator one level up
Summary:
- `kill_init.php` said "Moving 1000 files" - I hope that this is not some limit in `FileFinder`.
- [src/infrastructure/celerity] `git mv utils.php map.php; git mv api/utils.php api.php`
- Comment `phutil_libraries` in `.arcconfig` and run `arc liberate`.

NOTE: `arc diff` timed out so I'm pushing it without review.

Test Plan:
/D1234
Browsed around, especially in `applications/repository/worker/commitchangeparser` and `applications/` in general.

Auditors: epriestley

Maniphest Tasks: T1103
2012-06-01 12:32:44 -07:00
epriestley
09c8af4de0 Upgrade phabricator to libphutil v2
Summary: Mechanical changes from D2588. No "Class.php" moves yet.

Test Plan: See D2588.

Reviewers: vrana, btrahan, jungejason

Reviewed By: vrana

CC: aran

Maniphest Tasks: T1103

Differential Revision: https://secure.phabricator.com/D2591
2012-05-30 14:26:29 -07:00
epriestley
d0af617818 Add "final" to (almost) everything else
Summary: Last of the big final patches. Left a few debatable classes (12 out of about 400) that I'll deal with individually eventually.

Test Plan: Ran testEverythingImplemented.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T795

Differential Revision: https://secure.phabricator.com/D1881
2012-03-13 16:21:04 -07:00
epriestley
5065db5a2a Reduce spew of some daemons
Summary:
It used to be more useful for daemons to spew random debugging information, but
features like "phd debug" and some fixes to error reporting like D1101 provide
better ways to debug, test, develop and diagnose daemons.

  - Stop writing "." every time MetaMTA sends a message.
  - Stop spewing the entire IRC protocol from the IRC bot unless in debug mode.
  - Stop writing GC daemon log entries about collecting daemon logs (DURRR)
unless in debug mode.

Test Plan: Ran daemons in debug and non-debug modes, got expected level of
noisiness.

Reviewers: jungejason, nh, btrahan

Reviewed By: jungejason

CC: aran, jungejason

Differential Revision: https://secure.phabricator.com/D1268
2011-12-22 17:49:33 -08:00
epriestley
879431fb50 Improve GC performance for Herald Transcripts
Summary: This has to table scan a ginormous table right now, give it a fighting
chance with a more usable key.
Test Plan:
  - Launched GC daemon, no errors.
  - Used test console to create a new transcript.
  - Viewed some old transcripts.
  - Ran EXPLAIN on the SELECT and verified it was utilizing the garbageCollected
key.

Reviewed By: jungejason
Reviewers: jungejason, tuomaspelkonen, aran
CC: aran, epriestley, jungejason
Differential Revision: 735
2011-07-28 18:50:54 -07:00
epriestley
51c2726a34 Add Differential parse cache to the GC daemon
Summary:
Add the differential parse cache to the GC. This is the largest object in the
system by a wide margin, I think.

This table is potentially gigantic which is why the script truncates it before
doing a schema change.

Test Plan: Ran the GC daemon, it cleaned up some parse caches.
Reviewed By: jungejason
Reviewers: tuomaspelkonen, jungejason, aran
Commenters: tuomaspelkonen
CC: aran, jungejason, tuomaspelkonen, epriestley
Differential Revision: 620
2011-07-08 17:31:25 -07:00
epriestley
4ef918e213 Add a garbage collector daemon
Summary:
Phabricator generates a bunch of data that we don't need to keep around forever,
add a GC daemon to get rid of it with some basic configuration options.

This needs a couple more diffs to get some of the details but I think this is a
reasonable start.

I also fixed a couple of UI things related to this, e.g. the daemon logs page
going crazy when a daemon gets stuck in a loop and dumps tons of data to stdout.

Test Plan:
  - Ran gc daemon in 'phd debug' mode and saw it delete stuff, then sleep once
it had cleaned everything up.
  - Mucked around with TTLs and verified they work correctly.
  - Viewed gc'd transcripts in the web interface and made sure they displayed
okay.
  - Viewed daemon logs before/after garbage collection.
  - Running some run-at / run-for tests now, I'll update if the daemon doesn't
shut off in ~10-15 minutes. :P

Reviewed By: tuomaspelkonen
Reviewers: jungejason, tuomaspelkonen, aran
CC: aran, tuomaspelkonen, epriestley
Differential Revision: 583
2011-07-05 13:49:11 -07:00