1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-23 15:22:41 +01:00
No description
Find a file
epriestley ebff07d019 Automatically sever databases after prolonged unreachability
Summary:
Ref T4571. When a database goes down briefly, we fall back to replicas.

However, this fallback is slow (not good for users) and keeps sending a lot of traffic to the master (might be bad if the root cause is load-related).

Keep track of recent connections and fully degrade into "severed" mode if we see a sequence of failures over a reasonable period of time. In this mode, we send much less traffic to the master (faster for users; less load for the database).

We do send a little bit of traffic still, and if the master recovers we'll recover back into normal mode seeing several connections in a row succeed.

This is similar to what most load balancers do when pulling web servers in and out of pools.

For now, the specific numbers are:

  - We do at most one health check every 3 seconds.
  - If 5 checks in a row fail or succeed, we sever or un-sever the database (so it takes about 15 seconds to switch modes).
  - If the database is currently marked unhealthy, we reduce timeouts and retries when connecting to it.

Test Plan:
  - Configured a bad `master`.
  - Browsed around for a bit, initially saw "unrechable master" errors.
  - After about 15 seconds, saw "major interruption" errors instead.
  - Fixed the config for `master`.
  - Browsed around for a while longer.
  - After about 15 seconds, things recovered.
  - Used "Cluster Databases" console to keep an eye on health checks: it now shows how many recent health checks were good:

{F1213397}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4571

Differential Revision: https://secure.phabricator.com/D15677
2016-04-11 08:43:52 -07:00
bin Provide bin/nuance import and ngram indexes for sources 2016-03-08 10:30:24 -08:00
conf Mark some strings for translation 2015-06-09 23:06:52 +10:00
externals Improve Amazon SES code error handling behavior 2016-04-05 17:28:45 -07:00
resources When Phabricator is in read-only mode, explain why 2016-04-10 12:19:18 -07:00
scripts When cluster.databases is configured, read the master connection from it 2016-04-10 12:18:42 -07:00
src Automatically sever databases after prolonged unreachability 2016-04-11 08:43:52 -07:00
support Use AphrontRequestStream to read request input 2016-03-17 08:08:18 -07:00
webroot When Phabricator is in read-only mode, explain why 2016-04-10 12:19:18 -07:00
.arcconfig Use the configuration driven unit test engine 2015-08-11 07:57:11 +10:00
.arclint Begin adding test coverage to GitHub Events API parsers 2016-03-09 09:30:07 -08:00
.arcunit Use the configuration driven unit test engine 2015-08-11 07:57:11 +10:00
.editorconfig Fix text lint issues 2015-02-12 07:00:13 +11:00
.gitignore Add custom Cows and Figlet directories to .gitignore 2015-10-08 20:23:05 -07:00
LICENSE Fix text lint issues 2015-02-12 07:00:13 +11:00
NOTICE Update Phabricator NOTICE file to reflect modern legal circumstances 2014-06-25 13:42:13 -07:00
README.md Remove push to IRC from "readme.md" too 2015-10-24 18:39:16 -07:00

Phabricator is a collection of web applications which help software companies build better software.

Phabricator includes applications for:

  • reviewing and auditing source code;
  • hosting and browsing repositories;
  • tracking bugs;
  • managing projects;
  • conversing with team members;
  • assembling a party to venture forth;
  • writing stuff down and reading it later;
  • hiding stuff from coworkers; and
  • also some other things.

You can learn more about the project (and find links to documentation and resources) at Phabricator.org

Phabricator is developed and maintained by Phacility.


SUPPORT RESOURCES

For resources on filing bugs, requesting features, reporting security issues, and getting other kinds of support, see Support Resources.

NO PULL REQUESTS!

We do not accept pull requests through GitHub. If you would like to contribute code, please read our Contributor's Guide.

LICENSE

Phabricator is released under the Apache 2.0 license except as otherwise noted.