mirror of
https://we.phorge.it/source/phorge.git
synced 2024-11-10 00:42:41 +01:00
No description
ebff07d019
Summary: Ref T4571. When a database goes down briefly, we fall back to replicas. However, this fallback is slow (not good for users) and keeps sending a lot of traffic to the master (might be bad if the root cause is load-related). Keep track of recent connections and fully degrade into "severed" mode if we see a sequence of failures over a reasonable period of time. In this mode, we send much less traffic to the master (faster for users; less load for the database). We do send a little bit of traffic still, and if the master recovers we'll recover back into normal mode seeing several connections in a row succeed. This is similar to what most load balancers do when pulling web servers in and out of pools. For now, the specific numbers are: - We do at most one health check every 3 seconds. - If 5 checks in a row fail or succeed, we sever or un-sever the database (so it takes about 15 seconds to switch modes). - If the database is currently marked unhealthy, we reduce timeouts and retries when connecting to it. Test Plan: - Configured a bad `master`. - Browsed around for a bit, initially saw "unrechable master" errors. - After about 15 seconds, saw "major interruption" errors instead. - Fixed the config for `master`. - Browsed around for a while longer. - After about 15 seconds, things recovered. - Used "Cluster Databases" console to keep an eye on health checks: it now shows how many recent health checks were good: {F1213397} Reviewers: chad Reviewed By: chad Maniphest Tasks: T4571 Differential Revision: https://secure.phabricator.com/D15677 |
||
|---|---|---|
| bin | ||
| conf | ||
| externals | ||
| resources | ||
| scripts | ||
| src | ||
| support | ||
| webroot | ||
| .arcconfig | ||
| .arclint | ||
| .arcunit | ||
| .editorconfig | ||
| .gitignore | ||
| LICENSE | ||
| NOTICE | ||
| README.md | ||
Phabricator is a collection of web applications which help software companies build better software.
Phabricator includes applications for:
- reviewing and auditing source code;
- hosting and browsing repositories;
- tracking bugs;
- managing projects;
- conversing with team members;
- assembling a party to venture forth;
- writing stuff down and reading it later;
- hiding stuff from coworkers; and
- also some other things.
You can learn more about the project (and find links to documentation and resources) at Phabricator.org
Phabricator is developed and maintained by Phacility.
SUPPORT RESOURCES
For resources on filing bugs, requesting features, reporting security issues, and getting other kinds of support, see Support Resources.
NO PULL REQUESTS!
We do not accept pull requests through GitHub. If you would like to contribute code, please read our Contributor's Guide.
LICENSE
Phabricator is released under the Apache 2.0 license except as otherwise noted.