1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-18 12:52:42 +01:00
Commit graph

120 commits

Author SHA1 Message Date
epriestley
19c6ccb279 Allow bin/aphlict to start without a valid database connection
Summary:
Ref T11818. See that task for a description.

This is like a tiny pinch of hackiness but not really unreasonable. Basically, `aphlict` is already an "uber-server" and one copy can handle a ton of instances.

Test Plan: Started `bin/aphlict` without a valid database connection.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11818

Differential Revision: https://secure.phabricator.com/D16854
2016-11-13 16:43:23 -08:00
epriestley
fc950140b4 Blanket reject request which may have been poisoned by a "Proxy" header to mitigate the httpoxy vulnerability
Summary:
See accompanying discussion in T11359.

As far as I can tell we aren't vulnerable, but subprocesses could be (now, or in the future). Reject any request which may have a `Proxy:` header.

This will also do a false-positive reject if `HTTP_PROXY` is defined in the environment, but this is likely a misconfiguration (cURL does not read it). I'll provide guidance on this.

Test Plan:
  - Made requests using `curl -H Proxy:...`, got rejected.
  - Made normal requests, got normal pages.

Reviewers: chad, avivey

Reviewed By: avivey

Differential Revision: https://secure.phabricator.com/D16318
2016-07-21 20:18:06 -07:00
epriestley
bbb321395a Support Aphlict clustering
Summary:
Ref T6915. This allows multiple notification servers to talk to each other:

  - Every server has a list of every other server, including itself.
  - Every server generates a unique fingerprint at startup, like "XjeHuPKPBKHUmXkB".
  - Every time a server gets a message, it marks it with its personal fingerprint, then sends it to every other server.
  - Servers do not retransmit messages that they've already seen (already marked with their fingerprint).
  - Servers learn other servers' fingerprints after they send them a message, and stop sending them messages they've already seen.

This is pretty crude, and the first message to a cluster will transmit N^2 times, but N is going to be like 3 or 4 in even the most extreme cases for a very long time.

The fingerprinting stops cycles, and stops servers from sending themselves copies of messages.

We don't need to do anything more sophisticated than this because it's fine if some notifications get lost when a server dies. Clients will reconnect after a short period of time and life will continue.

Test Plan:
  - Wrote two server configs.
  - Started two servers.
  - Told Phabricator about all four services.
  - Loaded Chrome and Safari.
  - Saw them connect to different servers.
  - Sent messages in one, got notifications in the other (magic!).
  - Saw the fingerprinting stuff work on the console, no infinite retransmission of messages, etc.

(This pretty much just worked when I ran it the first time so I probably missed something?)

{F1218835}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T6915

Differential Revision: https://secure.phabricator.com/D15711
2016-04-14 13:26:30 -07:00
epriestley
07fc8f17cc Support "ssl.chain" in Aphlict configuration
Summary: Fixes T10806. Although browsers don't seem to care about this, it's more correct to support it, and the new test console uses normal `cURL` and does care.

Test Plan:
  - Hit the error case for providing a chain but no key/cert.
  - Used `openssl s_client -connect localhost:22280` to connect to local Aphlict servers.
  - With SSL but no chain, saw `openssl` fail to verify the remote.
  - With SSL and a chain, saw `openssl` verify the identify of the remote.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10806

Differential Revision: https://secure.phabricator.com/D15709
2016-04-14 10:41:21 -07:00
epriestley
d4bf2a147b Make paths and Aphlict instance names less ambiguous
Summary:
Fixes T10783 (what little of it remains). Ref T10697.

Aphlict currently uses request paths for two different things:

  - multi-tenant instancing in the Phacility cluster (each instance gets its own namespace within an Aphlict server);
  - some users configure nginx and apache to do proxying or SSL termination based on the path.

Currently, these can collide.

Put a "~" before the instance name to make it unambiguous. At some point we can possibly just use a GET parameter, but I think there was some reason I didn't do that originally and this sequence of changes is disruptive enough already.

Test Plan: Saw local Aphlict unambiguously recognize "local.phacility.com" as instance "local", with a "~"-style URI.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10697, T10783

Differential Revision: https://secure.phabricator.com/D15705
2016-04-14 04:57:21 -07:00
epriestley
2930733ac9 Complete modernization of Aphlict configuration
Summary:
Fixes T10697. This finishes bringing the rest of the config up to cluster power levels.

Phabricator is now given an arbitrarily long list of notification servers.

Each Aphlict server is given an arbitrarily long list of ports to run services on.

Users are free to make them meet in the middle by proxying whatever they want to whatever else they want.

This should also accommodate clustering fairly easily in the future.

Also rewrote the status UI and changed a million other things. 🐗

Test Plan:
{F1217864}

{F1217865}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10697

Differential Revision: https://secure.phabricator.com/D15703
2016-04-14 04:57:00 -07:00
epriestley
c6b0925954 Move Aphlict logging and PID configuration options to config file
Summary: Ref T10697. Mostly straightforward. Also allow the server to have multiple logs and log options in the future (e.g., different verbosities or separate admin/client logs or whatever). No specific plans for this, but the default log is pretty noisy today.

Test Plan: Set up a couple of logs, started server, saw it log to them.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10697

Differential Revision: https://secure.phabricator.com/D15702
2016-04-14 04:55:19 -07:00
epriestley
c84dee522b Move server-related Aphlict options to a configuration file
Summary: Ref T10697. This isn't everything but starts generalizing options and moving us toward a cluster-ready state of affairs.

Test Plan: Started server in various configurations, hit most (all?) of the error cases with bad configs, sent test notifications.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10697

Differential Revision: https://secure.phabricator.com/D15701
2016-04-14 04:54:42 -07:00
epriestley
e32ce529d7 Begin generalizing Aphlict server to prepare for clustering/sensible config file
Summary:
Ref T10697. Currently, `aphlict` takes a ton of command line flags to configure exactly one admin server and exactly one client server.

I want to replace this with a config file. Additionally, I plan to support:

  - arbitrary numbers of listening client ports;
  - arbitrary numbers of listening admin ports;
  - SSL on any port.

For now, just transform the arguments to look like they're a config file. In the future, I'll load from a config file instead.

This greater generality will allow you to do stuff like run separate HTTP and HTTPS admin ports if you really want. I don't think there's a ton of use for this, but it tends to make the code cleaner anyway and there may be some weird cross-datacneter cases for it. Certainly, we undershot with the initial design and lots of users want to terminate SSL in nginx and run only HTTP on this server.

(Some sort-of-plausible use cases are running separate HTTP and HTTPS client servers, if your Phabricator install supports both, or running multiple HTTPS servers with different certificates if you have a bizarre VPN.)

Test Plan: Started Aphlict, connected to it, sent myself test notifications, viewed status page, reviewed logfile.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10697

Differential Revision: https://secure.phabricator.com/D15700
2016-04-14 04:54:20 -07:00
epriestley
2b02024e23 Use AphrontRequestStream to read request input
Summary:
Ref T10604. This uses the new standalone stream reader introduced in D15483 to read request data, instead of putting the logic in PhabricatorStartup.

It also doesn't read request data until it specifically needs to. This supports, e.g., streaming Git LFS PUT requests, and streaming more types of requests in the future.

Test Plan: See D15483. Made various different types of requests and wasn't immediately able to break anything.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10604

Differential Revision: https://secure.phabricator.com/D15484
2016-03-17 08:08:18 -07:00
epriestley
42566379dc Fix HTTP body decompression in PHP 5.6
Summary:
Ref T10264. Under PHP 5.6, you are no longer allowed to use `compress.zlib://php://input` as an argument to either `fopen()` or `file_get_contents()`.

Instead, open `php://input` as a file handle, then add `zlib.inflate` as a stream wrapper. This requires some level of magic to work properly.

Test Plan:
First, I constructed a synthetic gzipped payload by typing some words into a file and using `gzcompress()` to compress it.

Then I used a `curl` command like this to make requests with it:

```
$ curl -X POST -H "Content-Length: 66" -H "Content-Type: text/plain" -H "Content-Encoding: gzip" --data-binary @payload.deflate -v http://127.0.0.1/
```

I modified Phabricator to just dump the raw request body and exit, and reproduced the issue under PHP 5.6 (no body, error in log) by brining up a micro instance in EC2 and installing php56 on it.

After this patch, it dumped the body properly instead, and PHP 5.5 also continued worked properly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10264

Differential Revision: https://secure.phabricator.com/D15314
2016-02-20 14:55:05 -08:00
epriestley
f5c686d6a4 Swap charts from gRaphael to D3
Summary:
Mostly, this has just been sitting in my sandbox for a long time. I may also touch some charting stuff with subprojects/milestones, but don't have particular plans to do that.

D3 seems a bit more flexible, and it's easier to push more of the style logic into CSS so you can fix my design atrocities. gRaphael also hasn't been updated in ~3+ years.

Test Plan:
{F1085433}

{F1085434}

Reviewers: chad

Reviewed By: chad

Subscribers: cburroughs, yelirekim

Differential Revision: https://secure.phabricator.com/D15155
2016-02-01 10:36:59 -08:00
epriestley
8269fd6e6c Decode "Content-Encoding: gzip" content
Summary:
Fixes T10228. When we receive a gzipped request (rare, but `git` may send them), decode it before providing it to the application.

This fixes the issue with proxying certain requests described in T10228.

Test Plan:
  - Applied this fix in production.
  - Cloned a problem repository cleanly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10228

Differential Revision: https://secure.phabricator.com/D15145
2016-01-30 16:48:48 -08:00
epriestley
4b1815d6cc Add a "Startup" to DarkConsole
Summary: Ref T8588. It looks like something slow is happening //before// we start DarkConsole. Add some crude reporting to try to narrow it down.

Test Plan: {F743050}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8588

Differential Revision: https://secure.phabricator.com/D13956
2015-08-21 14:53:29 -07:00
Joshua Spence
368f359114 Use PhutilClassMapQuery instead of PhutilSymbolLoader
Summary: Use `PhutilClassMaQuery` instead of `PhutilSymbolLoader`, mostly for consistency. Depends on D13588.

Test Plan: Poked around a bunch of pages.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D13589
2015-08-14 07:49:01 +10:00
epriestley
a15444aa79 Remove PhabricatorStartup::getGlobal/setGlobal mechanism
Summary:
Ref T8424. Fixes T7114. This was envisioned as a per-request cache for reusing interpreters, but isn't a good fit for that in modern Phabricator.

In particular, it isn't loaded by the daemons, but they have equal need for per-request caching.

Since I finally need such a cache for Spaces, throw the old stuff away before I built a more modern cache.

Also resolves T7114 by dropping filtering on $_SERVER. I'm pretty sure this is the simplest fix, see D12977 for a bit more discussion.

Test Plan: Called `didFatal()` from somewhere in normal code and verified it was able to use the access log.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7114, T8424

Differential Revision: https://secure.phabricator.com/D13152
2015-06-04 17:26:52 -07:00
Joshua Spence
36e2d02d6e phtize all the things
Summary: `pht`ize a whole bunch of strings in rP.

Test Plan: Intense eyeballing.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: hach-que, Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D12797
2015-05-22 21:16:39 +10:00
Joshua Spence
acb45968d8 Use __CLASS__ instead of hard-coding class names
Summary: Use `__CLASS__` instead of hard-coding class names. Depends on D12605.

Test Plan: Eyeball it.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: hach-que, Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D12806
2015-05-14 07:21:13 +10:00
epriestley
52b9e5ec14 Filter potentially problematic $_ENV variables
Summary:
Caught this in the production error logs. We can end up with `argv` defined and set to an array in an nginx + php-fpm configuration.

When we later run `ExecFuture` subprocesses, they won't be able to forward the value.

The error this produces looks like this:

```
015/04/27 12:17:35 [error] 10948#0: *674 FastCGI sent in stderr: "PHP message: [2015-04-27 12:17:35] ERROR 8: Array to string conversion at [/core/lib/libphutil/src/future/exec/ExecFuture.php:667]
PHP message: arcanist(head=master, ref.master=805ae12408e8), phabricator(head=master, ref.master=8ce8a761efe9), phutil(head=master, ref.master=fccf03d48e08)
PHP message:   #0 ExecFuture::isReady() called at [<phutil>/src/future/Future.php:39]
PHP message:   #1 Future::resolve(NULL) called at [<phutil>/src/future/exec/ExecFuture.php:413]
PHP message:   #2 ExecFuture::resolvex() called at [<phabricator>/src/applications/diffusion/query/rawdiff/DiffusionGitRawDiffQuery.php:40]
PHP message:   #3 DiffusionGitRawDiffQuery::executeQuery() called at [<phabricator>/src/applications/diffusion/query/rawdiff/DiffusionRawDiffQuery.php:17]
PHP message:   #4 DiffusionRawDiffQuery::loadRawDiff() called at [<phabricator>/src/applications/diffusion/conduit/DiffusionRawDiffQueryConduitAPIMethod.php:56]
PHP message:   #5 DiffusionRawDiffQueryConduitAPIMethod::getResult(ConduitAPIRequest) called at [<phabricator>/src/applications/diffusion/conduit/DiffusionQueryConduitAPIMethod.php:135]
PHP message:   #6 DiffusionQueryConduitAPIMethod::execute(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/method/ConduitAPIMethod.php:90]
PHP message:   #7 ConduitAPIMethod::executeMethod(ConduitAPIRequest) called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:134]
PHP message:   #8 ConduitCall::executeMethod() called at [<phabricator>/src/applications/conduit/call/ConduitCall.php:84]
PHP message:   #9 ConduitCall::execute() called at [<phabricator>/src/applications/diffusion/query/DiffusionQuery.php:81]
PHP message:   #10 DiffusionQuery::callConduitWithDiffusionRequest(PhabricatorUser, DiffusionGitRequest, string, array) called at [<phabricator>/src/applications/diffusion/controller/DiffusionController.php:184]
PHP message:   #11 DiffusionController::callConduitWithDiffusionRequest(string, array) called at [<phabricat
```

Test Plan: I'm just going to push this to make sure it fixes things, since I can't repro it locally.

Reviewers: btrahan

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D12571
2015-04-27 05:21:15 -07:00
epriestley
ebcab8edb6 Namespace Aphlict clients by request path, plus other fixes
Summary:
Fixes T7130. Fixes T7041. Fixes T7012.

Major change here is partitioning clients. In the Phacility cluster, being able to get a huge pile of instances on a single server -- without needing to run a process per instance -- is desirable.

To accomplish this, just bucket clients by the path they connect with. This will let us set client URIs to `/instancename/` and then route connections to a small set of servers. This degrades cleanly in the common case and has no effect on installs which don't do instancing.

Also fix two unrelated issues:

  - Fix the timeouts, which were incorrectly initializing in `open()` (which is called during reconnect, causing them to reset every time). Instead, initialize in the constructor. Cap timeout at 5 minutes.
  - Probably fix subscriptions, which were using a property with an object definition. Since this is by-ref, all concrete instances of the object share the same property, so all users would be subscribed to everything. Probably.

Test Plan:
  - Hit notification status page, saw version bump and instance/path name.
  - Saw instance/path name in client and server logs.
  - Stopped server, saw reconnects after 2, 4, 16, ... seconds.
  - Sent test notification; received test notification.
  - Didn't explicitly test the subscription thing but it should be obvious by looking at `/notification/status/` shortly after a push.

Reviewers: joshuaspence, btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T7041, T7012, T7130

Differential Revision: https://secure.phabricator.com/D11769
2015-02-16 11:31:15 -08:00
Pierre Moreau
172566a769 Aphlict - fix incrementation of _messagesIn
Summary: Ref T7124. The local version of `this` in the handler of 'end' was incremented rather than the global one.

Test Plan: Sending test notifications did not increment the `messages.in` value before this patch even if it should have. With the patch sending test notifications does increment `messages.in`.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T7124

Differential Revision: https://secure.phabricator.com/D11648
2015-02-03 08:02:23 -08:00
Pierre Moreau
a424fec7bb Aphlict - fix getActiveListenerCount return value
Summary: Ref T7126. Dictionaries do not have a `length` property unlike arrays resulting in the `getActiveListenerCount()` function returning undefined results. Using the `length` property on the array of keys will work.

Test Plan: Using `wscat` to generate multiple connections to the server, establish new ones and close others while keeping an eye on the displayed `clients.active` value.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Maniphest Tasks: T7126

Differential Revision: https://secure.phabricator.com/D11647
2015-02-03 07:01:49 -08:00
Pierre Moreau
4af1fd2a79 Aphlict - remove listeners when clients close the connection
Summary: Ref T7110. Listeners are now removed when clients close the connection to avoid stacking a never ending number of unused listeners.

Test Plan: Using `wscat` to connect to the Aphlict server; when closing the connection a 'Diconnected.' will appear in the logs and the number of active listeners is decreased by one.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Maniphest Tasks: T7110

Differential Revision: https://secure.phabricator.com/D11634
2015-02-02 14:57:20 -08:00
Joshua Spence
04ee853cec Add the logger earlier in the Aphlict startup process
Summary: Add the logger as soon as possible so that the log file will contain errors if the `ws` module cannot be loaded.

Test Plan: Ran `./bin/aphlict debug` without having the `ws` module installed. Saw errors in the logs.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11457
2015-01-22 07:45:55 +11:00
Joshua Spence
f61846b469 Fix Aphlict exit status
Summary: Fixes T6998. Based on https://groups.google.com/d/msg/nodejs/zF7GEoPccqw/n26c6gPaluwJ.

Test Plan: Ran Aphlict and forced an error. Saw error messages in the logs and also saw a non-zero exit status.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T6998

Differential Revision: https://secure.phabricator.com/D11456
2015-01-22 06:46:49 +11:00
Joshua Spence
8113e6d8c1 Enable the freeze option for JSHint
Summary:
This option prohibits overwriting prototypes of native objects such as `Array`, `Date` and so on.

```lang=js
// jshint freeze:true
Array.prototype.count = function (value) { return 4; };
// -> Warning: Extending prototype of native object: 'Array'.
```

Test Plan: Linted existing JavaScript files, found no violations.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11439
2015-01-20 21:25:12 +11:00
Joshua Spence
27422ffe8e Use single quotes in JavaScript files
Summary: Use single quotes to keep JSHint happy.

Test Plan: `arc lint`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11431
2015-01-20 08:53:47 +11:00
Joshua Spence
53834d1471 Enable "strict" mode for NodeJS
Summary:
In particular, this changes the behavior of NodeJS in the following ways:

- Any attempt to get or modify the global object will result in an error.
- `null` values of `this` will no longer be evaluated to the global object and primitive values of this will not be converted to wrapper objects.
- Writing or deleting properties which have there writeable or configurable attributes set to false will now throw an error instead of failing silently.
- Adding a property to an object whose extensible attribute is false will also throw an error now.
- A functions arguments are not writeable so attempting to change them will now throw an error `arguments = [...]`.
- `with(){}` statements are gone.
- Use of `eval` is effectively banned.
- `eval` and `arguments` are not allowed as variable or function identifiers in any scope.
- The identifiers `implements`, `interface`, `let`, `package`, `private`, `protected`, `public`, `static` and `yield` are all now reserved for future use (roll on ES6).

Test Plan: Verified that Aphlict was still functional.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11430
2015-01-20 07:43:10 +11:00
epriestley
4636833f3d Fix module imports in Aphlict server
Summary:
This was broken in D11383. Basically, I had the `ws` module installed globally whilst testing, but the changes made do not work if the `ws` module is installed locally (i.e. in the `./support/aphlict/server/node_modules` directory). After poking around, it seems that this is due to the sandboxing that is done by `JX.require`.

A quick fix is to just //not// use `JX.require`, although you may have a better idea?

The error that is occurring is as follows:

```
<<< UNCAUGHT EXCEPTION! >>>

Error: Cannot find module 'ws'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at extra.require (/usr/src/phabricator/webroot/rsrc/externals/javelin/core/init_node.js:48:16)
    at /usr/src/phabricator/support/aphlict/server/lib/AphlictClientServer.js:10:17
    at Script.(anonymous function) [as runInNewContext] (vm.js:41:22)
    at Object.JX.require (/usr/src/phabricator/webroot/rsrc/externals/javelin/core/init_node.js:58:6)
    at Object.<anonymous> (/usr/src/phabricator/support/aphlict/server/aphlict_server.js:102:4)
    at Module._compile (module.js:456:26)
>>> Server exited!
```

Test Plan: Now able to start the Aphlict server.

Reviewers: joshuaspence

Reviewed By: joshuaspence

Subscribers: Korvin, epriestley

Maniphest Tasks: T6987

Differential Revision: https://secure.phabricator.com/D11425
2015-01-19 11:46:14 -08:00
Joshua Spence
2b12f61602 Don't exit from the Aphlict Server prematurely
Summary: By calling `process.exit(1)` we are forcing the Node.js process to exit prematurely. Specifically, the process is terminated before the error is written to the log file. This can be verified by inspecting the value of `debug._logs[0]._writableState.writing` immediately before the process is terminated.

Test Plan: Ran `./bin/aphlict debug` without the `ws` module being installed. Verified that `<<< UNCAUGHT EXCEPTION! >>>` was echoed to the console as well as the log file. Also verified that the exit status was non-zero.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11426
2015-01-20 06:36:51 +11:00
Joshua Spence
9a7ad972cd Refactoring of the Aphlict server
Summary: Tidy the Aphlict server by splitting the functionality into two main modules, `AphlictClientServer` and `AphlictAdminServer. There is still further tidying that could be done here, but I feel that this puts us in a much better place.

Test Plan: Sent notifications via `/notification/status/`.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11383
2015-01-19 07:49:50 +11:00
Joshua Spence
270a0c54b4 Fix permissions on Aphlict log
Summary: Currently, the Aphlict server created the log file (if it doesn't exist) but then immediately fails with "Unable to open logfile". It seems that we don't set the permissions correctly.

Test Plan: Deleted log file and was able to start the Aphlict server.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11351
2015-01-13 08:26:04 +11:00
epriestley
0d070c91dc Fix Aphlict logging
Summary:
Yeahhhhhhhh....

  - Open a "stream", not a "steam".
  - Make error easier for users to understand.
  - Write to the log in debug mode so the issue is more apparent.

Test Plan:
  - Started server with bad permissions, got usable error message.
  - Started server with good permissions, got logfile.

Reviewers: joshuaspence, btrahan

Reviewed By: btrahan

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11339
2015-01-12 08:16:08 -08:00
Joshua Spence
638cf20c9d Allow the Aphlict server to bind to localhost
Summary: If you are running the Aphlict server behind a reverse proxy (such as `nginx`) then there's no need to bind to `0.0.0.0`. Add a `--client-host` flag to `aphlict_server.js` to allow binding to a different hostname. Also changed the other flags for consistency and clarity.

Test Plan: Started, stopped and debug the Aphlict server.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11288
2015-01-09 11:10:47 +11:00
epriestley
f0ade1be1c Stop advising users to use npm -g to install websockets
Summary: Fixes T6910. This advice is bad, doesn't work, and was based on me havng an outdated or incorrect understanding of Node and npm.

Test Plan: Read documentation.

Reviewers: richardvanvelzen, btrahan, chad, joshuaspence

Reviewed By: chad, joshuaspence

Subscribers: epriestley

Maniphest Tasks: T6910

Differential Revision: https://secure.phabricator.com/D11285
2015-01-08 14:02:14 -08:00
Joshua Spence
a87f2cd610 Fix --ssl-cert for Aphlict
Summary: `PhabricatorAphlictManagementWorkflow` passes `--ssl-cert` but `aphlict_server.js` expects `--ssl-certificate`.

Test Plan: Tested on a production system.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11283
2015-01-09 08:51:47 +11:00
epriestley
9e0f70e17d Rewrite Aphlict to use Websockets
Summary:
Fixes T6559. No more flash, use Websockets. This is less aggressive than the earlier version, and retains more server logic.

  - Support "wss".
  - Make the client work.
  - Remove "notification.user" entirely.
  - Seems ok?

Test Plan:
In Safari, Firefox and Chrome, saw the browsers connect. Made a bunch of comments/updates and saw notifications.

Notable holes in the test plan:

  - Haven't tested "wss" yet. I'll do this on secure.
  - Notifications are //too fast// now, locally. I get them after I hit submit but before the page reloads.
  - There are probably some other rough edges, this is a fairly big patch.

Reviewers: joshuaspence, btrahan

Reviewed By: joshuaspence, btrahan

Subscribers: fabe, btrahan, epriestley

Maniphest Tasks: T6713, T6559

Differential Revision: https://secure.phabricator.com/D11143
2015-01-08 10:03:00 -08:00
Joshua Spence
87f11a091d Minor improvements to handling Aphlict status code
Summary: Self explanatory.

Test Plan: `curl`ed a few URLs.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11147
2015-01-03 09:11:08 +11:00
Joshua Spence
9f31e023f4 Minor improvements for handling of /status/ for Aphlict
Summary: We don't need to handle any request data for the `/status/` route, so we can simplify this code slightly.

Test Plan:
```lang=bash
> curl http://127.0.0.1:22281/status/
{"uptime":2543,"clients.active":0,"clients.total":0,"messages.in":0,"messages.out":0,"log":"/var/log/aphlict.log","version":6}
```

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11145
2015-01-03 09:10:49 +11:00
Joshua Spence
44198d68a7 Use the correct Content-Type for the Aphlict status route
Summary: The `/status/` route for the Aphlict server returns JSON.

Test Plan:
```lang=bash
> curl -I http://127.0.0.1:22281/status/
HTTP/1.1 200 OK
Content-Type: application/json
Date: Fri, 02 Jan 2015 13:08:34 GMT
Connection: keep-alive
```

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11144
2015-01-03 09:09:48 +11:00
epriestley
08126d3904 Improve top-level exception handling
Summary:
Fixes T6692. Addresses two main issues:

  - The write guard would sometimes not get disposed of on exception pathways, generating an unnecessary secondary error which was just a symptom of the original root error.
    - This was generally confusing and reduced the quality of reports we received because users would report the symptomatic error sometimes instead of the real error.
    - Instead, reflow the handling so that we always dispose of the write guard if we create one.
  - If we missed the Controller-level error page generation (normally, a nice page with full CSS, etc), we'd jump straight to Startup-level error page generation (very basic plain text).
    - A large class of errors occur too early or too late to be handled by Controller-level pages, but many of these errors are not fundamental, and the plain text page is excessively severe.
    - Provide a mid-level simple HTML error page for errors which can't get full CSS, but also aren't so fundamental that we have no recourse but plain text.

Test Plan:
Mid-level errors now produce an intentional-looking error page:

{F259885}

Verified that setup errors still render properly.

@chad, feel free to tweak the exception page -- I just did a rough pass on it. Like the setup error stuff, it doesn't have Celerity, so we can't use `{$colors}` and no other CSS will be loaded.

Reviewers: chad, btrahan

Reviewed By: btrahan

Subscribers: epriestley, chad

Maniphest Tasks: T6692

Differential Revision: https://secure.phabricator.com/D11126
2015-01-02 10:49:27 -08:00
Joshua Spence
af86a35f6b Allow JX to be redefined for NodeJS files
Summary: This silences a bunch of JSHint warnings.

Test Plan: `arc lint -- support/aphlict/server/lib/AphlictLog.js`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11084
2014-12-31 08:39:25 +11:00
Joshua Spence
a00efcbfca Fix some functions being used before they are defined
Summary: Minor JSHint fixes.

Test Plan: `arc lint`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11073
2014-12-30 22:09:53 +11:00
Joshua Spence
fcc7dbbf15 Fix a few minor JSHint warnings
Summary: Basically, don't assign expressions to a variable if the variable is unused.

Test Plan: `arc lint`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11072
2014-12-30 03:02:58 -08:00
Joshua Spence
b6e0c76c7d Define a seperate JSHint configuration for NodeJS files
Summary: Currently, we assume that all JavaScript files are for use in a browser. This is not true for the NodeJS Aphlict server code. Split the current JSHint configuration into `jshint-browser` and `jshint-node`.

Test Plan: `arc lint`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11071
2014-12-30 03:02:13 -08:00
Joshua Spence
db56342615 Declare Raphael as a global for JSHint
Summary: Let JSHint know that `Raphael` is a global that can be used anywhere. This is not technically correct, but it silences a few JSHint warnings. See http://jshint.com/docs/ for more information.

Test Plan: `arc lint -- webroot/rsrc/js/application/maniphest/behavior-line-chart.js`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D11069
2014-12-30 02:58:22 -08:00
Joshua Spence
3cf9a5820f Minor formatting changes
Summary: Apply some autofix linter rules.

Test Plan: `arc lint` and `arc unit`

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin, hach-que

Differential Revision: https://secure.phabricator.com/D10585
2014-10-08 08:39:49 +11:00
epriestley
cae59d8345 Add an option to make it easier to debug page hangs
Summary:
Fixes T6044. We've had two cases (both the same install, coincidentally) where pages got hung doing too much data fetching.

When pages hang, we don't get a useful stack trace out of them, since nginx, php-fpm, or PHP eventually terminates things in a non-useful way without any diagnostic information.

The second time (the recent Macros issue) I was able to walk the install through removing limits on nginx, php-fpm, php, and eventually getting a profile by letting the page run for several minutes until the request completed. However, this install is exceptionally technically proficient and this was still a big pain for everyone, and this approach would not have worked if the page actually looped rather than just taking a long time.

Provide `debug.time-limit`, which should give us a better tool for reacting to this situation: by setting it to a small value (like 10), we'll kill the page after 10 seconds with a trace, before nginx/php-fpm/php/etc can kill it uselessly. Hopefully that will be enough information to find the issue (generally, getting a trace has been 95% of the problem in the two cases we've encountered).

Test Plan: Set this option to `3` and added a sleep loop, saw a termination after 3 seconds with a useful trace.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: csilvers, joshuaspence, epriestley

Maniphest Tasks: T6044

Differential Revision: https://secure.phabricator.com/D10465
2014-09-11 06:28:21 -07:00
epriestley
8efea3abe9 Add a configuration warning when memory_limit will limit file uploads
Summary: Fixes T6011. See that task for discussion. We can detect when `memory_limit` will be the limiting factor for drag-and-drop uploads and warn administrators about it.

Test Plan: Fiddled configuration values and hit, then resolved, the issue.

Reviewers: btrahan

Reviewed By: btrahan

Subscribers: epriestley

Maniphest Tasks: T6011

Differential Revision: https://secure.phabricator.com/D10413
2014-09-04 12:48:34 -07:00
Joshua Spence
f2fee5a84e Return a HTTP 500 instead of a HTTP 400 if an internal error occurs in the Aphlict server
Summary: Ref T5651. Only throw a HTTP 400 if the data is invalid (i.e. the request is bad). If something bad happens when trying to transmit the notification, throw a HTTP 500 instead.

Test Plan: Eye-ball it.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: epriestley, Korvin

Maniphest Tasks: T5651

Differential Revision: https://secure.phabricator.com/D9968
2014-07-18 09:20:00 +10:00