1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-09-19 16:58:48 +02:00
Commit graph

486 commits

Author SHA1 Message Date
epriestley
44f0664d2c Add a "lock log" for debugging where locks are being held
Summary: Depends on D19173. Ref T13096. Adds an optional, disabled-by-default lock log to make it easier to figure out what is acquiring and holding locks.

Test Plan: Ran `bin/lock log --enable`, `--disable`, `--name`, etc. Saw sensible-looking output with log enabled and daemons restarted. Saw no additional output with log disabled and daemons restarted.

Maniphest Tasks: T13096

Differential Revision: https://secure.phabricator.com/D19174
2018-03-05 17:55:34 -08:00
epriestley
07028cfc30 When bin/drydock lease is interrupted, release leases
Summary:
Depends on D19072. Ref T13073. Currently, you can leave leases stranded by using `^C` to interrupt the script. Handle signals and release leases on destruction if they haven't activated yet.

Also, print out more useful information before and after activation.

Test Plan: Mashed ^C while runnning `bin/drydock lease ... --trace`, saw the lease release.

Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Maniphest Tasks: T13073

Differential Revision: https://secure.phabricator.com/D19073
2018-02-13 13:14:21 -08:00
epriestley
0470125d9e Add skeleton code for webhooks
Summary: Ref T11330. Adds general support for webhooks. This is still rough and missing a lot of pieces -- and not yet useful for anything -- but can make HTTP requests.

Test Plan: Used `bin/webhook call ...` to complete requests to a test endpoint.

Maniphest Tasks: T11330

Differential Revision: https://secure.phabricator.com/D19045
2018-02-09 13:55:04 -08:00
epriestley
956c4058e6 Add a bin/conduit call support binary
Summary:
Ref T13060. See PHI343. Triaging this bug required figuring out where in the pipeline UTF8 was being dropped, and bisecting the pipeline required making calls to Conduit.

Currently, there's no easy way to debug/inspect arbitrary Conduit calls, especially when they are `diffusion.*` calls which route to a different host (even if you have a real session and use the web console for these, you just see an HTTP service call to the target host in DarkConsole).

Add a `bin/conduit` utility to make this kind of debugging easier, with an eye toward the Phacility production cluster (or other similar clusters) specifically.

Test Plan:
  - Ran `echo '{}' | bin/conduit call --method conduit.ping --input -` and similar.
  - Used a similar approach to successfully diagnose the UTF8 issue in T13060.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13060

Differential Revision: https://secure.phabricator.com/D18987
2018-02-05 12:20:49 -08:00
epriestley
2914613444 Fix failure to record pullerPHID in repository pull logs
Summary:
See PHI305. Ref T13046.

The SSH workflows currently extend `PhabricatorManagementWorkflow` to benefit from sharing all the standard argument parsing code. Sharing the parsing code is good, but it also means they inherit a `getViewer()` method which returns the ommnipotent viewer.

This is appropriate for everything else which extends `ManagementWorkflow` (like `bin/storage`, `bin/auth`, etc.) but not appropriate for SSH workflows, which have a real user.

This caused a bug with the pull logs where `pullerPHID` was not recorded properly. We used `$this->getViewer()->getPHID()` but the correct code was `$this->getUser()->getPHID()`.

To harden this against future mistakes:

  - Don't extend `ManagementWorkflow`. Extend `PhutilArgumentWorkflow` instead. We **only** want the argument parsing code.
  - Rename `get/setUser()` to `get/setSSHUser()` to make them explicit.

Then, fix the pull log bug by calling `getSSHUser()` instead of `getViewer()`.

Test Plan:
  - Pulled and pushed to a repository over SSH.
  - Grepped all the SSH stuff for the altered symbols.
  -  Saw pulls record a valid `pullerPHID` in the pull log.
  - Used `echo {} | ssh ... conduit conduit.ping` to test conduit over SSH.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13046

Differential Revision: https://secure.phabricator.com/D18912
2018-01-23 14:09:42 -08:00
epriestley
aa3b582c7b Remove "set password" from bin/accountadmin and let bin/auth recover recover anyone
Summary:
Ref T13043. This cleans some things up to prepare for moving account passwords to shared infrastructure.

Currently, the (very old, fairly unusual) `bin/accountadmin` tool can set account passwords. This is a bit weird, generally not great, and makes upgrading to shared infrastructure more difficult. Just get rid of this to simplify things. Many installs don't have passwords and this is pointless and unhelpful in those cases.

Instead, let `bin/auth recover` recover any account, not just administrator accounts. This was a guardrail against administrative abuse, but it has always seemed especially flimsy (since anyone who can run the tool can easily comment out the checks) and I use this tool in cluster support with some frequency, occasionally just commenting out the checks. This is generally a better solution than actually setting a password on accounts anyway. Just get rid of the check and give users enough rope to shoot themselves in the foot with if they truly desire.

Test Plan:
  - Ran `bin/accountadmin`, didn't get prompted to swap passwords anymore.
  - Ran `bin/auth recover` to recover a non-admin account.

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13043

Differential Revision: https://secure.phabricator.com/D18901
2018-01-23 10:58:11 -08:00
epriestley
3038d564a6 Allow bulk edits to be made silently if you have CLI access
Summary:
Fixes T13042. This hooks up the new "silent" mode from D18882 and makes it actually work.

The UI (where we tell you to go run some command and then reload the page) is pretty clumsy, but should solve some problems for now and can be cleaned up eventually. The actual mechanics (timeline aggregation, Herald interaction,  etc.) are on firmer ground.

Test Plan:
  - Made a normal bulk edit, got mail and feed stories.
  - Made a silent bulk edit, no mail and no feed.
  - Saw "Silent Edit" marker in timeline for silent edits:

{F5386245}

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T13042

Differential Revision: https://secure.phabricator.com/D18883
2018-01-19 13:24:54 -08:00
epriestley
1cb0d41367 Fix inbound mail handling for messages with no HTML body part
Summary:
See D18776. See <https://discourse.phabricator-community.org/t/cant-create-maniphest-tasks-by-email/754/2>.

The change in D18776 to improve handling of non-utf8 HTML parts broke handling of mail with //no// HTML parts. Partly, this is because MimeMailParser has a "traditional" PHP-style API where the return type is an exciting surprise.

Test Plan:
  - Sent a text-only message in `Mail.app`.
  - Used "Show Raw" to copy it to `mail.txt`, verifying that the raw message contains ONLY a text body.
  - Ran `cat mail.txt | ./scripts/mail/mail_handler.php --trace --process-duplicates`.
    - Before patch: error about bad `idx()` on a non-array.
    - After patch: clean mail processing.
  - Did the same with a message with both HTML and text bodies to make sure I didn't break anything.

Ideally we'd probably get test coverage on this, but it's been touched roughly once a year since 2013 so it'll probably hold.

Reviewers: amckinley, alexmv

Reviewed By: amckinley, alexmv

Differential Revision: https://secure.phabricator.com/D18778
2017-11-22 11:14:24 -08:00
Alex Vandiver
d2cff6a2cf Transcode the HTML part of incoming email into UTF-8 as well
Summary:
D1093 did this for just the text/plain part of incoming
email.  Most text/html parts choose to either use entity encoding
//or// are already UTF-8, thus obviating the need to transcode the
HTML part.  However, this is not always the case, and leads to dropped
messages, by way of:

```
EXCEPTION: (Exception) Failed to JSON encode value (#5: Malformed UTF-8 characters, possibly incorrectly encoded): Dictionary value at key "html" is not valid UTF8, and cannot be JSON encoded: [snip HTML part of message content]```

Generalize the charset transcoding to not apply to just the text/plain part, but
both text/plain and text/html parts.

Test Plan:
Fed in a Windows-1252-encoded text/html part with 0x92
bytes in it; verified that $content only contained valid UTF-8 after
this change.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Differential Revision: https://secure.phabricator.com/D18776
2017-11-16 10:08:39 -08:00
Dmitri Iouchtchenko
9bd6a37055 Fix spelling
Summary: Noticed a couple of typos in the docs, and then things got out of hand.

Test Plan:
  - Stared at the words until my eyes watered and the letters began to swim on the screen.
  - Consulted a dictionary.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam

Differential Revision: https://secure.phabricator.com/D18693
2017-10-09 10:48:04 -07:00
epriestley
03423c632c Treat commit hook execution in observed repositories as a no-op, not an error
Summary:
See PHI24. If you create a hosted Mercurial repository and switch it to observed, you can end up with a hook installed that runs on pulls and complains.

Instead, just bail out if we're running on a pull.

The corresponding Git hook doesn't run on pulls, so there's no issue in Git.

Test Plan: Executed the hook in an observed Mercurial repository, got a clean exit.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D18307
2017-08-01 08:32:42 -07:00
Chad Little
3a868940c7 Add a profileimage generation workflow for the cli
Summary: Ref T10319. This adds a basic means of generating default profile images for users. You can generate them for everyone, a group of users, or force updates. This only generated images and stores them in files. It does not assign them to users.

Test Plan:
`bin/people profileimage --all` to generate all images.
`bin/people profileimage --users chad` to generate a user.
`bin/people profileimage --all --force` to force rebuilding all images.

{F3662810}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T10319

Differential Revision: https://secure.phabricator.com/D17464
2017-03-04 15:43:13 -08:00
Chad Little
01b35cdc12 Add some sort of sort to Emoji Autocomplete
Summary: Ref T12139. Adds sorting by shortname. Also I sorted everything else. No reason. It didn't help

Test Plan: `:star`

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12139

Differential Revision: https://secure.phabricator.com/D17246
2017-01-24 20:21:06 -08:00
Chad Little
f930fd2e00 Add an Emoji Typeahead
Summary:
This adds a more complete emoji datasource, with a typeahead and autocomplete. It works by pulling in a raw datasource from EmojiOne (I chose Unicode 8, but they have a Unicode 9 datasource as well) and transforming it for speed/need. If we build more robustness or an actual picker into the Remarkup bar, having the additional keywords, etc, might be important. When Unicode 9 support is more prevalent, we should only need to update the single file.

 Tossing up as a proof of concept on engineering direction. Also I can't quite get the autocomplete to complete.

Test Plan: Test UIExamples, Autocomplete, and TypeaheadSource

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12139

Differential Revision: https://secure.phabricator.com/D17244
2017-01-24 13:13:10 -08:00
epriestley
e4f49f0806 When available, use async_signals in Phabricator
Summary: Ref T9640. See D17200 for the analogous change in libphutil.

Test Plan: See D17200.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9640

Differential Revision: https://secure.phabricator.com/D17201
2017-01-12 16:00:13 -08:00
epriestley
1e9a462baa Remove most of the legacy hunk code
Summary: Ref T8475. This gets rid of most of the old "legacy hunk" code. I'll nuke the rest (and drop the old table) once we're more sure that we're in the clear.

Test Plan: Browsed Differential.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8475

Differential Revision: https://secure.phabricator.com/D17040
2016-12-13 14:34:36 -08:00
epriestley
2d7abfd9fa Use HTTPS, not HTTP, in install scripts
Summary:
Via HackerOne. A researcher correctly reports that our install scripts use `HTTP`, not `HTTPS`, to fetch resources and execute them as `root`, which is a potentially significant vulnerability.

Instead, use `HTTPS`.

Test Plan: Verified that these URIs function correctly over `HTTPS`.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16958
2016-11-29 12:11:04 -08:00
epriestley
8fbb411a9a Respect user/pass flags to bin/storage for direct DatabaseRef-based queries
Summary:
Ref T11893. Previously, we excuted all `bin/storage` queries through `StorageManagementAPI` objects.

After D16848, we execute some queries through `PhabricatorDatabaseRef`. However, the refs we use weren't getting passed the `-u` / `-p` flags correctly, for specifying alternate administrative credentials.

Test Plan:
  - Created a second account (`trunk`).
  - Ran `bin/storage -u trunk`.
  - Made libphutil throw when not connecting as that user.
  - Before patch: some connections incorrectly used the default user, "root".
  - After patch: all connections correctly used the configured user, "trunk".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11893

Differential Revision: https://secure.phabricator.com/D16901
2016-11-19 08:39:45 -08:00
epriestley
19c6ccb279 Allow bin/aphlict to start without a valid database connection
Summary:
Ref T11818. See that task for a description.

This is like a tiny pinch of hackiness but not really unreasonable. Basically, `aphlict` is already an "uber-server" and one copy can handle a ton of instances.

Test Plan: Started `bin/aphlict` without a valid database connection.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11818

Differential Revision: https://secure.phabricator.com/D16854
2016-11-13 16:43:23 -08:00
epriestley
558d194302 Update bin/storage workflows to accommodate multiple masters
Summary: Depends on D16847. Ref T11044. This updates the remaining storage-related workflows from the CLI to accommodate multiple masters.

Test Plan:
  - Configured multiple masters.
  - Ran all `bin/storage` workflows.
  - Ran `arc unit --everything`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11044

Differential Revision: https://secure.phabricator.com/D16848
2016-11-12 16:37:47 -08:00
epriestley
ecc598f18d Support multiple database masters and convert easy callers
Summary:
Ref T11044. This moves toward partitioned application databases:

  - You can define multiple masters.
  - Convert all the easily-convertible code to become multi-master aware.

This doesn't convert most of `bin/storage` or "Config > Database (Stuff)" yet, as both are quite involved. They still work for now, but only operate on the first master instead of all masters.

Test Plan: Configured multiple masters, browsed around, ran `bin/storage` commands, ran `bin/storage --host ...`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11044

Differential Revision: https://secure.phabricator.com/D16115
2016-11-12 16:30:20 -08:00
epriestley
6982bded71 Remove ancient "Holiday" storage
Summary:
Ref T11809. This came out of Facebook many years ago for computing the number of business days that revisions had been stale.

We removed the little staleness marker a few months ago and haven't seen complaints about it.

If we did holidays now it would make sense to integrate them more directly with Calendar as real events, but I have no plans to pursue this anytime soon. It's easy enough to add the federal holidays manually (~5 minutes of work per year?) if you want them, and they're commentable/editable and you can add local holidays if you're not in the US.

Test Plan:
  - Ran `bin/storage upgrade -f`.
  - Grepped for `CalendarHoliday`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11809

Differential Revision: https://secure.phabricator.com/D16788
2016-11-01 16:03:44 -07:00
epriestley
6e6ae36dcf Add a skeleton for Calendar notifications
Summary:
Ref T7931. I'm going to do this separate from existing infrastructure because:

  - events start at different times for different users;
  - I like the idea of being able to batch stuff (send one email about several upcoming events);
  - triggering on ghost/recurring events is a real complicated mess.

This puts a skeleton in place that finds all the events we need to notify about and writes some silly example bodies to stdout, marking that we notified users so they don't get notified again.

Test Plan:
Ran `bin/calendar notify`, got a "great" notification in the command output.

{F1891625}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7931

Differential Revision: https://secure.phabricator.com/D16783
2016-11-01 10:41:15 -07:00
epriestley
c21a71f024 Cache generation of the SSH authentication keyfile for sshd
Summary:
Ref T11469. This isn't directly related, but has been on my radar for a while: building SSH keyfiles (particular for installs with a lot of keys, like ours) can be fairly slow.

At least one cluster instance is making multiple clone requests per second. While that should probably be rate limited separately, caching this should mitigate the impact of these requests.

This is pretty straightforward to cache since it's exactly the same every time, and only changes when users modify SSH keys (which is rare).

Test Plan:
  - Ran `bin/auth-ssh`, saw authfile generate.
  - Ran it again, saw it read from cache.
  - Changed an SSH key.
  - Ran it again, saw it regenerate.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11469

Differential Revision: https://secure.phabricator.com/D16744
2016-10-21 07:29:40 -07:00
epriestley
0e8ceeb690 Continue on bad database configuration from select scripts
Summary: Ref T11589. Provide a way for scripts to say "just continue if database config fails", and use it in `bin/config` and `bin/storage`.

Test Plan:
  - Broke database config.
  - Ran `bin/config`, worked fine.
  - Ran `bin/storage`, got helpful "set up the database" message.
  - Ran `bin/repository`, got fatal.
  - Ran normal site with valid/invalid config, got proper feedback.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11589

Differential Revision: https://secure.phabricator.com/D16502
2016-09-06 14:20:57 -07:00
Chad Little
52c0ec2700 Update Phabricator logo
Summary:
This updates the eye logo and removes the formal wordmark "Phabricator" as an image. Instead we'll use the new updated eye logo and plain text for "Phabricator", both of which are more friendly and less industrial.

Installs that already use the `header-logo` customization setting will need to rebuild their logo to 80px x 80px. They will then also get to use plain text to whitebox their install as they see fit.

Test Plan:
Tested new logo at desktop, tablet, and mobile sizes. Set a random instance name, saw new wordmark. Created a really long wordmark of MMMMMMMMMMMM, saw text cut off so UI doesn't break. May need some additional tweaking, but I think we covered the most edge cases here.

{F1751791, size=full}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: edibiase, bjshively, yelirekim, Korvin

Maniphest Tasks: T4214, T11096

Differential Revision: https://secure.phabricator.com/D16373
2016-08-07 11:35:21 -07:00
epriestley
89f9f97159 Provide basic support for Subversion revprops
Summary:
Ref T11208. See that task for a more detailed description of revprops.

This allows revprop changes in a hosted Subversion repository if the repository has the "allow dangerous changes" flag set.

In the future, we could expand this into real Herald support, but the only use case we have for now is letting `svnsync` work.

Test Plan:
Edited revprops with `svn propset --revprop -r 2 propkey propvalue repositoryuri`:

  - Tried before patch, got a "configure a commit hook" error.
  - Tried after patch, got a "dangerous change" error.
  - Allowed dangerous changes.
  - Did a revprop edit.
  - Prevented dangerous changes.
  - Got an error again.
  - Made a normal commit to an SVN repository.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11208

Differential Revision: https://secure.phabricator.com/D16174
2016-06-24 13:43:32 -07:00
epriestley
70463e8a16 Handle Subversion SSH on nonstandard ports
Summary:
Fixes T11203. Subversion handling of `SVN_SSH` commands requires some additional finesse for nonstandard remote SSH ports.

We get `domain.com:port` in the command, so parse it out if it's present.

Test Plan: @enckse confirmed this locally in T11203.

Reviewers: chad

Reviewed By: chad

Subscribers: enckse

Maniphest Tasks: T11203

Differential Revision: https://secure.phabricator.com/D16172
2016-06-23 07:40:36 -07:00
epriestley
72588d2eaa Allow device-to-device SSH to establish SSH sessions
Summary:
In D16167 I required users to be logged in to be "activated", but this logic doesn't account for that properly when the user is an Almanac device (a cluster host connecting to another cluster host).

Don't do this check for device connections, they can always establish sessions.

Test Plan: Will push.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16170
2016-06-22 13:43:49 -07:00
epriestley
f56f1b05c0 Install a SIGTERM handler in ssh-connect
Summary:
Ref T10547. This has been around for a while but I was never able to reproduce it. I caught a repro case in the cluster recently and I think this is the right fix.

We tell Subversion to run `ssh-connect` instead of `ssh` so we can provide options and credentials, by using `SVN_SSH` in the environment. Subversion will sometimes kill the SSH tunnel subprocess aggressively with SIGTERM -- as of writing, you can search for `SIGTERM` in `make_tunnel()` here:

http://svn.apache.org/repos/asf/subversion/trunk/subversion/libsvn_ra_svn/client.c

By default, when a PHP process gets SIGTERM it just exits immediately, without running destructors or shutdown functions. Since destructors/shutdown functions don't run, `TempFile` doesn't get a chance to remove the file.

I don't have a clear picture of //when// Subversion sends SIGTERM to the child process. I can't really get this to trigger locally via `svn`, although I was able to get it to trigger explicitly. So I'm only about 95% sure this fixes it, but it seems likely.

Test Plan:
Locally, I couldn't get this to reproduce "normally" even knowing the cause (maybe Subversion doesn't do the SIGTERM stuff on OSX?) but I was able to get it to reproduce reliabily by adding `posix_kill(getmypid(), SIGTERM);` to the body of the script.

With that added, running the script with `PHABRICATOR_CREDENTIAL=PHID-CDTL-...` in the environment reliably left straggler temporary files.

Adding `declare()` and a signal handler fixed this: the script now runs the `TempFile` destructor and longer leaves the stragglers around.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10547

Differential Revision: https://secure.phabricator.com/D16102
2016-06-13 10:05:46 -07:00
epriestley
814fa135b0 Centralize "this is the current user for the request" code
Summary:
Ref T11098. This primarily fixes Conduit calls to `*.edit` methods failing when trying to access user preferences.

(The actual access is a little weird, since it seems like we're building some UI stuff inside a policy query, but that's an issue for another time.)

To fix this, consolidate the "we're about to run some kind of request with this user" code and run it consistently for web, conduit, and SSH sessions.

Additionally, make sure we swap things to the user's translation.

Test Plan:
  - Ran `maniphest.edit` via `arc call-conduit`, no more settings exception.
  - Set translation to ALL CAPS, got all caps output from `ssh` and Conduit.

Reviewers: avivey, chad

Reviewed By: chad

Maniphest Tasks: T11098

Differential Revision: https://secure.phabricator.com/D16066
2016-06-07 07:43:50 -07:00
epriestley
0bf337b06c Lift ticks declaration to top level in Phabricator daemons
Summary: Ref T10811. This is a companion change for D16053, but affects the Phabricator version of this script.

Test Plan: Started daemons, ^C'd them, saw them handle the signal.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10811

Differential Revision: https://secure.phabricator.com/D16054
2016-06-06 06:39:09 -07:00
epriestley
0308d580d7 Deactivate SSH keys instead of destroying them completely
Summary:
Ref T10917. Currently, when you delete an SSH key, we really truly delete it forever.

This isn't very consistent with other applications, but we built this stuff a long time ago before we were as rigorous about retaining data and making it auditable.

In partiular, destroying data isn't good for auditing after security issues, since it means we can't show you logs of any changes an attacker might have made to your keys.

To prepare to improve this, stop destoying data. This will allow later changes to become transaction-oriented and show normal transaction logs.

The tricky part here is that we have a `UNIQUE KEY` on the public key part of the key.

Instead, I changed this to `UNIQUE (key, isActive)`, where `isActive` is a nullable boolean column. This works because MySQL does not enforce "unique" if part of the key is `NULL`.

So you can't have two rows with `("A", 1)`, but you can have as many rows as you want with `("A", null)`. This lets us keep the "each key may only be active for one user/object" rule without requiring us to delete any data.

Test Plan:
- Ran schema changes.
- Viewed public keys.
- Tried to add a duplicate key, got rejected (already associated with another object).
- Deleted SSH key.
- Verified that the key was no longer actually deleted from the database, just marked inactive (in future changes, I'll update the UI to be more clear about this).
- Uploaded a new copy of the same public key, worked fine (no duplicate key rejection).
- Tried to upload yet another copy, got rejected.
- Generated a new keypair.
- Tried to upload a duplicate to an Almanac device, got rejected.
- Generated a new pair for a device.
- Trusted a device key.
- Untrusted a device key.
- "Deleted" a device key.
- Tried to trust a deleted device key, got "inactive" message.
- Ran `bin/ssh-auth`, got good output with unique keys.
- Ran `cat ~/.ssh/id_rsa.pub | ./bin/ssh-auth-key`, got good output with one key.
- Used `auth.querypublickeys` Conduit method to query keys, got good active keys.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10917

Differential Revision: https://secure.phabricator.com/D15943
2016-05-18 14:54:28 -07:00
epriestley
c70f4815a9 Allow cluster devices to SSH to one another without acting as a user
Summary:
Ref T4292. When you run `git fetch` and connect to, say, `repo001.west.company.com`, we'll look at the current version of the repository in other nodes in the cluster.

If `repo002.east.company.com` has a newer version of the repository, we'll fetch that version first, then respond to your request.

To do this, we need to run `git fetch repo002.east.company.com ...` and have that connect to the other host and be able to fetch data.

This change allows us to run `PHABRICATOR_AS_DEVICE=1 git fetch ...` to use device credentials to do this fetch. (Device credentials are already supported and used, they just always connect as a user right now, but these fetches should be doable without having a user. We will have a valid user when you run `git fetch` yourself, but we won't have one if the daemons notice that a repository is out of date and want to update it, so the update code should not depend on having a user.)

Test Plan:
```
$ PHABRICATOR_AS_DEVICE=1 ./bin/ssh-connect local.phacility.com
Warning: Permanently added 'local.phacility.com' (RSA) to the list of known hosts.
PTY allocation request failed on channel 0
phabricator-ssh-exec: Welcome to Phabricator.

You are logged in as device/daemon.phacility.net.

You haven't specified a command to run. This means you're requesting an interactive shell, but Phabricator does not provide an interactive shell over SSH.

Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH.

Supported commands are: conduit, git-lfs-authenticate, git-receive-pack, git-upload-pack, hg, svnserve.
Connection to local.phacility.com closed.
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4292

Differential Revision: https://secure.phabricator.com/D15755
2016-04-19 13:04:41 -07:00
epriestley
5a0b7398ca Give bin/storage some replica-aware options
Summary:
Fixes T10758.

  - Adds a "--host" flag. If you specify this, we read your cluster config. This lets you dump from a replica.
  - Adds a "--for-replica" flag to `storage dump`. This makes `mysqldump` include a `CHANGE MASTER ...` statement in the output, which is useful when setting up a replica for the first time.

Test Plan:
  - Dumped master and replica cluster databases.
  - Dumped non-cluster databases.
  - Ran various other commands (help, status, etc).

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10758

Differential Revision: https://secure.phabricator.com/D15714
2016-04-14 13:23:35 -07:00
epriestley
6a4a9bb2d2 When cluster.databases is configured, read the master connection from it
Summary:
Ref T4571. Ref T10759. Ref T10758. This isn't complete, but gets most of the job done:

  - When `cluster.databases` is set up, most things ignore `mysql.host` now.
  - You can `bin/storage upgrade` and stuff works.
  - You can browse around in the web UI and stuff works.

There's still a lot of weird tricky stuff to navigate, and this has real no advantages over configuring a single server yet (no automatic failover, etc).

Test Plan:
  - Configured `cluster.databases` to point at my `t1.micro` hosts in EC2 (master + replica).
  - Ran `bin/storage upgrade`, got a new install setup on them properly.
  - Survived setup warnings, browsed around.
  - Switched back to local config, ran `bin/storage upgrade`, browsed around, went through setup checks.
  - Intentionally broke config (bad hosts, no masters) and things seemed to react reasonably well.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4571, T10758, T10759

Differential Revision: https://secure.phabricator.com/D15668
2016-04-10 12:18:42 -07:00
epriestley
2a3c3b2b98 Provide bin/nuance import and ngram indexes for sources
Summary:
Ref T10537. More infrastructure:

  - Put a `bin/nuance` in place with `bin/nuance import`. This has no useful behavior yet.
  - Allow sources to be searched by substring. This supports `bin/nuance import --source whatever` so you don't have to dig up PHIDs.

Test Plan:
  - Applied migrations.
  - Ran `bin/nuance import --source ...` (no meaningful effect, but works fine).
  - Searched for sources by substring in the UI.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10537

Differential Revision: https://secure.phabricator.com/D15436
2016-03-08 10:30:24 -08:00
epriestley
78fba426f6 Update import/clear symbols scripts for callsigns
Summary: Ref T4245. Accept identifiers instead of callsigns in these scripts so things continue to work in a future callsign-optional world.

Test Plan: Ran these scripts with both valid and invalid arguments; saw success and errors.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D15300
2016-02-18 09:55:57 -08:00
epriestley
973b8ace86 Remove dependence on callsigns from bin/commit-hook
Summary:
Ref T4245. Two effects:

  - First, let hooks work for future repositories without callsigns.
  - Second, provide a better error when users push directly to hosted repositories.

Test Plan: Ran `bin/commit-hook PHID-REPO-xxx`.

Reviewers: chad, avivey

Reviewed By: avivey

Maniphest Tasks: T4245

Differential Revision: https://secure.phabricator.com/D15293
2016-02-17 16:50:36 -08:00
epriestley
3f920d8904 Fix method call on possible null
Summary: Fixes T10317. If we failed to load `$device`, it will be null, so we won't be able to call `getName()` on it.

Test Plan: `SSH_CLIENT='127.0.0.1' ./bin/ssh-exec  --phabricator-ssh-device xyz` no longer fatals

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10317

Differential Revision: https://secure.phabricator.com/D15235
2016-02-10 12:44:52 -08:00
Chad Little
f7646b40aa Remove defunt project image choices
Summary: These are old project image choices, remove and only go with FontAwesome related images.

Test Plan: Project -> Edit Picture -> Save

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D15051
2016-01-19 08:50:59 -08:00
Chad Little
550793f9a4 Remove header gradients for flat colors
Summary: Removes header gradient images for flat, CSS controlled colors. I didn't convert the "pony" colors over, going with few options for easier theme-ability.

Test Plan:
Test each color choice.

{F1063828}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D15052
2016-01-19 07:17:02 -08:00
epriestley
0dd947cced Move diff extraction from commits to a separate test with a CLI command
Summary:
Ref T9319. When we discover a commit, we sometimes update the corresponding revision with a "this is the actual committed change" diff and send out a link to the changes between review and commit.

This is currently very difficult to test, because it only happens the first time and you have to either go set up a bunch of objects or add a bunch of special casing to the parser to hit the workflow.

I'm making some changes to how it pulls file content. To make those changes easier to test, first start extracting this stuff so the code can be run with `bin/differential extract ...` instead of needing to do a bunch of more complicated setup steps.

Test Plan:
  - Ran `bin/differential extract ...` to extract diffs from commits.
  - Forced my way through the daemon workflow by faking out a bunch of flags, got a clean extract + attach + update. After this patch, this should rarely be necessary.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9319

Differential Revision: https://secure.phabricator.com/D14967
2016-01-08 09:22:37 -08:00
epriestley
ba37149bf9 Improve bin/lipsum UX
Summary: Ref T9156. This makes the UX a little more modern/standard/safe.

Test Plan:
```
epriestley@orbital ~/dev/phabricator $ ./bin/lipsum generate
Choose which type or types of test data you want to generate, or select "all".

      - Differential Revisions
      - Files
      - Maniphest Tasks
      - Pastes
      - Pholio Mocks
      - Projects
      - User Accounts
```

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9156

Differential Revision: https://secure.phabricator.com/D14873
2015-12-24 09:06:35 -08:00
Joshua Spence
7164606285 Add a lock to storage upgrade and adjustment
Summary: Fixes T9715. Adds a MySQL-based lock to ensure that schema migrations are not applied on multiple hosts simultaneously.

Test Plan: Ran `./bin/storage upgrade` concurrently. One invocation was successful whilst the other hit a `PhutilLockException`.

Reviewers: #blessed_reviewers, epriestley

Subscribers: Korvin

Maniphest Tasks: T9715

Differential Revision: https://secure.phabricator.com/D14463
2015-12-02 06:18:28 +11:00
Joshua Spence
c35b564f4d Various translation improvements
Summary: Depends on D14070.

Test Plan: Eyeball it.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, hach-que

Differential Revision: https://secure.phabricator.com/D14073
2015-11-03 07:02:46 +11:00
epriestley
9c798e5cca Provide bin/garbage for interacting with garbage collection
Summary:
Fixes T9494. This:

  - Removes all the random GC.x.y.z config.
  - Puts it all in one place that's locked and which you use `bin/garbage set-policy ...` to adjust.
  - Makes every TTL-based GC configurable.
  - Simplifies the code in the actual GCs.

Test Plan:
  - Ran `bin/garbage collect` to collect some garbage, until it stopped collecting.
  - Ran `bin/garbage set-policy ...` to shorten policy. Saw change in web UI. Ran `bin/garbage collect` again and saw it collect more garbage.
  - Set policy to indefinite and saw it not collect garabge.
  - Set policy to default and saw it reflected in web UI / `collect`.
  - Ran `bin/phd debug trigger` and saw all GCs fire with reasonable looking queries.
  - Read new docs.

{F857928}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9494

Differential Revision: https://secure.phabricator.com/D14219
2015-10-02 09:17:24 -07:00
epriestley
bcc5e55af2 Push construction of routing maps into Sites
Summary:
This enables CORGI.

Currently, `AphrontSite` subclasses can't really have their own routes. They can do this sort of hacky rewriting of paths, but that's a mess and not desirable in the long run.

Instead, let subclasses build their own routing maps. This will let CORP and ORG have their own routing maps.

I was able to get rid of the `PhameBlogResourcesSite` since it can really just share the standard resources site.

Test Plan:
  - With no base URI set, and a base URI set, loaded main page and resources (from main site).
  - With file domain set, loaded resources from main site and file site.
  - Loaded a skinned blog from a domain.
  - Loaded a skinned blog from the main site.
  - Viewed "Request" tab of DarkConsole to see site/controller info.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D14008
2015-08-31 04:01:01 -07:00
epriestley
d22930c0fd Fix variable name in ssh-exec.php
Summary: Fixes T9176. This was renamed in D13589 but a use site was missed.

Test Plan:
Before:

```
epriestley@orbital ~/dev/phabricator $ SSH_ORIGINAL_COMMAND=conduit sudo -E ./bin/ssh-exec --phabricator-ssh-user admin
phabricator-ssh-exec: Invalid command.
```

After:

```
epriestley@orbital ~/dev/phabricator $ SSH_ORIGINAL_COMMAND=conduit sudo -E ./bin/ssh-exec --phabricator-ssh-user admin
phabricator-ssh-exec: No Conduit method provided.
```

Reviewers: chad, joshuaspence

Reviewed By: chad, joshuaspence

Maniphest Tasks: T9176

Differential Revision: https://secure.phabricator.com/D13893
2015-08-13 18:04:08 -07:00
Joshua Spence
368f359114 Use PhutilClassMapQuery instead of PhutilSymbolLoader
Summary: Use `PhutilClassMaQuery` instead of `PhutilSymbolLoader`, mostly for consistency. Depends on D13588.

Test Plan: Poked around a bunch of pages.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D13589
2015-08-14 07:49:01 +10:00