1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-25 16:22:43 +01:00
Commit graph

229 commits

Author SHA1 Message Date
Nick Harper
ef0f30b662 Use PhutilArgumentParser in reparse.php
Summary:
Update how arguments get parsed in reparse.php and add two new options
--force-local and --min-date (both to be used with --all). This diff also
fixes a bug in destroy_revision.php

Test Plan: ran the script

Reviewers: epriestley, vrana

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3275
2012-08-13 19:35:30 -07:00
Bob Trahan
aa2ef37e94 quick, ghetto script to rescale profile pictures
Summary: we had a hole in scaling pics from oauth from a bit. that was patched in D2848. this cleans up the old data.

Test Plan: ran it in production. issue as described on task is resolved. also spot checked some profiles and they look good.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T1634

Differential Revision: https://secure.phabricator.com/D3268
2012-08-13 15:29:34 -07:00
epriestley
3e29921f5e Use phutil_utf8_convert() in Phabricator
Summary: See D3252.

Test Plan: This one is nasty to test, I'm going to make some coffee first.

Reviewers: davidreuss, vrana, btrahan

Reviewed By: davidreuss

CC: aran

Maniphest Tasks: T452

Differential Revision: https://secure.phabricator.com/D3254
2012-08-12 08:50:19 -07:00
vrana
b1b2afce95 Support applications in aphrontpath.php
Test Plan:
  $ ./aphrontpath.php /diffusion/

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3233
2012-08-10 10:48:08 -07:00
vrana
80d9a276f3 Automatically resolve merge conflicts in Celerity map
Summary:
We can use `.gitattributes` instead but there's no way how to set repository config for all users in Git, right?
So provide a script writting to `.git/info/attributes` instead so that we don't have to .gitignore `.gitattributes`.

Test Plan:
  $ scripts/celerity/install_merge.sh
  $ git pull # with merge conflict in Celerity map

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3228
2012-08-09 15:59:18 -07:00
Alan Huang
fd3ba9841b Update symbol generation scripts
Summary:
Scripts now return scoped symbols -- in particular, PHP class constants, fields, and methods. ctags gives some for other languages.

(Turns out XHPAST doesn't support traits. But no one uses traits anyway so it's probably fine.)

I couldn't find a list of the context types ctags uses (class/struct/union/enum/maybe others?), so the context code just ignores that. Also, it uses a blacklist for the symbol type instead of a whitelist because there are a ton, they vary by language, and I didn't want to unintentionally exclude anything (P480).

Test Plan: Scrape symbols from arcanist and phabricator. Upload them to sandbox. Search for things.

Reviewers: epriestley

Reviewed By: epriestley

CC: nh, aran, Korvin

Maniphest Tasks: T1602

Differential Revision: https://secure.phabricator.com/D3202
2012-08-08 17:06:05 -07:00
Alan Huang
a8d6af0f42 Infrastructure changes to support scoped symbols
Summary:
 - import_project_symbols supports an optional extra field, which is the
   context of the symbol.
 - Symbol query can take a symbol argument, either as a parameter or a
   URL component (so you can now jump nav to `s Zerg/rush`, for
   example).
 - Conduit method not yet updated. Will do that later.

NOTE: Not providing a context is distinct from providing an empty
context, because an empty context stands for top-level context, i.e.
functions and classes for PHP. It will not find class methods, etc. It
is possible that we should use some weird token that could not normally
be a context name to stand in for empty context.

Test Plan: Do a bunch of symbol searches.

Reviewers: epriestley

Reviewed By: epriestley

CC: nh, aran, Korvin

Maniphest Tasks: T1602

Differential Revision: https://secure.phabricator.com/D3148
2012-08-07 09:28:49 -07:00
Marcel Beck
99e9a26192 Separates the PID and log directories of daemons
Summary: The Log and PID directory should be separable in the config file

Test Plan: Start the daemons, and check if the pid and log files are stored in directories that were specified in the config file.

Reviewers: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3149
2012-08-05 08:35:24 +02:00
epriestley
f0af273165 Add FactCursors and application fact datasources
Summary:
  - Add PhabricatorApplication. This is a general class that I have grand designs for, but used here to allow applications to provide objects for analysis by the facts appliction.
  - Add FactCursors, to keep track of where iterators are.
  - Make the daemon do something sort of useful.
  - Add `bin/fact cursors` for showing and managing objects and cursors.
  - Add some options to `bin/fact analyze`.

Test Plan:
  - `bin/fact cursors`, `bin/fact cursors --reset DifferentialRevision`, `bin/fact cursors --reset X`
  - `bin/fact analyze`, `bin/fact analyze --all`, `bin/fact analyze --iterator DifferentialRevision --skip-aggregates`
  - `bin/phd debug fact`

Reviewers: vrana, btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1562

Differential Revision: https://secure.phabricator.com/D3098
2012-07-30 10:43:49 -07:00
epriestley
7c934e4176 Add a basic "fact" application
Summary:
Basic "Fact" application with some storage, part of a daemon, and a control binary.

= Goals =

The general idea is that we have various statistics we'd like to compute, like the frequency of image macros, reviewer responsiveness, task close rates, etc. Computing these on page load is expensive and messy. By building an ETL pipeline and running it in a daemon, we can precompute statistics and just pull them out of "stats" tables.

One way to do this is just to completely hard-code everything, e.g. have a daemon that runs every hour which issues a big-ass query and dumps results into a table per-fact or per fact-group. But this has a bunch of drawbacks: adding new stuff to the pipeline is a pain, various fact aggregators can't share much code, updates are slow and expensive, we can never build generic graphs on top of it, etc.

I'm hoping to build an ETL pipeline which is generic enough that we can use it for most things we're interested in without needing schema changes, and so that installs can use it also without needing schema changes, while still being specific enough that it's fast and we can build useful stuff on top of it. I'm not sure if this will actually work, but it would be cool if it does so I'm starting pretty generally and we'll see how far I get. I haven't built this exact sort of thing before so I might be way off.

I'm basing the whole thing on analyzing entire objects, not analyzing changes to objects. So each part of the pipeline is handed an object and told "analyze this", not handed a change. It pretty much deletes all the old data about that thing and then writes new data. I think this is simpler to implement and understand, and it protects us from all sorts of weird issues where we end up with some kind of garbage in the DB and have to wipe the whole thing.

= Facts =

The general idea is that we extract "facts" out of objects, and then the various view interfaces just report those facts. This change has on type of fact, a "raw fact", which is directly derived from an object. These facts are concerete and relate specifically to the object they are derived from. Some examples of such facts might be:

  D123 has 9 comments.
  D123 uses macro "psyduck" 15 times.
  D123 adds 35 lines.
  D123 has 5 files.
  D123 has 1 object.
  D123 has 1 object of type "DREV".
  D123 was created at epoch timestamp 89812351235.
  D123 was accepted by @alincoln at epoch timestamp 8397981839.

The fact storage looks like this:

  <factType, objectPHID, objectA, valueX, valueY, epoch>

Currently, we supprot one optional secondary key (like a user PHID or macro PHID), two optional integer values, and an optional timestamp. We might add more later. Each fact type can use these fields if it wants. Some facts use them, others don't. For instance, this diff adds a "N:*" fact, which is just the count of total objects in the system. These facts just look like:

  <"N:*", "PHID-xxxx-yyyy", ...>

...where all other fields are ignored. But some of the more complex facts might look like:

  <"DREV:accept", "PHID-DREV-xxxx", "PHID-USER-yyyy", ..., ..., nnnn> # User 'yyyy' accepted at epoch 'nnnn'.
  <"FILE:macro", "PHID-DREV-xxxx", "PHID-MACR-yyyy", 17, ..., ...> # Object 'xxxx' uses macro 'yyyy' 17 times.

Facts have no uniqueness constraints. For @vrana's reviewer responsiveness stuff, we can insert multiple rows for each reviewer, e.g.

  <"DREV:reviewed", "PHID-DREV-xxxx", "PHID-USER-yyyy", nnnn, ..., mmmm> # User 'yyyy' reviewed revision 'xxxx' after 'nnnn' seconds at 'mmmm'.

The second value (valueY) is mostly because we need it if we sample anything (valueX = observed value, valueY = sample rate) but there might be other uses. We might need to add "objectB" at some point too -- currently we can't represent a fact like "User X used macro Y on revision Z", so it would be impossible to compute macro use rates //for a specific user// based on this schema. I think we can start here though and see how far we get.

= Aggregated Facts =

These aren't implemented yet, but the idea is that we can then take the "raw facts" and compute derived/aggregated/rollup facts based on the raw fact table. For example, the "count" fact can be aggregated to arrive at a count of all objects in the system. This stuff will live in a separate table which does have uniqueness constraints, and come in the next diff.

We might need some kind of time series facts too, not sure about that. I think most of our use cases today are covered by raw facts + aggregated facts.

Test Plan: Ran `bin/fact` commands and verified they seemed to do reasonable things.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran, majak

Maniphest Tasks: T1562

Differential Revision: https://secure.phabricator.com/D3078
2012-07-27 13:34:21 -07:00
epriestley
27f6cc3b27 Support PhabricatorOpaqueEnvelope for managing database passwords
Summary: Currently, MySQL/MySQLi connections store passwords in plain text on the object. Allow them to be stored in PhutilOpaqueEnvelopes instead. See D3053.

Test Plan: Loaded site.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D3054
2012-07-24 11:13:53 -07:00
epriestley
dd70c59465 Use OpaqueEnvelopes for all passwords in Phabricator
Summary:
See D2991 / T1526. Two major changes here:

  - PHP just straight-up logs passwords on ldap_bind() failures. Suppress that with "@" and keep them out of DarkConsole by enabling discard mode.
  - Use PhutilOpaqueEnvelope whenever we send a password into a call stack.

Test Plan:
  - Created a new account.
  - Reset password.
  - Changed password.
  - Logged in with valid password.
  - Tried to login with bad password.
  - Changed password via accountadmin.
  - Hit various LDAP errors and made sure nothing appears in the logs.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D2993
2012-07-17 12:06:33 -07:00
epriestley
5d8b75b4da Use the unified markup cache for Maniphest
Summary:
  - See D2945.
  - Drop `cache` field from ManiphestTransaction.
  - Render task descriptions and transactions through PhabricatorMarkupEngine.
  - Also pull the list of macros more lazily.

Test Plan:
  - Verified transactions and transaction preview work correctly and interact with cache correctly.
  - Verified tasks descriptions and task preview work correctly.
  - Verified we don't hit the imagemacro table when we're rendering everything from cache anymore.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D2946
2012-07-11 11:40:10 -07:00
epriestley
ddf67fce58 Add an example event listener, improve documentation, and add a commit discovery event
Summary: Improve documentation around Phabricator events.

Test Plan: Generated and read documentation. Ran test script.

Reviewers: btrahan, jungejason

Reviewed By: jungejason

CC: aran

Maniphest Tasks: T1092

Differential Revision: https://secure.phabricator.com/D2917
2012-07-03 16:46:27 -07:00
Jason Ge
61b79b5359 Use binary_safe_diff from arcanist
Summary:
binary_safe_diff is needed in arcanist too. Moved it over to
arcanist. See D2915.

Test Plan: diffusion page rendered correctly on binary file.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D2916
2012-07-03 13:51:37 -07:00
epriestley
da0da83431 Fix display of admin "new value" in accountadmin script
Summary: I broke this when converting to PhabricatorUserEditor.

Test Plan:
Ran `accountadmin`, created an admin account, verified the "new value" column showed "Y".

  ACCOUNT SUMMARY

                 OLD VALUE                        NEW VALUE
      Username                                    derp
     Real Name                                    derprp
         Email                                    derp@derp.com
      Password                                    Unchanged
         Admin   N                                Y

Reviewers: btrahan, nikil

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1434

Differential Revision: https://secure.phabricator.com/D2908
2012-07-02 15:22:27 -07:00
epriestley
13d96e6377 Introduce "bin/repository" for repository management
Summary:
Nothing new or exciting here yet, just moving the random scripts/repositories/ things to bin/repository. Also add `repository list`.

(Console stuff comes from D2841.)

Test Plan: Ran `repository list`, `repository pull`, `repository discover`, `repository discover --verbose`, `repository help`.

Reviewers: jungejason, vrana

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D2849
2012-06-25 12:35:37 -07:00
epriestley
a705f336a3 Add vebose logging to PhutilRepositoryPullDaemon
Summary: Add verbose logging. This logging is activated by setting "phd.verbose" in the config, running "phd debug", or explicitly in scripts/repository/pull.php and scripst/repository/discover.php

Test Plan:
  >>> orbital ~/devtools/phabricator $ ./scripts/repository/discover.php GTEST
  Discovering 'GTEST'...
  <VERB> PhabricatorRepositoryPullLocalDaemon Discovering commits in repository 'GTEST'...
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch '()_+abcd$100', at a37bc285a12efa7224fe19f3df54cd90fa2b897a.
  <VERB> PhabricatorRepositoryPullLocalDaemon Skipping, HEAD is known.
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch '+abcd$100', at a37bc285a12efa7224fe19f3df54cd90fa2b897a.
  <VERB> PhabricatorRepositoryPullLocalDaemon Skipping, HEAD is known.
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch '_+abcd$100', at a37bc285a12efa7224fe19f3df54cd90fa2b897a.
  <VERB> PhabricatorRepositoryPullLocalDaemon Skipping, HEAD is known.
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch 'abcd$100', at a37bc285a12efa7224fe19f3df54cd90fa2b897a.
  <VERB> PhabricatorRepositoryPullLocalDaemon Skipping, HEAD is known.
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch 'arcpatch', at 774c7737b2d560a291697126bf4513204ccf661a.
  <VERB> PhabricatorRepositoryPullLocalDaemon Skipping, HEAD is known.
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch 'arcpatch-1', at dc97539bee07293f95990d71f4638335a2531d69.
  <VERB> PhabricatorRepositoryPullLocalDaemon Skipping, HEAD is known.
  <VERB> PhabricatorRepositoryPullLocalDaemon Examining branch 'arcpatch-2', at 1acfaec313c46dd3caa90448800181fb91b0270f.

Reviewers: jungejason

Reviewed By: jungejason

CC: aran

Differential Revision: https://secure.phabricator.com/D2843
2012-06-24 15:06:40 -07:00
vrana
77f44ded98 Initialize translator after loading custom libraries
Summary: Installs may set global translation which may not be available before loading libraries.

Test Plan:
  $ aphrontpath.php /

Reviewers: nh

Reviewed By: nh

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D2819
2012-06-21 15:57:51 -07:00
Bob Trahan
26ca5cbbe3 phd - on start command accurately detect if daemons are running before erroring out
Summary: loadRunningDaemons loads daemons that may be running (have a file pid). other commands handle this appropriately so just make sure the start command knows whats up

Test Plan: phd start, stop, reload all seemed to work just fine.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T1284

Differential Revision: https://secure.phabricator.com/D2808
2012-06-20 12:36:31 -07:00
epriestley
d4b6b095cb Provide a script to completely destroy revisions
Summary:
Someone may or may not have accidentally uploaded secrets to Differential. Provide an administrative mechanism to permanently destroy a revision.

Also fix some of the transaction handling code.

Test Plan:
  $ ./scripts/differential/destroy_revision.php --trace D1
  >>> [0] <connect>
  <<< [0] <connect> 1,060 us
  >>> [1] <query> SELECT * FROM `differential_revision` WHERE `id` = 1
  <<< [1] <query> 473 us

      Really destroy 'D1: asdbas' forever? [y/N] y

  >>> [2] <connect>
  <<< [2] <connect> 628 us
  >>> [3] <query> START TRANSACTION
  <<< [3] <query> 190 us
  >>> [4] <query> SELECT * FROM `differential_diff` WHERE revisionID = 1
  <<< [4] <query> 510 us
  >>> [5] <query> SAVEPOINT Aphront_Savepoint_1
  <<< [5] <query> 122 us
  >>> [6] <query> SELECT * FROM `differential_changeset` WHERE diffID = 1
  <<< [6] <query> 307 us
  >>> [7] <query> SAVEPOINT Aphront_Savepoint_2
  <<< [7] <query> 241 us
  >>> [8] <query> SELECT * FROM `differential_hunk` WHERE changesetID = 1
  <<< [8] <query> 212 us
  >>> [9] <query> DELETE FROM `differential_hunk` WHERE `id` = 1
  <<< [9] <query> 216 us
  >>> [10] <query> DELETE FROM `differential_changeset` WHERE `id` = 1
  <<< [10] <query> 154 us
  >>> [11] <query> SAVEPOINT Aphront_Savepoint_2
  <<< [11] <query> 118 us
  >>> [12] <query> SELECT * FROM `differential_hunk` WHERE changesetID = 2
  <<< [12] <query> 194 us
  >>> [13] <query> DELETE FROM `differential_hunk` WHERE `id` = 2
  <<< [13] <query> 179 us
  >>> [14] <query> DELETE FROM `differential_changeset` WHERE `id` = 2
  <<< [14] <query> 163 us
  >>> [15] <query> SAVEPOINT Aphront_Savepoint_2
  <<< [15] <query> 105 us
  >>> [16] <query> SELECT * FROM `differential_hunk` WHERE changesetID = 3
  <<< [16] <query> 211 us
  >>> [17] <query> DELETE FROM `differential_hunk` WHERE `id` = 3
  <<< [17] <query> 159 us
  >>> [18] <query> DELETE FROM `differential_changeset` WHERE `id` = 3
  <<< [18] <query> 152 us
  >>> [19] <query> SAVEPOINT Aphront_Savepoint_2
  <<< [19] <query> 124 us
  >>> [20] <query> SELECT * FROM `differential_hunk` WHERE changesetID = 4
  <<< [20] <query> 191 us
  >>> [21] <query> DELETE FROM `differential_hunk` WHERE `id` = 4
  <<< [21] <query> 155 us
  >>> [22] <query> DELETE FROM `differential_changeset` WHERE `id` = 4
  <<< [22] <query> 149 us
  >>> [23] <query> SELECT * FROM `differential_diffproperty` WHERE diffID = 1
  <<< [23] <query> 242 us
  >>> [24] <query> DELETE FROM `differential_diffproperty` WHERE `id` = 1
  <<< [24] <query> 196 us
  >>> [25] <query> DELETE FROM `differential_diff` WHERE `id` = 1
  <<< [25] <query> 169 us
  >>> [26] <query> DELETE FROM `differential_relationship` WHERE revisionID = 1
  <<< [26] <query> 178 us
  >>> [27] <query> DELETE FROM `differential_commit` WHERE revisionID = 1
  <<< [27] <query> 164 us
  >>> [28] <query> SELECT * FROM `differential_comment` WHERE revisionID = 1
  <<< [28] <query> 221 us
  >>> [29] <query> DELETE FROM `differential_comment` WHERE `id` = 1
  <<< [29] <query> 172 us
  >>> [30] <query> SELECT * FROM `differential_inlinecomment` WHERE revisionID = 1
  <<< [30] <query> 296 us
  >>> [31] <query> SELECT * FROM `differential_auxiliaryfield` WHERE revisionPHID = 'PHID-DREV-ooky7ozqukpmwget32oc'
  <<< [31] <query> 308 us
  >>> [32] <query> SELECT * FROM `differential_affectedpath` WHERE revisionID = 1
  <<< [32] <query> 4,173 us
  >>> [33] <query> DELETE FROM `differential_revision` WHERE `id` = 1
  <<< [33] <query> 231 us
  >>> [34] <query> COMMIT
  <<< [34] <query> 686 us
  OK, destroyed revision.

Reviewers: csilvers, vrana, jungejason

Reviewed By: csilvers

CC: aran

Differential Revision: https://secure.phabricator.com/D2796
2012-06-19 11:52:50 -07:00
vrana
0acb7734cd Use pht()
Summary:
This is the first step in Phabricator internationalization.
It adds a translation selector and calls it at startup.
Installations can add custom selectors to override some texts.
We can add official translations in future.

Next step is to allow user to choose his translation which will override the global one.

This is currently used only for English plurals.

Test Plan: Displayed a diff with unit test error, verified that it says 'Detail' or 'Details' and not 'Detail(s)'.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T1139

Differential Revision: https://secure.phabricator.com/D2753
2012-06-14 16:25:20 -07:00
epriestley
259638e900 Fix minor issues with D2630
Summary:
  - The config is called "resource-path" and the script references "resource-path", but the actual value checked for is "resource-map".
  - Use nonempty(), since defaulting with getEnvConfig() will give you null if the setting exists but is set to null. This default is nearly useless so maybe we should change it to use coalesce().
  - Remove Celerity map initialization from warmup. We don't currently initialize the environment in warmup, and Celerity initialization now depends on the environment.

Test Plan: Ran patch locally and on FPM-Warmup.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: hsb, aran

Differential Revision: https://secure.phabricator.com/D2662
2012-06-06 09:12:42 -07:00
epriestley
0a7b4591ef Allow usernames to include ".", "-" and "_"
Summary:
See T1303, which presents a reasonable case for inclusion of these characters in valid usernames.

Also, unify username validity handling.

Test Plan: Created a new user with a valid name. Tried to create a new user with an invalid name. Ran unit tests.

Reviewers: btrahan, vrana

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1303

Differential Revision: https://secure.phabricator.com/D2651
2012-06-06 07:09:05 -07:00
vrana
8883c9494f Allow specifying custom celerity resource map
Summary:
We have custom static resources.
We currently include them in Phabricator's celerity resource map which is causing some pain - we need to regenerate the file without our custom resources before pushing upstream, we need to discard our changes before pulling from upstream and we need to rebuild with our changes to run Phabricator.

This diff allows writing and reading the map in other location.
The plan is this - I will run `celerity_mapper.php` twice - once to build Phabricator-only resources (to push to upstream) and once to build Phabricator + ours resoruces to put in our directory.

Better solution would be to create a map just with our resources and read and combine it with Phabricator resources.
But it is complicated because we have dependencies on Phabricator resources.

Test Plan:
`celerity_mapper.php webroot`
`celerity_mapper.php webroot ../facebook/src/__celerity_resource_map__.php`
Delete Phabricator's celerity map, set 'celerity.resource-path' and successfully load Phabricator.

Reviewers: epriestley, edward

Reviewed By: epriestley

CC: aran, Koolvin

Maniphest Tasks: T721

Differential Revision: https://secure.phabricator.com/D2630
2012-06-04 18:45:03 -07:00
vrana
ec9589fb3b Ignore errors in svn diff
Summary: Otherwise attaching the commit diff doesn't work.

Test Plan: Reparsed previously failing commit message.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Koolvin

Differential Revision: https://secure.phabricator.com/D2605
2012-06-01 21:45:33 -07:00
vrana
1ebf9186b4 Depend on class autoloading
Test Plan:
Run setup.
/differential/

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Koolvin

Maniphest Tasks: T1103

Differential Revision: https://secure.phabricator.com/D2612
2012-05-30 16:57:21 -07:00
epriestley
70fd96037b Consolidate user editing code
Summary:
  - We currently have some bugs in account creation due to nontransactional user/email editing.
    - We save $user, then try to save $email. This may fail for various reasons, commonly because the email isn't unique.
    - This leaves us with a $user with no email.
  - Also, logging of edits is somewhat inconsistent across various edit mechanisms.
  - Move all editing to a `PhabricatorUserEditor` class.
  - Handle some broken-data cases more gracefully.

Test Plan:
  - Created and edited a user with `accountadmin`.
  - Created a user with `add_user.php`
  - Created and edited a user with People editor.
  - Created a user with OAuth.
  - Edited user information via Settings.
  - Tried to create an OAuth user with a duplicate email address, got a proper error.
  - Tried to create a user via People with a duplicate email address, got a proper error.

Reviewers: btrahan, vrana, jungejason

Reviewed By: btrahan

CC: tberman, aran

Maniphest Tasks: T1184

Differential Revision: https://secure.phabricator.com/D2569
2012-05-25 07:30:44 -07:00
epriestley
f22d9131c6 Fix "phd repository-launch-readonly"
Summary:
  - Delete an unreachable block of code.
  - Pass "--" to terminate overseer args.

Test Plan: Ran "phd repository-launch-readonly", didn't get argument errors out of the daemon.

Reviewers: jungejason

Reviewed By: jungejason

CC: aran

Differential Revision: https://secure.phabricator.com/D2456
2012-05-10 20:27:55 -07:00
epriestley
6fd88641c3 Minor, instructions suggest "phd list" but really mean "phd status".
"phd list" lists available (not running) daemons.

Auditors: csilvers
2012-05-09 11:33:10 -07:00
epriestley
b800df8c1b Simplify daemon management: "phd start"
Summary:
  - Merge CommitTask daemon into PullLocal daemon. This is another artifact of past instability (and order-dependent parsers). We still publish to the timeline, although this was the last consumer. Long term we'll probably delete timeline and move to webhooks, since everyone who has asked about this stuff has been eager to trade away the durability and ordering of the timeline for the ease of use of webhooks. There's also no reason to timeline this anymore since parsing is no longer order-dependent.
  - Add `phd start` to start all the daemons you need. Add `phd restart` to restart all the daemons you need. So cool~
  - Simplify and improve phd and Diffusion daemon documentation.

Test Plan:
  - Ran `phd start`.
  - Ran `phd restart`.
  - Generated/read documentation.
  - Imported some stuff, got clean parses.

Reviewers: btrahan, csilvers

Reviewed By: csilvers

CC: aran, jungejason, nh

Differential Revision: https://secure.phabricator.com/D2433
2012-05-09 10:29:37 -07:00
epriestley
d2b01aead0 Use one daemon to discover commits in all repositories, not one per repository
Summary:
See D2418. This merges the commit discovery daemon into the same single daemon, and applies all the same rules to it.

There are relatively few implementation changes, but a few things did change:

  - I simplified/improved Mercurial importing, by finding full branch tip hashes with "--debug branches" and using "parents --template {node}" so we don't need to do separate "--debug id" calls.
  - Added a new "--not" flag to exclude repositories, since I switched to real arg parsing anyway.
  - I removed a web UI notification that you need to restart the daemons, this is no longer true.
  - I added a web UI notification that no pull daemon is running on the machine.

NOTE: @makinde, this doesn't change anything from your perspective, but it something breaks this is the likely cause.

This implicitly resolves T792, because discovery no longer runs before pulling.

Test Plan:

  - Swapped databases to a fresh install.
  - Ran "pulllocal" in debug mode. Verified it correctly does nothing (fixed a minor issue with min() on empty array).
  - Added an SVN repository. Verified it cloned and discovered correctly.
  - Added a Mercurial repository. Verified it cloned and discovered correctly.
  - Added a Git repository. Verified it cloned and discovered correctly.
  - Ran with arguments to verify behaviors: "--not MTEST --not STEST", "P --no-discovery", "P".

Reviewers: btrahan, csilvers, Makinde

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T792

Differential Revision: https://secure.phabricator.com/D2430
2012-05-08 12:53:41 -07:00
epriestley
1c62a35710 Run one daemon to pull all working copies, not one daemon per working copy
Summary:
Allow the pull daemon to take a list of repositories. By default, pull all repositories.

Make some effort to respect pull frequencies, although we'll necessarily suffer a bit if running with only one process.

NOTE: We still launch one discovery daemon per working copy, so this only cuts the daemon count in half.

Test Plan:
  - Ran `phd debug pulllocal`, verified behavior.
  - Ran `pull.php P MTEST SVNTEST --trace`, verified it pulled the repos and ran the right commands.
  - Ran `phd repository-launch-master`, verified the right daemons launched, checked daemon console.
  - Ran `phd repository-launch-readonly`, verified the right daemon launched, checked daemon console.

Reviewers: btrahan, csilvers, davidreuss

Reviewed By: csilvers

CC: aran

Differential Revision: https://secure.phabricator.com/D2418
2012-05-07 15:01:10 -07:00
epriestley
87207b2f4e Allow users to have multiple email addresses, and verify emails
Summary:
  - Move email to a separate table.
  - Migrate existing email to new storage.
  - Allow users to add and remove email addresses.
  - Allow users to verify email addresses.
  - Allow users to change their primary email address.
  - Convert all the registration/reset/login code to understand these changes.
  - There are a few security considerations here but I think I've addressed them. Principally, it is important to never let a user acquire a verified email address they don't actually own. We ensure this by tightening the scoping of token generation rules to be (user, email) specific.
  - This should have essentially zero impact on Facebook, but may require some minor changes in the registration code -- I don't exactly remember how it is set up.

Not included here (next steps):

  - Allow configuration to restrict email to certain domains.
  - Allow configuration to require validated email.

Test Plan:
This is a fairly extensive, difficult-to-test change.

  - From "Email Addresses" interface:
    - Added new email (verified email verifications sent).
    - Changed primary email (verified old/new notificactions sent).
    - Resent verification emails (verified they sent).
    - Removed email.
    - Tried to add already-owned email.
  - Created new users with "accountadmin". Edited existing users with "accountadmin".
  - Created new users with "add_user.php".
  - Created new users with web interface.
  - Clicked welcome email link, verified it verified email.
  - Reset password.
  - Linked/unlinked oauth accounts.
  - Logged in with oauth account.
  - Logged in with email.
  - Registered with Oauth account.
  - Tried to register with OAuth account with duplicate email.
  - Verified errors for email verification with bad tokens, etc.

Reviewers: btrahan, vrana, jungejason

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1184

Differential Revision: https://secure.phabricator.com/D2393
2012-05-07 10:29:33 -07:00
vrana
74ef98e76e Delete __init_env__.php
Summary: This script is not used since D976.

Test Plan:
  grep init_env

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Koolvin

Differential Revision: https://secure.phabricator.com/D2379
2012-05-03 14:28:52 -07:00
vrana
73c82e5a94 Display holidays
Summary:
We will need it for two purposes:

- Status tool.
- Nagging tool - @aran suggested using "3 business days" and I don't want it to fall on New Year's Eve or such.

I don't plan working on any interface for editing this as this kind of data should be always imported.

Test Plan:
`bin/storage upgrade`
`scripts/calendar/import_us_holidays.php`
/calendar/

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Koolvin

Differential Revision: https://secure.phabricator.com/D2375
2012-05-03 09:22:52 -07:00
epriestley
f04d8ab1a7 Further improve unit/lint rendering
Summary:
I think this improves things, let me know if you have feedback.

Also addresses T840.

Test Plan: See screenshots...

Reviewers: vrana, btrahan, jungejason

Reviewed By: vrana

CC: aran, zeeg

Maniphest Tasks: T840

Differential Revision: https://secure.phabricator.com/D2357
2012-05-01 10:15:56 -07:00
epriestley
570feee199 Make default database namespace configurable
Summary: Allow the default namespace to be set in configuration, so you can juggle multiple copies of sandbox test data or whatever.

Test Plan: Changed default namespace, verified web UI and "storage" script respect it.

Reviewers: btrahan, vrana, jungejason

Reviewed By: vrana

CC: aran

Maniphest Tasks: T345

Differential Revision: https://secure.phabricator.com/D2341
2012-04-30 11:56:58 -07:00
vrana
f362e5785f Ignore non-existing diffs in copied code detector script
Auditors: epriestley
2012-04-30 11:05:30 -07:00
vrana
e08b4cbb2c Inform about moved code and prefer it over copied code
Summary:
Also reduce the memory usage a little bit (before increasing it again).

I use the same CSS class as for the copied code.

Test Plan: Parsed 100 diffs and checked about 10 of them - looks good.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Koolvin

Differential Revision: https://secure.phabricator.com/D2339
2012-04-30 11:01:15 -07:00
epriestley
087cc0808a Make SQL patch management DAG-based and provide namespace support
Summary:
This addresses three issues with the current patch management system:

  # Two people developing at the same time often pick the same SQL patch number, and then have to go rename it. The system catches this, but it's silly.
  # Second/third-party developers can't use the same system to manage auxiliary storage they may want to add.
  # There's no way to build mock databases for unit tests that need to do reads.

To resolve these things, you can now name your patches whatever you want and conflicts are just merge conflicts, which are less of a pain to fix than filename conflicts.

Dependencies are now a DAG, with implicit dependencies created on the prior patch if no dependencies are specified. Developers can add new concrete subclasses of `PhabricatorSQLPatchList` to add storage management, and define the dependency branchpoint of their patches so they apply in the correct order (although, generally, they should not depend on the mainline patches, presumably).

The commands `storage upgrade --namespace test1234` and `storage destroy --namespace test1234` will allow unit tests to build and destroy MySQL storage.

A "quickstart" mode allows an upgrade from scratch in ~1200ms. Destruction takes about 200ms. These seem like fairily reasonable costs to actually use in tests. Building from scratch patch-by-patch takes about 6000ms.

Test Plan:
  - Created new databases from scratch with and without quickstart in a separate test namespace. Pointed the webapp at the test namespaces, browsed around, everything looked good.
  - Compared quickstart and no-quickstart dump states, they're identical except for mysqldump timestamps and a few similar things.
  - Upgraded a legacy database to the new storage format.
  - Destroyed / dumped storage.

Reviewers: edward, vrana, btrahan, jungejason

Reviewed By: btrahan

CC: aran, nh

Maniphest Tasks: T140, T345

Differential Revision: https://secure.phabricator.com/D2323
2012-04-30 07:54:00 -07:00
epriestley
20a5c9b261 Use "closed", not "committed", in Differential
Summary: "Committed" is SVN-specific language, and confusing in Git and Mercurial. Use neutral language instead.

Test Plan: Inspection.

Reviewers: btrahan, Makinde, vrana, jungejason

Reviewed By: vrana

CC: aran

Maniphest Tasks: T909

Differential Revision: https://secure.phabricator.com/D2087
2012-04-23 17:40:57 -07:00
epriestley
3f6f8421b2 Update package definitions
Summary: So much Javascript. :(

Test Plan: Verified major Diffusion, Differential, Maniphest interfaces pull in no unpackaged CSS/JS. Rebuilt Celerity map.

Reviewers: btrahan, vrana, jungejason

Reviewed By: jungejason

CC: aran

Differential Revision: https://secure.phabricator.com/D2248
2012-04-16 17:51:30 -07:00
vrana
88cba92477 Fix English
Summary: I usually don't dare to fix English but this one doesn't seem correct even to me.

Test Plan: Read.

Reviewers: epriestley, jungejason

Reviewed By: jungejason

CC: aran

Differential Revision: https://secure.phabricator.com/D2214
2012-04-12 13:38:14 -07:00
vrana
32d2395a45 Unify links to www.phabricator.com and phabricator.com
Test Plan:
  scripts/sql/upgrade_schema.php

Verify links at /directory/2/.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran

Maniphest Tasks: T1096

Differential Revision: https://secure.phabricator.com/D2172
2012-04-09 14:32:03 -07:00
vrana
2c8e6f99bd Standardize mysql.configuration-provider
Summary: NOTE: BC break!

Test Plan: /

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, nh

Differential Revision: https://secure.phabricator.com/D2130
2012-04-08 21:32:15 -07:00
epriestley
e4df959064 Use Celerity to version all static resources
Summary:
We don't use versioned URIs for images, so when they change users may get old versions.

This was a particular issue with the recent logo change, which several users reported cache-related issues from.

Instead, use Celerity to manage image URI versions in addition to CSS/JS.

This is complicated, because we need to rewrite image URIs inside of CSS, which means the hash of a CSS file has to be derived from the current image data. Otherwise, when we updated an image the CSS wouldn't update, so we wouldn't be any better off.

So basically we:

  - Find all the "raw" files, and put them into the map.
  - Find all the CSS/JS, perform content-altering transformations on it (i.e., not minification) based on the partial map, and then put it into the map based on transformed hashes.

(If we wanted, we could now do CSS variables or whatever for "free", more or less.)

Test Plan:
  - Regenerated celerity map, browsed site, verified images generated with versioned URIs.
  - Moved "blue" flag image over "green" flag image, regenerated map, verified "green" flag image and the associated CSS changed hashes.
  - Added transformation unit tests; ran unit tests.

Reviewers: btrahan, vrana, jungejason

Reviewed By: vrana

CC: aran

Maniphest Tasks: T1073

Differential Revision: https://secure.phabricator.com/D2146
2012-04-08 10:07:51 -07:00
epriestley
b90d41dd90 Add an "--ignore-duplicates" flag to import_project_symbols.php
Summary: People are hella lazy and don't want to do this themselves.

Test Plan: Generated a symbol file with duplicates and piped it in, got an import under --ignore-duplicates.

Reviewers: kdeggelman, btrahan, vrana, jungejason

Reviewed By: jungejason

CC: aran

Differential Revision: https://secure.phabricator.com/D2145
2012-04-07 17:25:38 -07:00
vrana
d4c5761f41 Customizable MySQL implementation
Test Plan:
- /
- upgrade_schema.php
- Setup
- Try disabling mysql_connect.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran

Differential Revision: https://secure.phabricator.com/D2133
2012-04-07 10:54:12 -07:00
vrana
23988ca482 Support Windows
Test Plan:
Enable setup.
Disable setup.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran

Differential Revision: https://secure.phabricator.com/D2118
2012-04-06 09:34:06 -07:00
vrana
23d5d7a1a6 Delete unused CSS
Summary: Unused since D2062.

Test Plan:
View commit.
View task.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran

Differential Revision: https://secure.phabricator.com/D2108
2012-04-05 10:54:06 -07:00
Jason Ge
4aef876138 Add script to delete revisions without diff
Summary: detect all revisions that don't have a diff, then delete them.

Test Plan:
we have been using this script for several months in
Facebook and it's working well.

Reviewers: epriestley, nh, btrahan

Reviewed By: epriestley

CC: aran, epriestley

Maniphest Tasks: T605

Differential Revision: https://secure.phabricator.com/D2061
2012-03-30 14:14:38 -07:00
epriestley
eaa2ff71d3 Minify static resources
Summary: For production servers, minify CSS and JS by stripping comments, whitespace, etc.

Test Plan: Looked at CSS/JS, it was much smaller.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T264

Differential Revision: https://secure.phabricator.com/D2034
2012-03-28 10:13:53 -07:00
epriestley
a3028558ae Make includes free to the user (application support component)
Summary:
We spend a significant amount of time running includes, even with APC. However, we have rigidly structured includes and can safely run them all in workers before requests occur.

Right now, requests go like this:

  - Apache spawns a worker.
  - Client sends an HTTP request.
  - Apache interprets it.
  - Apache sees it's ".php", so it hands it off to the PHP SAPI.
  - The PHP SAPI starts the PHP interpreter in the worker.
  - The request is handled, etc.

Instead, we want to do this:

  - Worker spawns and loads the world.
  - Client sends an HTTP request.
  - Webeserver interprets it.
  - Sees it's a ".php", hands it off to the SAPI.
  - SAPI executes it on a loaded world.

No SAPIs I know of support this, but I added support to PHP-FPM fairly easily (in the sense that it took me 6 hours and I have a hacky, barely-working mess). Over HTTP (vs HTTPS) the performance improvement is pretty dramatic.

HPHP doesn't significantly defray this cost so we're probably quite a bit faster (to the user) under nginx+PHP-FPM than HPHP after this works for real.

I have the php-fpm half of this patch in a messy state, I'm going to try to port it to be vs php 5.4.

Test Plan: Ran a patched php-fpm, browsed around, site works, appears dramatically faster.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D2030
2012-03-27 16:06:13 -07:00
epriestley
914f044b62 More Drydock Stuff
Summary:
  - Still really really rough.
  - Adds a full synchronous mode for debugging.
  - Adds some logging.
  - It can now allocate EC2 machines and put webroots on them in a hacky, terrible way.
  - Adds a base query class.

Test Plan: oh hey look a test page? http://ec2-50-18-65-151.us-west-1.compute.amazonaws.com:2011/

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T1049

Differential Revision: https://secure.phabricator.com/D2026
2012-03-26 20:54:26 -07:00
Julius Seporaitis
53b06d1a52 Project symbol import from 'ctags'
Summary:
I noticed that documentation said it is possible to have 'ctags' symbol import, so I hacked a quick version. I tested it on Python based project and successfuly imported symbols.

It is limited to classes right now, as the importer script complained about not-unique method names (there are a lot of 'get' & 'post' methods accross classes in my project).

If you would have any feedback about this, I would definetly try to wrap it up for possibly merging into main repository.

Test Plan:
Required 'ctags' tool (ctags.sourceforge.net/) Tested to work with version 5.8+ and didn't work with 3.x.

1. `find . -type f '*.py' | ./generate_ctags_symbols.php > /tmp/symbols`
2. `./import_project_symbols.php` < /tmp/symbols

Reviewers: epriestley, btrahan

Reviewed By: epriestley

CC: seporaitis, aran, epriestley

Maniphest Tasks: T1034

Differential Revision: https://secure.phabricator.com/D1995
2012-03-25 09:50:42 -07:00
vrana
4fba549a99 Use PhabricatorEnv::newObjectFromConfig() wherever possible
Test Plan:
/mail/send/
scripts/aphront/aphrontpath.php /

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1983
2012-03-21 14:57:52 -07:00
epriestley
097e62b45c Use libphutil script initializer to initialize scripts in Phabricator
Summary:
libphutil has some basic environmental sanity checks that we should use when initializing scripts in Phabricator. Principally this:

https://secure.phabricator.com/diffusion/PHU/browse/master/scripts/__init_script__.php;db643ee9f5f524e7$26

Without this, the default ini may set CLI errors to go to some logfile, which means exceptions aren't shown on stderr.

See https://github.com/facebook/phabricator/issues/98/

Test Plan:
  - Ran "php -derror_log=/dev/null -f ./bin/phd debug adslkfnasdfnalks" prior to change; got confusing lack of output.
  - Ran "phd -derror_log=/dev/null -f ./bin/phd debug asdkflnaslfdnala" after change; got exception on stderr.

Reviewers: btrahan, killermonk

Reviewed By: btrahan

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1950
2012-03-19 19:21:10 -07:00
epriestley
d5c3d83a3f Add a little SQL-info script that is sort of useful maybe?
Summary: I'm not aware of an easy way to get this information through normal tools. I'm sure there's some fancy GUI client that has it but this seems worth keeping around.

Test Plan: ran script, got helpful information about data sizes

Reviewers: nh, btrahan

Reviewed By: nh

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1916
2012-03-15 14:16:40 -07:00
vrana
814f4adf6c Stabilize sort in celerity map
Summary:
Smaller diffs, better blame, less noise on a file which is usually ignored.

Less conflicts in merge!

Test Plan:   php -l __celerity_resource_map__.php

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1909
2012-03-15 11:08:14 -07:00
vrana
0c728bd7a3 Don't include requirements of non-providers in celerity map
Summary: Tests and docstubs don't provide anything and they are not required in celerity map.

Test Plan: View Options in Differential.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1893
2012-03-14 22:05:18 -07:00
epriestley
1cca22f3fd Provide a script to "undo" the negative effects of an accidental push in Differential
Summary: If someone accidentally pushes a bunch of commits, revisions might get marked as "Committed" incorrectly. This will restore them to their previous state without too much fuss.

Test Plan: Ran the script on some commits to undo them, it seemed to work correctly.

Reviewers: davidreuss, btrahan

Reviewed By: btrahan

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1877
2012-03-13 11:18:22 -07:00
vrana
4c1e356658 Convert database to UTF-8
Summary: This is the script used for conversion: P319

Test Plan:
Update diff with UTF-8 characters in description.
`sql/upgrade_schema.php`
Verify data in DB and that it looks good on web.

Reviewers: epriestley, nh

Reviewed By: epriestley

CC: aran, epriestley

Maniphest Tasks: T327

Differential Revision: https://secure.phabricator.com/D1830
2012-03-12 12:11:02 -07:00
epriestley
ef74b9b88b Improve messaging around passworded SSH keys
Summary: It would be better to test if a key is passworded, but I couldn't figure out how to do that.

Test Plan: Ran "test_connection.php"

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T924

Differential Revision: https://secure.phabricator.com/D1856
2012-03-12 10:34:27 -07:00
epriestley
4bf559baa0 Add a script to programmatically add image macros
Summary: Urgent request.

Test Plan: Ran "add_macro.php" on some things; verified results in web console.

Reviewers: btrahan, tcook

Reviewed By: btrahan

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1832
2012-03-09 08:55:11 -08:00
epriestley
ce919b0822 Resolve implicit fallthrough in Phabricator
Summary: New implicit fallthrough linter detected a few issues; none of these have behavioral impacts but they can clearly be tightened up. See D1824.

Test Plan: Lint; inspection.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1825
2012-03-08 12:46:29 -08:00
epriestley
adb7f9ed1c Add a script to close all open audits in a repository
Summary: If you import a repository you may trigger a large number of irrelevant audits. Provide a tool to nuke them.

Test Plan: Ran "audit.php Q" (does not exist), "audit.php P" (phabricator) from various repository states.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T904, T940

Differential Revision: https://secure.phabricator.com/D1791
2012-03-06 15:10:25 -08:00
Nick Harper
756faad2aa Provide upgrade script for transition away from Meta MTA daemon
Summary:
When switching from using the MetaMTADaemon to a Taskmaster for sending mail,
if there are messages queued for delivery, they need to be re-queued into the
task system. This patch does that.

Task ID: #

Blame Rev:

Test Plan:
Ran it.

Revert Plan:

Tags:

Reviewers: epriestley, btrahan, jungejason

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1780
2012-03-05 15:15:19 -08:00
Nick Harper
b5897ec9c0 Add dry-run, max version options to upgrade_schema.php script
Summary:
1) Dry run option allows you to run the script to see what patches would get
applied (without actually applying them).
2) Max version allows you to do a partial upgrade.

Task ID: #

Blame Rev:

Test Plan:
ran a dry run twice (to make sure it didn't do anything) and no max version
ran with a dry run and max version to check max version logic works correctly
ran with just a max version, and only patches up to that point got applied

Revert Plan:

Tags:

Reviewers: epriestley, btrahan, jungejason

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1758
2012-03-02 11:23:25 -08:00
epriestley
974efd6d65 Minor, remove metamta reference from update script. 2012-03-02 06:42:31 -08:00
epriestley
41a265b464 Update Javelin and packaging
Summary:
  - Update the Javelin submodule to pick up recent fixes (like D1749).
  - Update the package definitions do do a slightly better job of packaging
resources.

Test Plan:
Up and down work in tokenizers now. Pages load slightly fewer
resources.

Reviewers: vrana, btrahan

Reviewed By: btrahan

CC: aran, epriestley

Maniphest Tasks: T927

Differential Revision: https://secure.phabricator.com/D1751
2012-03-01 17:23:00 -08:00
vrana
9b56c7e8a1 Create script printing controller name based on URL
Summary:
With this script, I am able to add bookmarklet to my browser which edits the
controller responsible for displaying current page by a single click.
Plus it can be useful also for other uses.

Test Plan:
  ./aphrontpath.php /
  ./aphrontpath.php D123
  ./aphrontpath.php /D123
  ./aphrontpath.php /D123/ # doesn't exist
  ./aphrontpath.php https://secure.phabricator.com/D123
  ./aphrontpath.php https://secure.phabricator.com/D123?x=2
  ./aphrontpath.php https://secure.phabricator.com/D123#comment-1
  ./aphrontpath.php differential
  ./aphrontpath.php /differential/
  ./aphrontpath.php /w/ # rewritten by custom config

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley

Maniphest Tasks: T349

Differential Revision: https://secure.phabricator.com/D1746
2012-03-01 15:17:50 -08:00
epriestley
c8b4bfdcd1 Encode "<" and ">" in JSON/Ajax responses to prevent content-sniffing attacks
Summary:
Some browsers will still sniff content types even with "Content-Type" and
"X-Content-Type-Options: nosniff". Encode "<" and ">" to prevent them from
sniffing the content as HTML.

See T865.

Also unified some of the code on this pathway.

Test Plan: Verified Opera no longer sniffs the Conduit response into HTML for
the test case in T865. Unit tests pass.

Reviewers: cbg, btrahan

Reviewed By: cbg

CC: aran, epriestley

Maniphest Tasks: T139, T865

Differential Revision: https://secure.phabricator.com/D1606
2012-02-14 14:51:51 -08:00
epriestley
f19d5fbeae Fix existence test in "reconcile.php"
Summary:
"rev-parse --verify" is actually a terrible test, it only survived my test cases
because I put garbage into the tables with RAND() or similar, not
properly-formatted garbage.

Use "cat-file -t" instead to ensure we perform an object-existence test.

Test Plan: Ran "reconcile.php P" locally, ran "cat-file -t" and "rev-parse
--verify" on properly-formatted but invalid hashes like
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", worked with @rguerin to resolve
commit issues.

Reviewers: btrahan, rguerin

Reviewed By: rguerin

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1590
2012-02-07 10:46:26 -08:00
epriestley
a15130b47c Add a maintenance script for reconciling repositories to disk state
Summary:
@rguerin ran into an issue in his install where Phabricator appears to have
discovered commits which no longer exist, and thus is failing to proceed with
its repository import.

It's not clear how we got into this state. Previously, it was possible by, e.g.,
parsing a different repository's working copy and then switching them back, but
there are now safeguards against that.

I'm taking a three-pronged approach to try to sort this out:

  - Provide a script to get out of this state (this script) and reconcile
Phabricator's view of a repository with an authoritative copy of it. This
basically "un-discovers" any discovered commits which don't actually exist (any
queued tasks to parse them will fail permanently when they fail to load the
commit object).
  - Add more logging to the discovery daemon so we can figure out where commits
came from.
  - Improve Diffusion's UI when stuff is partially discovered (T776).

(This script should also clean up some nonsense on secure.phabricator.com from a
botched Diviner import.)

Test Plan: Ran "reconcile.php" with bogus commits and bogus differential/commit
links, had them expunged. Will work with @rguerin to see if this resolves
things.

Reviewers: btrahan, rguerin

Reviewed By: btrahan

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1552
2012-02-02 16:03:50 -08:00
epriestley
7d1f62409d Move resource allocation to task queue
Summary: Run the actual resource allocation for Drydock out-of-process via the
task queue.

Test Plan: Ran "drydock_control.php", saw it insert a task and wait for task
completion. Ran "phd debug taskmaster" and saw it run the task.

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran, epriestley

Differential Revision: https://secure.phabricator.com/D1470
2012-01-24 09:44:14 -08:00
epriestley
d1ee08b2df Drydock Rough Cut
Summary:
Rough cut of Drydock. This is very basic and doesn't do much of use yet (it
//does// allocate EC2 machines as host resources and expose interfaces to them),
but I think the overall structure is more or less reasonable.

== Interfaces

Vision: Applications interact with Drydock resources through DrydockInterfaces,
like **command**, **filesystem** and **httpd** interfaces. Each interface allows
applications to perform some kind of operation on the resource, like executing
commands, reading/writing files, or configuring a web server. Interfaces have a
concrete, specific API:

  // Filesystem Interface
  $fs = $lease->getInterface('filesystem'); // Constants, some day?
  $fs->writeFile('index.html', 'hello world!');

  // Command Interface
  $cmd = $lease->getInterface('command');
  echo $cmd->execx('uptime');

  // HTTPD Interface
  $httpd = $lease->getInterface('httpd');
  $httpd->restart();

Interfaces are mostly just stock, although installs might add new interfaces if
they expose different ways to interact with resources (for instance, a resource
might want to expose a new 'MongoDB' interface or whatever).

Currently: We have like part of a command interface.

== Leases

Vision: Leases keep track of which resources are in use, and what they're being
used for. They allow us to know when we need to allocate more resources (too
many sandcastles on the existing hosts, e.g.) and when we can release resources
(because they are no longer being used). They also give applications something
to hold while resources are being allocated.

  // EXAMPLE: How this should work some day.
  $allocator = new DrydockAllocator();
  $allocator->setResourceType('sandcastle');
  $allocator->setAttributes(
    array(
      'diffID' => $diff->getID(),
    ));
  $lease = $allocator->allocate();
  $diff->setSandcastleLeaseID($lease->getID());

  // ...

  if ($lease->getStatus() == DrydockLeaseStatus::STATUS_ACTIVE) {
    $sandcastle_link = $lease->getInterface('httpd')->getURI('/');
  } else {
    $sandcastle_link = 'Still building your sandcastle...';
  }
  echo "Sandcastle for this diff: ".$sandcastle_link;

  // EXAMPLE: How this actually works now.
  $allocator = new DrydockAllocator();
  $allocator->setResourceType('host');
  // NOTE: Allocation is currently synchronous but will be task-driven soon.
  $lease = $allocator->allocate();

Leases are completely stock, installs will not define new lease types.

Currently: Leases exist and work but are very very basic.

== Resources

Vision: Resources represent some actual thing we've put somewhere, whether it's
a host, a block of storage, a webroot, or whatever else. Applications interact
through resources by acquiring leases to them, and then getting interfaces
through these leases. The lease acquisition process has a side effect of
allocating new resources if a lease can't be acquired on existing resources
(e.g., the application wants storage but all storage resources are full) and
things are configured to autoscale.

Resources may themselves acquire leases in order to allocate. For instance, a
storage resource might first acquire a lease to a host resource. A 'test
scaffold' resource might lease a storage resource and a mysql resource.

Not all resources are auto-allocate: the entry-level version of Drydock is that
you manually allocate a couple boxes and configure them through the web console.
Then, e.g.,  'storage' / 'webroot' resources allocate on top of them, but the
host pool itself does not autoscale.

Resources are completely stock, they are abstract shells representing any
arbitrary thing.

Currently: Resource exist ('host' only) but are very very basic.

== Blueprints

Vision: Blueprints contain instructions for building interfaces to, (possibly)
allocating, updating, managing, and destroying a specific type of resource in a
specific location. One way to think of them is that they are scripts for
creating and deleting resources. For example, the LocalHost, RemoteHost and
EC2Host blueprints can all manage 'host' resources.

Eventually, we will support more types of resources (storage, webroot,
sandcastle, test scaffold, phacility deployment) and more providers for resource
types, some of which will be in the Phabricator mainline and some of which will
be custom.

Blueprints are very custom and specific to application types, so installs will
define new blueprints if they are making significant use of Drydock.

Currently: They exist but have few capabilities. The stock blueprints do nearly
nothing useful. There is a technically functional blueprint for host allocation
in EC2.

== Allocator

This is just the actual code to execute the lease acquisition process.

Test Plan: Ran "drydock_control.php" script, it allocated a machine in EC2,
acquired a lease on it, interfaced with it, and then released the lease. Ran it
again, got a fresh lease on the existing resource.

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D1454
2012-01-19 21:12:57 -08:00
epriestley
0402e4e06e Make "purge_cache.php --differential" also purge the inline comment cache
Summary: We purge the cache for primary comments only right now, not inline
comments.

Test Plan: Ran "purge_cache.php --differential".

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T683

Differential Revision: https://secure.phabricator.com/D1451
2012-01-18 15:20:33 -08:00
epriestley
82c0795e54 Unify logic for username validation
Summary: Revisit of D1254. Don't require lowercase, just standardize the logic.
The current implementation has nonuniform logic -- PeopleEditController forbids
uppercase.

Test Plan: Ran unit tests, see also D1254.

Reviewers: btrahan, jungejason, aran

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D1415
2012-01-16 11:52:59 -08:00
epriestley
5fd46dce66 Improve order of operations in upgrade script
Summary:
  - Run "phd stop" before stopping apache. This is essentially a smoke test for
PHABRICATOR_ENV being set.
  - Run documentation generation after everything else. Between the pull and the
restart we have some minor exposure to APC issues with deleted files and
out-of-date module definitions, and this limits that.
  - Pull commands out of (x && y) stuff, this prevents "set -e" from working
correctly.

Test Plan: Ran upgrade script locally.

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D1419
2012-01-16 11:52:01 -08:00
Bob Trahan
cf61f0e32d Adding an "ssh" client for conduit
Summary: ..."ssh" is in quotes 'cuz this is step 1 and there's no ssh in sight
at the moment.

Test Plan:
ran api.php PHID-USER-xee4ju2teq7mflitwfcs differential.query a few times...
 - tried valid input, it worked!
 - tried bad input, it worked in that it failed and told me so!
ran api.php crap_user differential.query a few times...
 - verified error message with respect to crap_user
ran api.php PHID-USER-xee4ju2teq7mflitwfcs crap_method a few times...
 - verified error message with respect to crap_method
visited http://phabricator.dev/conduit/method/differential.query a few times...
 - tried valid input, it worked!

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, btrahan, epriestley

Maniphest Tasks: T550

Differential Revision: https://secure.phabricator.com/D1357
2012-01-13 11:54:13 -08:00
jungejason
49cf1ca10c Support deleting individual changeset
Summary:
add support in purge_cache.php. We might want to add support
for maniphest comments and differential comments later.

Test Plan:
- purging all worked
- purging individual changeset worked
- when input is not integer, correct error message reported

Reviewers: epriestley, btrahan, nh

Reviewed By: nh

CC: nh, aran, jungejason, epriestley

Maniphest Tasks: T683

Differential Revision: https://secure.phabricator.com/D1372
2012-01-11 20:14:55 -08:00
epriestley
58f2cb2509 Provide a script for batch creating user accounts
Summary: Make it a little easier to create a bunch of accounts if your company
has more than like 5 employees.

Test Plan: Ran "add_user.php" to create new users. Created new users from the
web console.

Reviewers: btrahan, jungejason, rguerin

Reviewed By: btrahan

CC: aran, btrahan, rguerin

Differential Revision: https://secure.phabricator.com/D1336
2012-01-06 11:50:51 -08:00
epriestley
fd8303aa75 Document how to use the symbol importer
Summary: Some day we might have a fancy daemon for this, but for now at least
provide some instructions on using the existing importers, etc., to index
project symbols.

Test Plan:
  - Generated documentation, read over the result.
  - Ran the example code.

Reviewers: btrahan, jungejason, davidreuss

Reviewed By: jungejason

CC: aran, jungejason

Maniphest Tasks: T315

Differential Revision: https://secure.phabricator.com/D1262
2011-12-22 06:45:59 -08:00
epriestley
c93fc91e96 Update Javelin; improve package definitions
Summary:
  - Update Javelin to HEAD -- this doesn't pick up anything in particular, but
lets us smoke test some stuff like {D1217}.
  - Do a little more packaging since we've picked up a handful of 10-line
behaviors and such for various UI tweaks.

Test Plan:
  - Generally, this should be very low-risk.
  - Browed Maniphest, Differential, Diffusion and tried to hit all the JS
interactions.
  - Looked over the Javelin changes we're pulling in to see if I forgot
anything. The only API change I caught was removal of "JX.defer()", but that was
already cleared in Phabricator in D803.

Reviewers: aran, btrahan, jungejason

Reviewed By: aran

CC: aran

Differential Revision: 1240
2011-12-20 08:27:54 -08:00
epriestley
074bf4ed7d Add a script for purging long-lived caches
Summary: See task; installs occasionally need to do this themselves, and a
script is much better than me telling them to truncate tables.

Test Plan:
Ran various flavors of this command:

  - purge_cache.php
  - purge_cache.php derp
  - purge_cache.php --help
  - purge_cache.php --all
  - purge_cache.php --differential
  - purge_cache.php --differential --maniphest

Then I verified the actual behavior:

  - Visited a Differential revision with comments, observed cache update in
'Services' tab.
  - Visited a Maniphest task with comments, observed cache update in 'Services'
tab.
  - Reloaded a diff standalone view, got a cache update.

Reviewers: Makinde, btrahan, jungejason

Reviewed By: jungejason

CC: aran, jungejason

Maniphest Tasks: T676

Differential Revision: 1214
2011-12-16 12:43:31 -08:00
jungejason
c13b7da290 Add Related Commits for Owners
Summary:
For each commit, find the affected packages, and provide a way to
search by package.

Test Plan:
create commits that touch and don't touch two packages, and verify
that they display correctly in all the UI pages.

Reviewers: epriestley, blair, nh, tuomaspelkonen

Reviewed By: epriestley

CC: benmathews, aran, epriestley, btrahan, jungejason, mpodobnik, prithvi

Maniphest Tasks: T83

Differential Revision: 1208
2011-12-14 22:48:57 -08:00
David Reuss
dc0d3f3fef Fix encodings for mime headers and body if not UTF-8
Summary:
If mails are not sent in UTF-8 we cannot just it verbatim, so we have to
encode it into UTF-8 if it is not the case. Mime headers use different
encodings like "quoted-printable", which we have to handle.

It looks like "Subject: =?iso-8859-1?Q?opr=E6t_s=E5_den_task?=", and can
be decoded by ##iconv_mime_decode##.

Furthermore the body of the email might be in various encodings as well,
which we attempt to pull from the content-type header of the plain text
part of the mail.

Test Plan:
Attempted receiving mails in a variety of flavors. These could be
converted to test-cases once i know if this is a sane solution. Got
expected results from mails sent with Windows-1252 and ISO-8859-1.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley, davidreuss

Differential Revision: 1093
2011-12-02 08:47:45 -08:00
Evan Priestley
0511bf36b3 Merge pull request #81 from moos3/master
Fix php binary location from /usr/bin/php to /usr/bin/env php
2011-11-30 09:26:48 -08:00
moos3
31c3f2a03e Fix to calling the php binary
Summary: by default it was looking for php to be in /usr/bin/php on some systems
this is incorrect. It should be using /usr/bin/env php for calling php with the
correct path.

Test Plan: patch and try to run.

Reviewers: epriestley

Reviewed By: epriestley

CC: svemir, aran, epriestley

Differential Revision: 1136
2011-11-30 12:21:41 -05:00
Bob Trahan
0795cd4baa Add cycle detection to celerity mapper
Summary: create CelerityResourceGraph, which extends AbstractDirectedGraph.
since we've done a bunch of work already to load the resource graph into memory
CelerityResourceGraph simply stores a copy and makes loadEdges work off that
stored copy.

Test Plan:
made phabricator-prefab require herald-rule-editor

~/code/phabricator> ./scripts/celerity_mapper.php webroot
Finding static resources...
Processing 154
files..........................................................................................................................................................
[2011-11-22 11:28:29] EXCEPTION: (Exception) Cycle detected in resource graph:
phabricator-prefab => herald-rule-editor => phabricator-prefab at
[/Users/btrahan/Dropbox/code/phabricator/scripts/celerity_mapper.php:173]

fixed phabricator-prefab requiring herald-rule-editor.  re-ran celerity_mapper
and no errors!

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, btrahan, epriestley

Differential Revision: 1132
2011-11-29 12:09:08 -08:00
adonohue
75e1a0d5a8 Fix celerity_mapper.php @generated issue
Summary: Easy

Test Plan: Run ##scripts/celerity_mapper.php webroot/##, verify
src/__celerity_resource_map__.php

Reviewers: epriestley, btrahan

Reviewed By: epriestley

CC: aran, epriestley

Differential Revision: 1133
2011-11-28 11:44:59 -08:00
David Reuss
13d15276a5 Catch any exceptions thrown when handling mail
Summary:
This makes debugging issues a lot easier, since any exceptions thrown
gets logged and can be seen at /mail/received/, for inspection.

Test Plan:
Created a task via the public-author functionality, and saw no errors
without this patch applied. Even worse, the exception was never shown at
command line either. Not sure what's up with that. Output buffering, or
whatever?

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley, davidreuss

Differential Revision: 1042
2011-10-28 08:05:27 -07:00
Jason Ge
13eee1a344 Add test to check all symbols can be loaded
Summary:
make sure all symboles can be loaded to avoid issues like missing
methods in descendants of abstract base class.

Test Plan:
ran it and verified it passes; remove a method in a descendant class
and verified that the test failed.

Reviewers: epriestley, nh

Reviewed By: nh

CC: aran, nh, jungejason

Differential Revision: 1023
2011-10-20 16:43:13 -07:00
John Chee
bafb75bdea Fix string matching bug in Ubuntu install script.
Commit f58e63a917 introduced a bug that
always set GIT to 'git-core'.
2011-10-17 10:18:18 -07:00
John Chee
f58e63a917 Install proper git package in Ubuntu install script.
'git-core' provided the 'git' command in Ubuntu LTS.
2011-10-14 17:06:12 -07:00
John Chee
d5431f6d9d Install php5-cli in Ubuntu install script.
php5-cli is necessary to run php on the command line.
2011-10-14 17:03:45 -07:00
epriestley
e4e5c39457 Merge __init_env__.php into __init_script__.php
Summary: There are currently two files, but all scripts require both of them,
which is clearly silly. In the longer term I want to rewrite all of this init
stuff to be more structured (e.g., merge webroot/index.php and __init_script__
better) but this reduces the surface area of the ad-hoc "include files" API we
have now, at least.

Test Plan:
  - Grepped for __init_env__.php (no hits)
  - Ran a unit test (to test unit changes)
  - Ran a daemon (to test daemon changes)

Reviewers: jungejason, nh, tuomaspelkonen, aran

Reviewed By: jungejason

CC: aran, jungejason

Differential Revision: 976
2011-10-02 11:48:09 -07:00
Ricky Elrod
10570635b5 Stop 'stop' from being in phd's list twice, and provide a way to kill one particular PID.
Summary:
This is a pretty bad, but working implmentation of a way to kill one particular PID that
is controlled by Phabricator. Also remove the second 'stop' from the ##phd help## list.

Test Plan:
  [ricky@rhelpad01 phabricator] (phd-stop-twice)$ ./bin/phd status
  PID  	Started                 	Daemon
  30154	Oct 1 2011, 2:38:08 AM  	PhabricatorMetaMTADaemon
  30172	Oct 1 2011, 2:38:09 AM  	PhabricatorMetaMTADaemon
  30190	Oct 1 2011, 2:38:09 AM  	PhabricatorMetaMTADaemon
  30210	Oct 1 2011, 2:38:09 AM  	PhabricatorMetaMTADaemon

  [ricky@rhelpad01 phabricator] (phd-stop-twice)$ ./bin/phd stop 30190
  Stopping daemon 'PhabricatorMetaMTADaemon' (30190)...
  Daemon 30190 exited normally.

  [ricky@rhelpad01 phabricator] (phd-stop-twice)$ ./bin/phd stop 123456
  123456 is not controlled by Phabricator. Not killing.

  [ricky@rhelpad01 phabricator] (phd-stop-twice)$ ./bin/phd stop
  Stopping daemon 'PhabricatorMetaMTADaemon' (30154)...
  Stopping daemon 'PhabricatorMetaMTADaemon' (30172)...
  Stopping daemon 'PhabricatorMetaMTADaemon' (30210)...
  Daemon 30210 exited normally.
  Daemon 30154 exited normally.
  Daemon 30172 exited normally.

Reviewers: epriestley

CC:

Differential Revision: 975
2011-10-01 17:31:20 -04:00
epriestley
fc5cc180ab Fix __init_script__.php issue with event engine. 2011-09-30 13:17:45 -07:00