1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-26 07:20:57 +01:00
Commit graph

728 commits

Author SHA1 Message Date
epriestley
612b29fff4 Implement basic one-up and test renderers
Summary:
This is a half-step toward one-up and test renderers. This is mostly a structural change, and has no user-facing impact. It splits the rendering hierarchy like this:

  - Renderer (more methods are abstract than before)
    - HTML Renderer (most HTML stuff has moved down from the base to here)
      - HTML 1-up (placeholder only -- not yet a functional implementation)
      - HTML 2-up (minimal changes, uses mostly old code)
    - Test Renderer (unit-testable renderer base, implements text versions of the HTML stuff)
      - Test 1-up (selects 1-up mode for test rendering)
      - Test 2-up (selects 2-up mode for test rendering)

Broadly, I'm trying to share as much code as possible by splitting rendering into more, smaller stages. Specifically, we do this:

  - Combine the various sorts of inputs (changes, context, inlines, etc.) into a single, relatively homogenous list of "primitives". This happens in the base class.
    - The primitive types are: old (diff left side), new (diff right side), context ("show more context"), no-context ("context not available") and inline (inline comment).
  - Possibly, apply a filtering/reordering step to the primitives to get them ready for 1-up rendering. This mostly removes information, and does a small amount of reordering. This also happens in the base class.
  - Pass the primitives to the actual renderer, to convert them into HTML, text, or whatever else. This happens in the leaf class.

The primitive implementation is not yet complete (it doesn't attach as much information to the primitives as it should -- stuff like coverage and copies), but covers the basics.

The existing HTMLTwoUp renderer does not use the primitive path; instead, it still goes down the old path.

Test Plan: Ran unit tests, looked at a bunch of diffs.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2009

Differential Revision: https://secure.phabricator.com/D4421
2013-01-14 14:20:06 -08:00
Ricky Elrod
9470ab0576 First go at Files settings.
Summary:
Based on @epriestley's https://secure.phabricator.com/differential/diff/9186/
plus others that seemed like they belong here.

Test Plan: Saw the new settings show up in the web interface.

Reviewers: epriestley, btrahan, chad

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4425
2013-01-13 17:01:02 -08:00
Ricky Elrod
76c10f497f Fix error in PhabricatorSetupIssueView
Summary:
- Move `prettyPrintJSON()` and make it static.
- Use it from `PhabricatorSetupIssueView`
- Update other `config/` places that use it to call it from the new class.

This fixes a bug in `PhabricatorSetupIssueView` which showed up if the value
was an array and couldn't be rendered by `phutil_escape_html()`.

Test Plan:
- Rendered some config options.
- Went to /config/issue/config.unknown.phame.skins/ without error.

Reviewers: epriestley, btrahan, chad

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4411
2013-01-11 15:28:39 -08:00
Ricky Elrod
08687c0b17 Search options.
Test Plan:
- Looked at the new options
- Saved `search.engine-selector` successfully.

Reviewers: epriestley, chad, btrahan

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4410
2013-01-11 15:28:19 -08:00
Ricky Elrod
30345cfbdd Port Maniphest options.
Test Plan: Viewed the new options.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4401
2013-01-11 10:24:37 -08:00
epriestley
ebf2435c49 Make feed its own application
Summary:
Fixes the two-level nav issue introduced by D4376.

(My claim that this page is device ready in the code is something of a lie, but it's fairly close.)

(@chad, this could use an icon at some point, or you can point me at which one you want and I can take a stab at slicing it.)

Test Plan: Looked at feed; saw it not-broken. Also checked public feed (which should just merge at some point).

Reviewers: btrahan, chad

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D4381
2013-01-10 16:06:29 -08:00
Ricky Elrod
054f09fb50 Add syntax highlighting options.
Test Plan: Quick view of each option in the web interface.

Reviewers: epriestley, btrahan, chad

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4386
2013-01-10 09:56:39 -08:00
Bob Trahan
84c27ae255 re-factor DifferentialChangesetParser pass 3 / N
Summary: introducing a new friend called DifferentialHunkParser. Sort of like the DifferentialChangesetParser but works with hunks only. tried to grab hunk parsing type things from across the code base and move them into this new class.

Test Plan: unit tests and played around in Differential a bit.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2009

Differential Revision: https://secure.phabricator.com/D4351
2013-01-09 13:11:17 -08:00
vrana
fc30a6eb33 Add Atom export to Phame
Summary:
There's no link to this yet, I'll add it to some skin.

Fixes T2272.

Test Plan: http://validator.w3.org/appc/

Reviewers: btrahan, epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2272

Differential Revision: https://secure.phabricator.com/D4366
2013-01-09 12:25:50 -08:00
Ricky Elrod
cdaf898b9c PHD Options.
Test Plan: Looked at the options render properly.

Reviewers: epriestley, btrahan, chad

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4367
2013-01-09 06:05:36 -08:00
epriestley
4adf55919c Port Diviner Core to Phabricator
Summary:
This implements most/all of the difficult parts of Diviner on top of Phabricator instead of as standalone components. See T988. In particular, here are the things I want to fix:

**Performance** The Diviner parser works in two stages. The first stage breaks source files into "Atoms". The second stage renders atoms into a display format (e.g., HTML). Diviner currently has a good caching story on the first step of the pipeline, but zero caching in the second step. This means it's very slow, even for a fairly small project like Phabricator. We must re-render every piece of documentation every time, instead of only changed documentation. Most of this diff concerns itself with addressing this problem. There's a fairly large explanatory comment about it, but the trickiest part is that when an atom changes, other atoms (defined in other places) may also change -- for example, if `class B extends A`, editing A should dirty B, even if B is in an entirely different file. We perform analysis in two stages to propagate these changes: first detecting direct changes, then detecting indirect changes. This isn't completely implemented -- we need to propagate 'extends' through more levels -- but I believe it's structurally correct and good enough until we actually document classes.

**Inheritance** Diviner currently has a very weak story on inheritance. I want to inherit a lot more metas/docs. If an interface documents a method, we should just pull that documentation in to every implementation by default (implementations can still override it if they want). It can be shown in grey or something, but it should be desirable and correct to omit documentation of a method implementation when you are implementing a parent. Similarly, I want to pull in inherited methods and @tasks and such. This diff sets up for that, by formalizing "extends" relationships between atoms.

**Overspecialization** Diviner currently specializes atoms (FileAtom, FunctionAtom, ClassAtom, etc.). This is pretty much not useful, because Atomizers (which produce the atoms) need to be highly specialized, and Renderers/Publishers (which consume the atoms) also need to be highly specialized. Nothing interesting actually lives in the atom specializations, and we don't benefit from having them -- it just costs us generality in storage/caches for them. In the new code, I've used a single Atom class to represent any type of atom.

**URIs** We have fairly hideous URIs right now, which are very cumbersome  For in-app doc links, I want to provide nice URIs ("/h/notfications" or similar) which are stable redirects, and probably add remarkup for it: !{notifications} or similar. This diff isn't related to that since it's too premature.

**Search** Once we have a database generation target, we can index the documentation.

**Design** Chad has some nice mocks.

Test Plan: Ran `bin/diviner generate`, `bin/diviner generate --clean`. Saw appropriate graph propagation after edits. This diff doesn't do anything very useful yet.

Reviewers: btrahan, vrana

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T988

Differential Revision: https://secure.phabricator.com/D4340
2013-01-07 14:04:23 -08:00
Ricky Elrod
9aa772ed00 First go at Authentication config options.
Test Plan: Looked at them in the web UI.

Reviewers: epriestley, chad, btrahan

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4355
2013-01-07 12:48:43 -08:00
Ricky Elrod
49b842187e Add Policy options.
Test Plan: Looked at the setting and available options from the dropdown.

Reviewers: epriestley, chad

Reviewed By: chad

CC: aran, Korvin

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4354
2013-01-07 12:47:29 -08:00
Mailson Menezes
712e22208c Store width and height metadata of image files
Summary: Also provide a way to update old files metadata.

Test Plan: Create a revision which includes a image file. Check whether the widht, height metadata exists. Run `scripts/files/manage_files.php metadata --all` to update previously uploaded files.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2101

Differential Revision: https://secure.phabricator.com/D4347
2013-01-07 09:46:43 -08:00
Ricky Elrod
ae0773b789 Add translations config group.
Summary:
Adds the translations group as per T2255. Currently `translation.override` is
`wild` -- it should be changed to dict<string, string> when that exists.

Also fixes a small bug from D4326 which caused "class" types to not ever
validate.

Test Plan:
- Looked at the settings.
- Successfully saved a setting relating to classes.

Reviewers: epriestley, btrahan

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4350
2013-01-04 16:22:56 -08:00
epriestley
9a936b5bf3 Move notification configuration into PHP
Summary: Bring notification settings to PHP.

Test Plan: Viewed notification settings in /config/.

Reviewers: codeblock, btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4337
2013-01-03 09:29:19 -08:00
epriestley
af604464d7 Move Recaptcha config to PHP
Summary: Bring these over. Also sort the group list.

Test Plan: Viewed config.

Reviewers: btrahan, codeblock, chad

Reviewed By: chad

CC: aran

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4338
2013-01-03 09:17:38 -08:00
epriestley
0902543fc8 Port MySQL settings to PHP
Summary:
  - Ports MySQL settings to PHP.
  - Removes "mysql.retries" -- this existed only because Magic Numbers Are Bad, but there is no concievable reason it should ever be set to anything other than 3.
  - Introduced "Hidden" config, which isn't visible from the web (for SaaS, we'll just mark anything with secret keys as "hidden").
  - Introduced "Masked" config, which will be masked in darkconsole once that gets updated.
  - "Hidden" implies "Masked" and "Locked".
  - Moved "storage.default-namespace" here -- it probably makes more sense than core; this was my bad in T2255.
  - Put cancel button back for hidden/locked config.
  - Introduce 'class' config type.

Test Plan: Viewed MySQL options. None are editable.

Reviewers: codeblock, btrahan

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4326
2013-01-03 06:01:14 -08:00
Ricky Elrod
f84e0ca533 Add the "Security" config group options.
Summary: Added all the "Security" group options listed in T2255.

Test Plan:
- Looked at all the options.
- Tested validation on `security.alternate-file-domain`

Reviewers: epriestley, btrahan

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4334
2013-01-03 05:46:37 -08:00
Ricky Elrod
7b2ab1a4bc Add the "Extending Phabricator" config group.
Summary: Refs T2255 and takes care of the "EXTENDING PHABRICATOR" group thereof.

Test Plan: Looked at each of the new options.

Reviewers: epriestley, btrahan

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4325
2013-01-02 15:52:36 -08:00
epriestley
962b432bfb Move LDAP config into PHP
Summary: This config section is weak (poorly documented) and inconsistent (keys with "_" instead of "-") but I'm going to keep punting on improving it until after T1536.

Test Plan: Loaded, examined LDAP config.

Reviewers: btrahan, codeblock

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4322
2013-01-02 14:04:05 -08:00
epriestley
a3fdb20a8e Move GC into PHP and simplify it
Summary:
  - Move GC options into PHP.
  - Remove the "run at" and "run for" options. The GC daemon doesn't actually do any table scans, is very gentle, and runs for like 3 seconds per day in any normal install. Just limit it to running once every 4 hours when it's caught up and call it a day.

Test Plan: Edited GC options.

Reviewers: btrahan, codeblock

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4321
2013-01-02 14:03:08 -08:00
epriestley
ec7d799b2f Add "Core" config, with complex validation
Summary: This is more or less a copy of the validation which lives in `webroot/index.php` right now, but I don't want to wipe that out just yet because there's no way for normal users to see this new validation.

Test Plan: Tried to set "phabricator.base-uri" to crazy nonsense, was harshly rebuffed.

Reviewers: codeblock, btrahan

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4316
2013-01-01 18:22:48 -08:00
epriestley
a86fd38394 Allow configuration to be explicitly validated, including validation of complex attributes
Summary:
  - Allows us to implement setup warnings for edits which don't go through the web UI, e.g. "you edited a config file and set value X to something goofy".
  - Allows us to implement more sophisticated validations, beyond basic type checks (e.g., "phabricator.base-uri" must be a URI).
  - Fixes T358 (or, close enough -- fixes it for all options which have been migrated as per T2255.

Test Plan: Set "darkconsole.enabled" to "xyz" in my config, observed setup warning. Added fake validation, observed web UI edit error.

Reviewers: codeblock, btrahan

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2255, T358

Differential Revision: https://secure.phabricator.com/D4315
2013-01-01 18:15:03 -08:00
epriestley
32e4a7a37f Use transactions to show edit history for Configuration
Summary: Use ApplicationTransactions in Config to create an edit history. Resolves T2256.

Test Plan: {F28477}

Reviewers: btrahan, codeblock

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2256

Differential Revision: https://secure.phabricator.com/D4314
2013-01-01 18:14:41 -08:00
epriestley
21efc7cb64 Show all configuration defaults when editing configuration
Summary: Show the value for all loaded configuration sources.

Test Plan:
{F28469}

{F28470}

{F28471}

Reviewers: btrahan, codeblock

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4312
2013-01-01 14:10:33 -08:00
epriestley
8a52a6d585 Add Disqus, Facebook, Google, GitHub auth config; AWS config
Summary: Also improve behavior for the "unknown config" warning.

Test Plan: Looked at configs, went through unknown config workflow.

Reviewers: btrahan, codeblock

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4310
2013-01-01 14:09:29 -08:00
epriestley
a3bff35b2b Add "Developer" and "Access Log" config option groups, some types
Summary:
  - Add a "developer" option group.
  - Add an "access log" option group.
  - Render the types "bool", "int" and "string" in a more tailored way.
  - Add a config check for dead config. Right now this serves as a "TODO" list of things that need to be migrated.

Test Plan: Looked at config options, setup issues. Edited bool, int, string options.

Reviewers: codeblock, btrahan

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4308
2013-01-01 14:09:17 -08:00
epriestley
c32295aab6 Improve resolution process for nonfatal setup issues
Summary:
  - When a setup issue is nonfatal (i.e., a warning), instruct the user to edit the value from the web UI instead of using `bin/config`.
  - When the user edits configuration in response to a setup issue, send them back to the issue when they're done.
  - When an issue relates to PHP configuration, link to the PHP documentation on configuration.
  - Add new-style setup check for timezone issues.

Test Plan: Mucked with my timezone config, resolved the issues I created.

Reviewers: codeblock, btrahan, vrana

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2221, T2228

Differential Revision: https://secure.phabricator.com/D4298
2012-12-30 17:04:38 -08:00
epriestley
b852f213c3 Begin moving Phabricator configuration into PHP
Summary: Ref T2255. Ref T2221. Lay the groundwork to move configuration into PHP, so we can show descriptions in the web UI, do typechecking, disable application options when an application is uninstalled, etc.

Test Plan:
{F28421}
{F28420}
{F28422}

Reviewers: codeblock, btrahan, vrana

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2221, T2255

Differential Revision: https://secure.phabricator.com/D4306
2012-12-30 15:36:06 -08:00
epriestley
96839d35f4 Detect and raise setup warnings from within Phabricator
Summary:
This is basicaly a light version of D4286. The major problem with D4286 is that it's a huge leap and completely replaces the setup process in one step.

Instead, I want to do this:

  - Add the post-setup warnings (yellow bar with "6 unresolved warnings...").
  - Copy all setup checks into post-setup warnings (so every check has an old-style check and a new-style check).
  - Run that for a little bit and make sure it's stable.
  - Implement fatal post-setup checks (the red screen, vs the yellow bar).
  - Run that for a little bit.
  - Nuke setup mode and delete all the old checks.

This should give us a bunch of very gradual steps toward the brave new world of simpler setup.

Test Plan:
 - Faked APC setup failures, saw warnings raise.
 - Verified that this runs after restart (get + set).
 - Verified that this costs us only one cache hit after first-run (get only).

Reviewers: btrahan, codeblock, vrana, chad

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2228

Differential Revision: https://secure.phabricator.com/D4295
2012-12-30 06:37:49 -08:00
epriestley
ba489f9d85 Add a local configuration source and a non-environmental ENV config source
Summary:
See discussion in T2221. Before we can move configuration to the database, we have a bootstrapping problem: we need database credentials to live //somewhere// if we can't guess them (and we can only really guess localhost / root / no password).

Some options for this are:

  - Have them live in ENV variables.
    - These are often somewhat unfamiliar to users.
    - Scripts would become a huge pain -- you'd have to dump a bunch of stuff into ENV.
    - Some environments have limited ability to set ENV vars.
    - SSH is also a pain.
  - Have them live in a normal config file.
    - This probably isn't really too awful, but:
    - Since we deploy/upgrade with git, we can't currently let them edit a file which already exists, or their working copy will become dirty.
    - So they have to copy or create a file, then edit it.
    - The biggest issue I have with this is that it will be difficult to give specific, easily-followed directions from Setup. The instructions need to be like "Copy template.conf.php to real.conf.php, then edit these keys: x, y, z". This isn't as easy to follow as "run script Y".
  - Have them live in an abnormal config file with script access (this diff).
    - I think this is a little better than a normal config file, because we can tell users 'run phabricator/bin/config set mysql.user phabricator' and such, which is easier to follow than editing a config file.

I think this is only a marginal improvement over a normal config file and am open to arguments against this approach, but I think it will be a little easier for users to deal with than a normal config file. In most cases they should only need to store three values in this file -- db user/host/pass -- since once we have those we can bootstrap everything else. Normal config files also aren't going away for more advanced users, we're just offering a simple alternative for most users.

This also adds an ENVIRONMENT file as an alternative to PHABRICATOR_ENV. This is just a simple way to specify the environment if you don't have convenient access to env vars.

Test Plan: Ran `config set x y`, verified writes. Wrote to ENVIRONMENT, ran `PHABRICATOR_ENV= ./bin/repository`.

Reviewers: btrahan, vrana, codeblock

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2221

Differential Revision: https://secure.phabricator.com/D4294
2012-12-30 06:16:15 -08:00
Ricky Elrod
a774620042 Start of a config web interface.
Summary:
This is somewhat clowny, particularly in how it handles JSON encode/decode, but
I've commented why I did things the way I did. The goal is to store minified JSON
but show pretty-printed JSON where possible, to the user editing it.

Test Plan:
* Went to /config/ and saw a list of keys from the `default` config.
* Clicked on one of them, submitted the default value successfully.
* Changed the value to invalid JSON and got a decent error.
* Changed the value to valid JSON and checked the DB to confirm it saved.
* Confirmed the DB values were minified.
* Confirmed the user-facing values were pretty-printed where they could be.
* Confirmed that PHIDs were getting assigned properly and that isDeleted
  properly defaulted to false/0.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2246

Differential Revision: https://secure.phabricator.com/D4290
2012-12-27 15:21:21 -08:00
epriestley
19b2c3d3d0 Formalize configuration sources and source stacks
Summary: Currently, we have a configuration stack for unit tests, but they're built in to `PhabricatorEnv`. Pull them out and formalize them, so we can add more configuration sources (e.g., database).

Test Plan: Ran unit tests, web requests, scripts. This code had fairly good existing test coverage.

Reviewers: btrahan, vrana

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2223, T2221

Differential Revision: https://secure.phabricator.com/D4284
2012-12-25 06:44:29 -08:00
epriestley
9e6d59829c Consolidate environmental initialization
Summary:
We have a bunch of code duplication now between __init_script__.php and webroot/index.php. Consoldiate these methods and move them into PhabricatorEnv.

Merge PhabricatorRequestOverseer into PhabricatorStartup.

Test Plan: Loaded page, ran script. Wiped PHABRICATOR_ENV; loaded page, ran script; got errors.

Reviewers: btrahan, vrana

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2223

Differential Revision: https://secure.phabricator.com/D4283
2012-12-25 06:15:28 -08:00
epriestley
a88b69a4b6 Add a "setup" cache
Summary:
See T2062. This cache allows us to essentially implement this sort of block:

  if (this_code_has_not_run_since_the_last_server_restart()) {
    ...
  }

This will let us do setup checks automatically (i.e., without a specialized setup mode) without imposing hundreds of milliseconds of `git submodule status` and similar checks on every page load, even if an install does not have APC.

Broadly, the major goals here are:

  - Reduce user errors and support costs related to misconfiguration (e.g., failure to update submodules).
  - Simplify setup and configuration (remove 'phabricator.setup', remove/reduce PHABRICATOR_ENV).
  - Move as much configuration to the web as possible (required for SaaS).

Test Plan:
Added this block to webroot/index.php:

  $cache = PhabricatorCaches::getSetupCache();
  $result = $cache->getKeys(array('x'));
  if (empty($result['x'])) {
    phlog('Cache miss + set.');
    $cache->setKeys(array('x' => 'y'));
  } else {
    phlog('Cache hit.');
  }

Verified it used APC correctly.
Disabled APC and verified it degraded to a reasonable disk-based behavior.

If we miss both of these we end up with no actual caching, but that's the best we can do. This code will also run too early in setup for it to be appropriate to raise exceptions out of this pathway -- later on, we can raise a warning that APC is not installed.

Reviewers: btrahan, vrana

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2227, T2062

Differential Revision: https://secure.phabricator.com/D4281
2012-12-25 06:09:51 -08:00
epriestley
8f0bab73ef Support search indexing in ApplicationTransaction
Summary: Hook D4261 into ApplicationTransaction

Test Plan: Edited a mock; searched for it.

Reviewers: btrahan, vrana

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2104

Differential Revision: https://secure.phabricator.com/D4262
2012-12-21 14:21:50 -08:00
epriestley
f6b1964740 Improve Search architecture
Summary:
The search indexing API has several problems right now:

  - Always runs in-process.
    - It would be nice to push this into the task queue for performance. However, the API currently passses an object all the way through (and some indexers depend on preloaded object attributes), so it can't be dumped into the task queue at any stage since we can't serialize it.
    - Being able to use the task queue will also make rebuilding indexes faster.
    - Instead, make the API phid-oriented.
  - No uniform indexing API.
    - Each "Editor" currently calls SomeCustomIndexer::indexThing(). This won't work with AbstractTransactions. The API is also just weird.
    - Instead, provide a uniform API.
  - No uniform CLI.
    - We have `scripts/search/reindex_everything.php`, but it doesn't actually index everything. Each new document type needs to be separately added to it, leading to stuff like D3839. Third-party applications can't provide indexers.
    - Instead, let indexers expose documents for indexing.
  - Not application-oriented.
    - All the indexers live in search/ right now, which isn't the right organization in an application-orietned view of the world.
    - Instead, move indexers to applications and load them with SymbolLoader.

Test Plan:
  - `bin/search index`
    - Indexed one revision, one task.
    - Indexed `--type TASK`, `--type DREV`, etc., for all types.
    - Indexed `--all`.
  - Added the word "saboteur" to a revision, task, wiki page, and question and then searched for it.
    - Creating users is a pain; searched for a user after indexing.
    - Creating commits is a pain; searched for a commit after indexing.
    - Mocks aren't currently loadable in the result view, so their indexing is moot.

Reviewers: btrahan, vrana

Reviewed By: btrahan

CC: 20after4, aran

Maniphest Tasks: T1991, T2104

Differential Revision: https://secure.phabricator.com/D4261
2012-12-21 14:21:31 -08:00
epriestley
aae5f9efd3 Implement a more compact, general database-backed key-value cache
Summary:
See discussion in D4204. Facebook currently has a 314MB remarkup cache with a 55MB index, which is slow to access. Under the theory that this is an index size/quality problem (the current index is on a potentially-384-byte field, with many keys sharing prefixes), provide a more general index with fancy new features:

  - It implements PhutilKeyValueCache, so it can be a component in cache stacks and supports TTL.
  - It has a 12-byte hash-based key.
  - It automatically compresses large blocks of data (most of what we store is highly-compressible HTML).

Test Plan:
  - Basics:
    - Loaded /paste/, saw caches generate and save.
    - Reloaded /paste/, saw the page hit cache.
  - GC:
    - Ran GC daemon, saw nothing.
    - Set maximum lifetime to 1 second, ran GC daemon, saw it collect the entire cache.
  - Deflate:
    - Selected row formats from the database, saw a mixture of 'raw' and 'deflate' storage.
    - Used profiler to verify that 'deflate' is fast (12 calls @ 220us on my paste list).
  - Ran unit tests

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D4259
2012-12-21 14:17:56 -08:00
vrana
2cc7f82ece Move Conduit methods inside applications
Test Plan:
/conduit/
/conduit/method/arcanist.projectinfo/
Call method

  $ echo '{}' | arc call-conduit user.whoami

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D4268
2012-12-21 12:21:59 -08:00
epriestley
0fd77783a4 Add previews to ApplicationTransaction
Summary:
Implements previews for Macros and Pholio.

(Design is nonfinal -- kind of split the difference between `diff_full_view.png`, laziness, and space concerns. Next couple diffs will add more stuff here.)

Test Plan: {F28055}

Reviewers: btrahan, chad

Reviewed By: btrahan

CC: aran, vrana

Maniphest Tasks: T2104

Differential Revision: https://secure.phabricator.com/D4246
2012-12-21 05:51:33 -08:00
epriestley
4af2e3c4e2 Add PhabricatorHash::digestForIndex()
Summary: Does this seem reasonable? It's a bit more compact than digest() (6 bits / byte instead of 4 bits / byte) and 72 bits of entropy @ 12 bytes instead of 128 bits of entropy @ 32 bytes. I feel like it's important to preserve the printability, though, and this seemed like a fairly good balance of concerns.

Test Plan: unit tests

Reviewers: vrana

Reviewed By: vrana

CC: aran, yemao932

Differential Revision: https://secure.phabricator.com/D4253
2012-12-21 05:43:33 -08:00
epriestley
e78898970a Implement SSHD glue and Conduit SSH endpoint
Summary:
  - Build "sshd-auth" (for authentication) and "sshd-exec" (for command execution) binaries. These are callable by "sshd-vcs", located [[https://github.com/epriestley/sshd-vcs | in my account on GitHub]]. They are based on precursors [[https://github.com/epriestley/sshd-vcs-glue | here on GitHub]] which I deployed for TenXer about a year ago, so I have some confidence they at least basically work.
    - The problem this solves is that normally every user would need an account on a machine to connect to it, and/or their public keys would all need to be listed in `~/.authorized_keys`. This is a big pain in most installs. Software like Gitosis/Gitolite solve this problem by giving you an easy way to add public keys to `~/.authorized_keys`, but this is pretty gross.
    - Roughly, instead of looking in `~/.authorized_keys` when a user connects, the patched sshd instead runs `echo <public key> | sshd-auth`. The `sshd-auth` script looks up the public key and authorizes the matching user, if they exist. It also forces sshd to run `sshd-exec` instead of a normal shell.
    - `sshd-exec` receives the authenticated user and any command which was passed to ssh (like `git receive-pack`) and can route them appropriately.
    - Overall, this permits a single account to be set up on a server which all Phabricator users can connect to without any extra work, and which can safely execute commands and apply appropriate permissions, and disable users when they are disabled in Phabricator and all that stuff.
  - Build out "sshd-exec" to do more thorough checks and setup, and delegate command execution to Workflows (they now exist, and did not when I originally built this stuff).
  - Convert @btrahan's conduit API script into a workflow and slightly simplify it (ConduitCall did not exist at the time it was written).

The next steps here on the Repository side are to implement Workflows for Git, SVN and HG wire protocols. These will mostly just proxy the protocols, but also need to enforce permissions. So the approach will basically be:

  - Implement workflows for stuff like `git receive-pack`.
  - These workflows will implement enough of the underlying protocol to determine what resource the user is trying to access, and whether they want to read or write it.
  - They'll then do a permissons check, and kick the user out if they don't have permission to do whatever they are trying to do.
  - If the user does have permission, we just proxy the rest of the transaction.

Next steps on the Conduit side are more simple:

  - Make ConduitClient understand "ssh://" URLs.

Test Plan: Ran `sshd-exec --phabricator-ssh-user epriestley conduit differential.query`, etc. This will get a more comprehensive test once I set up sshd-vcs.

Reviewers: btrahan, vrana

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T603, T550

Differential Revision: https://secure.phabricator.com/D4229
2012-12-19 11:08:07 -08:00
epriestley
db89e23761 Make Repositories partially policy-aware
Summary: Small step toward repository hosting. No user-visible changes.

Test Plan: Looked at repositories in Diffusion.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T603

Differential Revision: https://secure.phabricator.com/D4227
2012-12-19 11:07:06 -08:00
vrana
8816f08765 Throw instead of log for invalid properties
Summary: Continue work started at D3601.

Test Plan:
Commented declaration `AphrontController::$request`, saw exception.
Brought it back, didn't see exception.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D4233
2012-12-18 16:15:01 -08:00
epriestley
53c1483ee5 Make most Drydock web interfaces work with mobile
Summary: The logs bits still need some work but add crumbs/lists to everything else. Also build a propery DrydockResourceQuery.

Test Plan: Looked at lease list/detail; resource list/detail.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D4221
2012-12-17 14:47:21 -08:00
epriestley
97045077c7 Show Drydock resource leases, add DrydockLeaseQuery, allow reuse of working copies
Summary: Minor updates to Drydock things to make them work better. In particular, after this patch working copies are correctly allocated or reused.

Test Plan: Ran "reparse.php --harbormaster <derp derp>", saw reuse of working copies when unleased resources were avilable.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D4216
2012-12-17 13:53:32 -08:00
epriestley
adfe84ffce Add HarbormasterRunnerWorker, for running CI tests
Summary:
This is very preliminary and doesn't actually do anything useful. In theory, it uses Drydock to check out a working copy and run tests. In practice, it's not actually capable of running any of our tests (because of complicated interdependency stuff), but does check out a working copy and //try// to run tests there.

Adds various sorts of utility methods to various things as well.

Test Plan: Ran `reparse.php --harbormaster --trace <commit>`, observed attempt to run tests via Drydock.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015, T1049

Differential Revision: https://secure.phabricator.com/D4215
2012-12-17 13:43:26 -08:00
epriestley
221562b294 Modernize file uploads
Summary:
Modernizes file uploads. In particular:

  - Adds a mobile menu, with an "Upload File" item.
  - Adds crumbs to the list view, detail view and upload view.
  - Adds "Upload File" action to crumbs.
  - Moves upload file to a separate page.
  - Removes the combined upload file + recent files page.
  - Makes upload file use a normal file control by default (works on mobile).
  - Home page, file list and file upload page are now global drop targets which accept files dropped anywhere on them. Dragging a file into the window shows a mask and an instructional message.
    - User education on this is a little weak but I think that's a big can of worms?
  - Fixes a bug where dropping multiple files into a Remarkup text area produced bad results (resolves T2190).

T879 is related, although it's specifically about Maniphest. I've declined to make global drop targets yet there because there are multiple drop targets on the page with different meanings. That UI needs updating in general.

@chad, do we have an "upload" icon (counterpart to "download")?

Test Plan: Uploaded files in Maniphest, Differential, Files, and from Home. Dragged and dropped multiple files into Differential. Used crumbs, mobile.

Reviewers: chad, btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2190

Differential Revision: https://secure.phabricator.com/D4200
2012-12-16 16:34:01 -08:00
epriestley
7e37eb4827 Provide a highlighter cache for Paste
Summary:
D4188 adds a preview of Paste contents to the list, but gets slow for large lists if you have Pygments installed since it has to spend ~200ms/item highlighting them. Instead, cache the highlighted output.

  - Adds a Paste highlighting cache. This uses RemarkupCache because it has automatic GC and we don't have a more generalized cache for the moment.
  - Uses the Paste cache on paste list and paste detail.
  - Adds a little padding to the summary.
  - Adds "..." if there's more content.
  - Adds line count and language, if available. These are hidden on Mobile.
  - Nothing actually uses needRawContent() but I left it there since it doesn't hurt anything and is used internally (I thought the detail view did, but it uses the file content directly, and must for security reasons).

Test Plan:
{F27710}

  - Profiled paste list, saw good performance and few service calls.
  - Viewed paste.
  - Viewed raw paste content.

Reviewers: codeblock, btrahan, chad

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D4204
2012-12-16 16:33:42 -08:00