Summary:
The search indexing API has several problems right now:
- Always runs in-process.
- It would be nice to push this into the task queue for performance. However, the API currently passses an object all the way through (and some indexers depend on preloaded object attributes), so it can't be dumped into the task queue at any stage since we can't serialize it.
- Being able to use the task queue will also make rebuilding indexes faster.
- Instead, make the API phid-oriented.
- No uniform indexing API.
- Each "Editor" currently calls SomeCustomIndexer::indexThing(). This won't work with AbstractTransactions. The API is also just weird.
- Instead, provide a uniform API.
- No uniform CLI.
- We have `scripts/search/reindex_everything.php`, but it doesn't actually index everything. Each new document type needs to be separately added to it, leading to stuff like D3839. Third-party applications can't provide indexers.
- Instead, let indexers expose documents for indexing.
- Not application-oriented.
- All the indexers live in search/ right now, which isn't the right organization in an application-orietned view of the world.
- Instead, move indexers to applications and load them with SymbolLoader.
Test Plan:
- `bin/search index`
- Indexed one revision, one task.
- Indexed `--type TASK`, `--type DREV`, etc., for all types.
- Indexed `--all`.
- Added the word "saboteur" to a revision, task, wiki page, and question and then searched for it.
- Creating users is a pain; searched for a user after indexing.
- Creating commits is a pain; searched for a commit after indexing.
- Mocks aren't currently loadable in the result view, so their indexing is moot.
Reviewers: btrahan, vrana
Reviewed By: btrahan
CC: 20after4, aran
Maniphest Tasks: T1991, T2104
Differential Revision: https://secure.phabricator.com/D4261
Summary:
See discussion in D4204. Facebook currently has a 314MB remarkup cache with a 55MB index, which is slow to access. Under the theory that this is an index size/quality problem (the current index is on a potentially-384-byte field, with many keys sharing prefixes), provide a more general index with fancy new features:
- It implements PhutilKeyValueCache, so it can be a component in cache stacks and supports TTL.
- It has a 12-byte hash-based key.
- It automatically compresses large blocks of data (most of what we store is highly-compressible HTML).
Test Plan:
- Basics:
- Loaded /paste/, saw caches generate and save.
- Reloaded /paste/, saw the page hit cache.
- GC:
- Ran GC daemon, saw nothing.
- Set maximum lifetime to 1 second, ran GC daemon, saw it collect the entire cache.
- Deflate:
- Selected row formats from the database, saw a mixture of 'raw' and 'deflate' storage.
- Used profiler to verify that 'deflate' is fast (12 calls @ 220us on my paste list).
- Ran unit tests
Reviewers: vrana, btrahan
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D4259
Summary: This data structure is a `dict<int, list<Comment>>` now, where the `int` is the line number.
Test Plan:
- Created a diff changing an image.
- Added inline comments on the left and right sides of the diff.
- Saw some exceptions and general sadness.
- Applied patch.
- Reloaded page.
- Everything worked great.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D4264
Summary: D4270 missed a spot I think.
Test Plan: Called it.
Reviewers: btrahan, epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4271
Summary: D4249 missed a spot I think.
Test Plan: no more fatal
Reviewers: vrana, epriestley
Reviewed By: vrana
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4270
Summary: basically when we're on a page we have a URIPath so set that and pass it over. I wasn't super duper happy with this but it seems to be the best way to pass this data.
Test Plan: verified the og:url was correct on my dev instance of phacility blog for both the main blog and individual post view
Reviewers: epriestley
Reviewed By: epriestley
CC: chad, aran, Korvin
Differential Revision: https://secure.phabricator.com/D4269
Summary: this makes it more sensical when you hit "share" from a bookmarklet or cut and paste a link into FB, basically by having post-specific data when sharing a post.
Test Plan: looked at generated HTML on my test blog
Reviewers: epriestley
Reviewed By: epriestley
CC: chad, aran, Korvin
Differential Revision: https://secure.phabricator.com/D4266
Summary: Sets any event text color to white, reguardless if its an anchor or not.
Test Plan: Have events from multiple people in the calendar, view events.
Reviewers: btrahan, epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4263
Summary:
When previewing, save drafts. When loading objects, restore drafts if they are available.
Depends on: D665
Test Plan:
- Viewed a Mock.
- Typed text into the comment box.
- Reloaded the page.
- Text still there.
- Hit submit, got my comment.
- Reloaded the page.
- Draft correctly deleted.
- Repeated for Macros.
Reviewers: btrahan, chad, vrana
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2104
Differential Revision: https://secure.phabricator.com/D4252
Summary:
Implements previews for Macros and Pholio.
(Design is nonfinal -- kind of split the difference between `diff_full_view.png`, laziness, and space concerns. Next couple diffs will add more stuff here.)
Test Plan: {F28055}
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran, vrana
Maniphest Tasks: T2104
Differential Revision: https://secure.phabricator.com/D4246
Summary: Does this seem reasonable? It's a bit more compact than digest() (6 bits / byte instead of 4 bits / byte) and 72 bits of entropy @ 12 bytes instead of 128 bits of entropy @ 32 bytes. I feel like it's important to preserve the printability, though, and this seemed like a fairly good balance of concerns.
Test Plan: unit tests
Reviewers: vrana
Reviewed By: vrana
CC: aran, yemao932
Differential Revision: https://secure.phabricator.com/D4253
Summary: Fixes T2210. Recently, we require unique keys on menu items, but it's currently possible in Maniphest to save the same custom query under multiple names. Avoid exploding in this case (we'll hide the duplicates). This isn't a great fix, but makes Maniphest usable again.
Test Plan: Saved the same query twice, laoded page, got exception, applied patch, loaded page, saw duplicate query stripped.
Reviewers: btrahan, vrana
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2210
Differential Revision: https://secure.phabricator.com/D4247
Summary: We use "D<id>" for revisions.
Test Plan: Looked at revision.
Reviewers: btrahan, epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4258
Summary: 'cuz new fluid layouts require the westerlyness. Looks like D4126 started the N and W implementation but didn't finish it...? note I had to do the shifting of the 5 pixels in javascript; using the CSS didn't work for me in chrome.
Test Plan: uiexample, and hoping it goes well when deployed in prod for differential case
Reviewers: epriestley
Reviewed By: epriestley
CC: chad, aran, Korvin
Maniphest Tasks: T2211
Differential Revision: https://secure.phabricator.com/D4257
Summary: This is used in every other view.
Test Plan: Browsed around.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4248
Summary:
I messed up D4244 (and didn't test it properly) - this change should load
the raw content so we can use it.
Test Plan: use the conduit console and actually test the paste conduit methods
Reviewers: vrana, epriestley, btrahan
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4245
Summary:
The caching introduced by rP7e37eb48273eef87e6e6811703fb5d85a3e07a81
broke the use of PhabricatorPaste::getContent() for forking/editing
pastes and downloading pastes with arc paste. I think this fixes all
the appropriate callsites.
Test Plan: fork a paste in the web ui
Reviewers: vrana, epriestley, btrahan
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4244
Summary:
See f5c2a2ab4b (commitcomment-2333247)
Copy of working implementation from PHPMailerLite.
Also expose the SSL/TLS options.
Test Plan: Switched to this mailer, configured Gmail SMTP, sent email. Verified email arrived intact.
Reviewers: btrahan, vrana
Reviewed By: btrahan
CC: aran, mbeck
Differential Revision: https://secure.phabricator.com/D4239
Summary: We allow user to specify the host to listen to but we listen on 127.0.0.1 instead.
Test Plan:
Hardcoded the path and verified that I can connect to the host from other machine.
Verified that `localhost` still doesn't allow remote connections.
Reviewers: ddfisher, epriestley, nh
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4238
Summary:
- Original command is in SSH_ORIGINAL_COMMAND, not normal argv.
- Use PhutilShellLexer to parse it.
- Fix a protocol encoding issue with ConduitSSHWorkflow. I think I'm going to make this protocol accept multiple commands anyway because SSH pipes are crazy expensive to build (even locally, they're ~300ms).
Test Plan: With other changes, successfully executed "arc list --conduit-uri=ssh://localhost:2222".
Reviewers: btrahan, vrana
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T550
Differential Revision: https://secure.phabricator.com/D4232
Summary:
- Build "sshd-auth" (for authentication) and "sshd-exec" (for command execution) binaries. These are callable by "sshd-vcs", located [[https://github.com/epriestley/sshd-vcs | in my account on GitHub]]. They are based on precursors [[https://github.com/epriestley/sshd-vcs-glue | here on GitHub]] which I deployed for TenXer about a year ago, so I have some confidence they at least basically work.
- The problem this solves is that normally every user would need an account on a machine to connect to it, and/or their public keys would all need to be listed in `~/.authorized_keys`. This is a big pain in most installs. Software like Gitosis/Gitolite solve this problem by giving you an easy way to add public keys to `~/.authorized_keys`, but this is pretty gross.
- Roughly, instead of looking in `~/.authorized_keys` when a user connects, the patched sshd instead runs `echo <public key> | sshd-auth`. The `sshd-auth` script looks up the public key and authorizes the matching user, if they exist. It also forces sshd to run `sshd-exec` instead of a normal shell.
- `sshd-exec` receives the authenticated user and any command which was passed to ssh (like `git receive-pack`) and can route them appropriately.
- Overall, this permits a single account to be set up on a server which all Phabricator users can connect to without any extra work, and which can safely execute commands and apply appropriate permissions, and disable users when they are disabled in Phabricator and all that stuff.
- Build out "sshd-exec" to do more thorough checks and setup, and delegate command execution to Workflows (they now exist, and did not when I originally built this stuff).
- Convert @btrahan's conduit API script into a workflow and slightly simplify it (ConduitCall did not exist at the time it was written).
The next steps here on the Repository side are to implement Workflows for Git, SVN and HG wire protocols. These will mostly just proxy the protocols, but also need to enforce permissions. So the approach will basically be:
- Implement workflows for stuff like `git receive-pack`.
- These workflows will implement enough of the underlying protocol to determine what resource the user is trying to access, and whether they want to read or write it.
- They'll then do a permissons check, and kick the user out if they don't have permission to do whatever they are trying to do.
- If the user does have permission, we just proxy the rest of the transaction.
Next steps on the Conduit side are more simple:
- Make ConduitClient understand "ssh://" URLs.
Test Plan: Ran `sshd-exec --phabricator-ssh-user epriestley conduit differential.query`, etc. This will get a more comprehensive test once I set up sshd-vcs.
Reviewers: btrahan, vrana
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603, T550
Differential Revision: https://secure.phabricator.com/D4229
Summary: in the re-factor quest this got broken. Turns out NewLines (aka $this->new in the parser) gets updated right up until the very end for showing text changes so instead pass over the total line count to the renderer to get this codepath right.
Test Plan: showed content of some generated files in diffusion and all was well
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Maniphest Tasks: T2204
Differential Revision: https://secure.phabricator.com/D4237
Summary:
Because of the way PhabricatorOwnersPackage works, the code to save package
changes when parsing commit changes was raising a few undefined index errors,
and was throwing an exception trying to call a method on null (because not
all of the phids related to the package had their handles loaded). This fix
doesn't load the missing handles, it just avoids trying to use them.
Test Plan:
./scripts/repository/reparse.php on a commit with path changes that triggered
a package change
Reviewers: epriestley, vrana
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4236
Summary: Some fallout from D4191.
Test Plan: Sent "D12" in IRC and got a response.
Reviewers: epriestley, vrana, zeeg
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4235
Summary: Continue work started at D3601.
Test Plan:
Commented declaration `AphrontController::$request`, saw exception.
Brought it back, didn't see exception.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4233
Summary:
Load the data for daemon worker tasks when viewing them, and present
the information in a useful way. This defaults to printing the json data,
but for some classes of worker it will also link to the corresponding
object, to make debugging problems with workers easier.
Test Plan:
load /daemon/task/NNN for a CommitParserWorker and a MetaMTAWorker, and
see the addition of a data field with useful content and link.
Reviewers: epriestley, vrana
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4226
Summary: This is probably not the most useful app to have work on mobile, but get the log view to do something fairly sensible.
Test Plan: Looked at all Drydock views in mobile.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2015
Differential Revision: https://secure.phabricator.com/D4224
Test Plan: Called it on a diff with postponed linters and no messages.
Reviewers: mgummelt, epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4223
Summary: its a bit confusing but "newer" posts are the "previous" page and "older" posts are the "next" page. this is because newer posts are those with higher ids. also make the title be the title of the post if we have an actual post.
Test Plan: set page limit to 5 and got somewhat sensical results (note this pagination seems to break with my test data set where there's fun gaps in the contiguity of the ids in a given blog) viewed an actual post and noted the page title was the post title
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4222
Summary: The logs bits still need some work but add crumbs/lists to everything else. Also build a propery DrydockResourceQuery.
Test Plan: Looked at lease list/detail; resource list/detail.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2015
Differential Revision: https://secure.phabricator.com/D4221
Summary: arc incorrectly passes a "user" parameter to differential.createrevision (long ago, we respected it, I think). After D4191 this fatals. Provide a stub call until the next version bump.
Test Plan: inspection
Reviewers: vrana, btrahan
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D4220
Summary: Minor updates to Drydock things to make them work better. In particular, after this patch working copies are correctly allocated or reused.
Test Plan: Ran "reparse.php --harbormaster <derp derp>", saw reuse of working copies when unleased resources were avilable.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2015
Differential Revision: https://secure.phabricator.com/D4216
Summary:
This is very preliminary and doesn't actually do anything useful. In theory, it uses Drydock to check out a working copy and run tests. In practice, it's not actually capable of running any of our tests (because of complicated interdependency stuff), but does check out a working copy and //try// to run tests there.
Adds various sorts of utility methods to various things as well.
Test Plan: Ran `reparse.php --harbormaster --trace <commit>`, observed attempt to run tests via Drydock.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2015, T1049
Differential Revision: https://secure.phabricator.com/D4215
Summary:
This feature isn't necessarily intentional, but might as well make it work better.
Fixes T2201.
Test Plan: Verified that ##[[link | `name`]]## renders in blue.
Reviewers: chad, btrahan, vrana
Reviewed By: chad
CC: aran
Maniphest Tasks: T2201
Differential Revision: https://secure.phabricator.com/D4217
Summary: We now remember this value, it's remembered also after page refresh.
Test Plan: Reloaded revision about to be accepted.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4196
Summary: I plan to use this in Arcanist.
Test Plan:
$ echo '{}' | arc call-conduit x
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4192
