Summary:
Ref T13120. Ref T12414. See PHI145. See PHI473. This partially modernizes AlmanacService transactions by moving them to ModularTransactions.
This isn't complete because the "update property" and "remove property" transactions aren't modularized. They still //work//, since the parent Editor implements them, but they no longer render properly on the timeline since the `Transaction` object no longer has rendering logic for them.
Tentatively, I'm going to try to convert the rest of the Almanac objects and then modularize those transactions. (Currently, all of Binding, Device, Namespace and Service support properties, although they can only actually be edited on Service, Device and Binding.)
If that turns out to be really tricky for some reason I can just copy/paste the timeline rendering for now, but I think it won't be too hard.
Test Plan:
- Created and edited Services.
- Tried to create a service with: a bad name, no name, a name which put it in a namespace I can't edit (got errors in all cases).
- Edited and removed properties. The edits worked, the timeline just renders a generic story now ('X edited this object (transaction type "almanac:property:update").').
Reviewers: amckinley
Reviewed By: amckinley
Maniphest Tasks: T13120, T12414
Differential Revision: https://secure.phabricator.com/D19317
Summary: Ref T13105. This needs refinement but blame sort of works again, now.
Test Plan: Viewed files in Diffusion and Files; saw blame in Diffusion when viewing in source mode.
Reviewers: mydeveloperday
Reviewed By: mydeveloperday
Maniphest Tasks: T13105
Differential Revision: https://secure.phabricator.com/D19309
Summary:
Ref T13105. This breaks about 9,000 features but moves Diffusion to DocumentEngine for rendering. See T13105 for a more complete list of all the broken stuff.
But you can't bake a software without breaking all the features every time you make a change, right?
Test Plan: Viewed various files in Diffusion, used DocumentEngine features like highlighting and rendering engine selection.
Reviewers: mydeveloperday
Reviewed By: mydeveloperday
Subscribers: mydeveloperday
Maniphest Tasks: T13105
Differential Revision: https://secure.phabricator.com/D19302
Summary:
Ref T13105. This separates document rendering from the Controllers which trigger it so it can be reused elsewhere (notably, in Diffusion).
This shouldn't cause any application behavior to change, it just pulls the rendering logic out so it can be reused elsewhere.
Test Plan: Viewed various types of files in Files; toggled rendering, highlighting, and encoding.
Reviewers: mydeveloperday
Reviewed By: mydeveloperday
Maniphest Tasks: T13105
Differential Revision: https://secure.phabricator.com/D19301
Summary: Ref T13110. Installs have various reasons for sending unreviewable changes (changes where the text of the change will never be reviewed by a human) through Differential anyway. Prepare for accommodating this more gracefully by building a standalone changeset list page which paginates the changesets.
Test Plan: Clicked the new "Changeset List" button on a revision, was taken to a separate page.
Maniphest Tasks: T13110
Differential Revision: https://secure.phabricator.com/D19295
Summary:
Depends on D19280. Ref T13110. Although Harbormaster cares about all builds, Differential does not practically care about local lint and unit results in determining build status.
In Differential, orient publishing around "remote builds" instead of "builds".
This does not yet change any of the draft logic, it just makes the timeline story use newer logic.
Test Plan: Used `bin/harbormaster publish` (with some guard-clause removal) to publish some buildables to revisions without anything crashing.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13110
Differential Revision: https://secure.phabricator.com/D19281
Summary:
Depends on D19279. Ref T13110. This implements the existing publishing logic for buildables, but does so via ModularTransactions instead of a core transaction type.
Since each application is implementing build transactions independently, this removes the core type.
Next, Differential will get a similar treatment.
Test Plan: Used `bin/harbormaster publish` (with some commenting-out-guard-clauses) to publish a commit Buildable; saw unchanged feed behavior.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13110
Differential Revision: https://secure.phabricator.com/D19280
Summary:
Depends on D19278. Ref T13110. This moves most of the structural logic for publishing builds to BuildableEngine and provides a `bin/harbormaster publish` to make publishing easy to retry/debug.
This intentionally removes the bit which actually does anything when builds publish. Followup changes will implement application-specific versions of the publishing logic in Differential and Diffusion.
Test Plan: Ran `bin/harbormaster publish Bxxx`, saw it do nothing (but not crash).
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13110
Differential Revision: https://secure.phabricator.com/D19279
Summary:
Ref T13110. Currently, build status is published the same way for every Buildable by the BuildEngine.
I want to change this to delegate publishing to each Buildable, particularly so that Differential may use more detailed rules for handling builds and drafts.
Rather than add additional methods to the existing `BuildableInterface`, add an engine generator method instead. This is a pattern which has seen more use recently (e.g., in Ferret) and lets us pay a little more upfront to pull complex pieces of logic out of the main class and let them use inheritence more easily. If we had Traits that might cover this to some degree.
I'd expect to eventually reduce the size of `BuildableInterface` and move the `CircleCI` and `BuildKite` interfaces so that the `BuildableEngine` implements them instead of the main object.
Here, this new engine does nothing and is never instantiated. In upcoming changes, publishing logic will move into it so that Differential can handle publishing differently.
Test Plan: Ran `arc liberate`, loaded pages, grepped for `BuildableInterface`.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13110
Differential Revision: https://secure.phabricator.com/D19278
Summary:
Ref T13114. See PHI514. This makes some attempt to undo the damage caused by incorrectly publishing a repository.
Don't run this.
Test Plan: Yikes.
Maniphest Tasks: T13114
Differential Revision: https://secure.phabricator.com/D19271
Summary:
Ref T13114. See PHI511. Ref T13072. This makes Buildables, Builds, Targets and Artifacts destructible with `bin/remove destroy`.
This might not be totally exhaustive. In particular:
- File artifacts won't destroy the file. This is sort of okay because file artifacts are currently just a file reference, but probably shouldn't be how things work in the long term.
- `BuildCommand` doesn't get cleaned up, but `BuildMessage` does on `Build`. See T13072 for more.
Test Plan: Used `bin/remove destroy` to nuke a bunch of builds, buildables, etc. Loaded stuff in the web UI and it all looked like it got nuked properly.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13114, T13072
Differential Revision: https://secure.phabricator.com/D19269
Summary:
Ref T13105. This adds various small client-side improvements to document rendering.
- In the menu, show which renderer is in use.
- Make linking to lines work.
- Make URIs persist information about which rendering engine is in use.
- Improve the UI feedback for transitions between document types.
- Load slower documents asynchronously by default.
- Discard irrelevant requests if you spam the view menu.
Test Plan: Loaded files, linked to lines, swapped between modes, copy/pasted URLs.
Maniphest Tasks: T13105
Differential Revision: https://secure.phabricator.com/D19256
Summary: Depends on D19254. This engine just formats JSON files in a nicer, more readable way.
Test Plan: Looked at some JSON files, saw them become formatted nicely.
Reviewers: mydeveloperday
Reviewed By: mydeveloperday
Differential Revision: https://secure.phabricator.com/D19255
Summary: Ref T13105. Allow normal text files to be rendered as documents, and add a "source code" rendering engine.
Test Plan: Viewed some source code.
Reviewers: mydeveloperday
Reviewed By: mydeveloperday
Maniphest Tasks: T13105
Differential Revision: https://secure.phabricator.com/D19254
Summary:
Depends on D19252. Ref T13105. This very roughly renders Jupyter notebooks.
It's probably better than showing the raw JSON, but not by much.
Test Plan:
- Viewed various notebooks with various cell types, including markdown, code, stdout, stderr, images, HTML, and Javascript.
- HTML and Javascript are not live-fired since they're wildly dangerous.
Maniphest Tasks: T13105
Differential Revision: https://secure.phabricator.com/D19253
Summary:
Depends on D19251. Ref T13105. This adds rendering engine support for PDFs.
It doesn't actually render them, it just renders a link which you can click to view them in a new window. This is much easier than actually rendering them inline and at least 95% as good most of the time (and probably more-than-100%-as-good some of the time).
This makes PDF a viewable MIME type by default and adds a narrow CSP exception for it. See also T13112.
Test Plan:
- Viewed PDFs in Files, got a link to view them in a new tab.
- Clicked the link in Safari, Chrome, and Firefox; got inline PDFs.
- Verified primary CSP is still `object-src 'none'` with `curl ...`.
- Interacted with the vanilla lightbox element to check that it still works.
Maniphest Tasks: T13105
Differential Revision: https://secure.phabricator.com/D19252
Summary:
Ref T13105. Although Markdown is trickier to deal with, we can handle Remarkup easily.
This may need some support for encoding options.
Test Plan: Viewed `.remarkup` files, got remarkup document presentation by default. Viewed other text files, got an option to render as remarkup.
Reviewers: avivey
Reviewed By: avivey
Subscribers: mydeveloperday, avivey
Maniphest Tasks: T13105
Differential Revision: https://secure.phabricator.com/D19251
Summary: Depends on D19237. Ref T13105. This adds a (very basic) "Hexdump" engine (mostly just to have a second option to switch to) and a selector for choosing view modes.
Test Plan: Viewed some files, switched between audio/video/image/hexdump.
Maniphest Tasks: T13105
Differential Revision: https://secure.phabricator.com/D19238
Summary:
Ref T13105. This change begins modularizing document rendering. I'm starting in Files since it's the use case with the smallest amount of complexity.
Currently, we hard-coding the inline rendering for images, audio, and video. Instead, use the modular engine pattern to make rendering flexible and extensible.
There aren't any options for switching modes yet and none of the renderers do anything fancy. This API is also probably very unstable.
Test Plan: Viewwed images, audio, video, and other files. Saw reasonable renderings, with "nothing can render this" for any other file type.
Maniphest Tasks: T13105
Differential Revision: https://secure.phabricator.com/D19237
Summary: See PHI433. Ref T13102. Users in the wild have mixed expecations about exactly what "draft" means. Recent changes have tried to make behavior more clear. As part of clarifying messaging, make it explicit that `@mention` does not work on drafts by showing users a warning when they try to `@mention` a user.
Test Plan:
- Mentioned users on drafts, got a warning.
- Posted normal comments on drafts, no warning.
- Posted normal/mention comments on non-drafts, no warning.
Maniphest Tasks: T13102
Differential Revision: https://secure.phabricator.com/D19210
Summary: Ref T13103. Make favicons customizable, and perform dynamic compositing to add marker to indicate things like "unread messages".
Test Plan: Viewed favicons in Safari, Firefox and Chrome. With unread messages, saw pink dot composited into icon.
Maniphest Tasks: T13103
Differential Revision: https://secure.phabricator.com/D19209
Summary:
Depends on D19198. Ref T13101. Ref T5258. Pull compositing logic out of the `Controller`.
This is moving toward fixing memes in email.
Test Plan: Used new and old memes. Used API memes.
Maniphest Tasks: T13101, T5258
Differential Revision: https://secure.phabricator.com/D19200
Summary:
Depends on D19192. Ref T4190. Ref T13101. Instead of directly including the proxy endpoint with `<img src="..." />`, emit a placeholder and use AJAX to make the request. If the proxy fetch fails, replace the placeholder with an error message.
This isn't the most polished implementation imaginable, but it's much less mysterious about errors.
Test Plan: Used `{image ...}` for valid and invalid images, got images and useful error messages respectively.
Maniphest Tasks: T13101, T4190
Differential Revision: https://secure.phabricator.com/D19193
Summary: Ref T13099. Move most of the "Update" logic to modular transactions
Test Plan: Created and updated revisions. Flushed the task queue. Grepped for `TYPE_UPDATE`. Reviewed update transactions in the timeline and feed.
Maniphest Tasks: T13099
Differential Revision: https://secure.phabricator.com/D19175
Summary: Depends on D19173. Ref T13096. Adds an optional, disabled-by-default lock log to make it easier to figure out what is acquiring and holding locks.
Test Plan: Ran `bin/lock log --enable`, `--disable`, `--name`, etc. Saw sensible-looking output with log enabled and daemons restarted. Saw no additional output with log disabled and daemons restarted.
Maniphest Tasks: T13096
Differential Revision: https://secure.phabricator.com/D19174
Summary:
See PHI413. You can pre-generate these with `bin/people profileimage --all`, but they're needlessly expensive to generate.
Streamline the workflow and cache some of the cacheable parts to reduce the generation cost.
Test Plan:
- Ran `bin/people profileimage --all` and saw cost drop from {nav 15.801s > 4.839s}.
- Set `defaultProfileImagePHID` to `NULL` in `phabricator_user.user` and purged caches with `bin/cache purge --all`.
- Loaded user directory.
- Saw default images regenerate relatively quickly.
Differential Revision: https://secure.phabricator.com/D19168
Summary:
Depends on D19150. Ref T13088. Allow clients to retrieve information about build logs, including log data, over the API.
(To fetch log data, take the `filePHID` to `file.search`, then issue a normal GET against the URI. Use a `Content-Range` header to get part of the log.)
Test Plan: Ran `harbormaster.log.search`, got sensible-looking results.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19151
Summary:
Ref T13093. See PHI396. These are possibly somewhat niche, but reasonable to support and consistent with the existing "Pusher's projects".
Also relabel "Pusher's projects" and "Project tags" for consistency and, hopefully, clarity.
Test Plan:
- Created new "commit" and "hook: commit content" Herald rules which run against "Author's projects" and "Committer's projects".
- Test console'd the "Commit" rules.
- Pushed through the "Hook" rule.
- In all cases, saw fields populate appropriately.
Maniphest Tasks: T13093
Differential Revision: https://secure.phabricator.com/D19145
Summary: Depends on D19139. Ref T13088. This doesn't actually work, but is close enough that a skilled attacker might be able to briefly deceive a small child.
Test Plan:
- Viewed some very small logs under very controlled conditions, saw content.
- Larger logs vaguely do something resembling working correctly.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19141
Summary:
Depends on D19138. Ref T13088. When we want to read the last part of a logfile //and show accurate line numbers//, we need to be able to get from byte offsets to line numbers somehow.
Our fundamental unit must remain byte offsets, because a test can emit an arbitrarily long line, and we should accommodate it cleanly if a test emits 2GB of the letter "A".
To support going from byte offsets to line numbers, compute a map with periodic line markers throughout the offsets of the file. From here, we can figure out the line numbers for arbitrary positions in the file with only a constant amount of work.
Test Plan: Added unit tests; ran unit tests.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19139
Summary: Depends on D19133. Ref T13088. Allows build logs to be formally destroyed, cleaning up their chunks and file data.
Test Plan:
- Used `bin/remove destroy` to destroy a log, verified chunks and files were removed.
- Used `bin/harbormaster rebuild-log` to force a log to rebuild, verified files were destroyed and regenerated.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19134
Summary: Depends on D19132. Ref T13088. This implements an extremely skeletal dedicated log page with a more-or-less functional "Download Log" button.
Test Plan: Downloaded a recent log. Tried to download an old (un-finalized) log, couldn't. Used `bin/harbormaster write-log` to get a convenient standalone link to a log.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19133
Summary: Depends on D19131. Ref T13088. During log finalization, stream the log into Files to support "Download Log", archive to Files, and API access.
Test Plan: Ran `write-log` and `rebuild-log`, saw Files objects generate with log content and appropriate permissions.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19132
Summary: Depends on D19130. Ref T13088. Currently, when a build log is closed we compress it in the same process. Separate this out into a dedicated worker since the plan is to do a lot more work during finalization, none of which needs to happen inline during builds (or, particuarly, inline during a Conduit call for API writes in the future).
Test Plan: Ran `bin/harbormaster write-log --trace`, saw compression run inline.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19131
Summary: Ref T13088. This is currently minimal but the modify-execute development loop on build logs is extremely long without it.
Test Plan: Ran `echo hi | ./bin/harbormaster write-log --target 12345`, saw the log show up in the web UI.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13088
Differential Revision: https://secure.phabricator.com/D19130
Summary:
Ref T13090. The default width changed recently to become much wider, but the behavior on this control isn't great. Instead:
- Pick a default width somewhere between the two.
- Make the width sticky across show/hide (pressing "f" twice remembers your width instead of resetting it).
- Make the width sticky across reloads (dragging the bar, then reloading the page keeps the bar in the same place).
Test Plan:
- Without settings, loaded page: got medium-width bar.
- Dragged bar wide/narrow, toggled on/off with "f", got persistent width.
- Dragged bar wide/narrow, reloaded page, got persistent width.
- Dragged bar wide/narrow, toggled it off, reloaded page, toggled it on, got persistent width.
Maniphest Tasks: T13090
Differential Revision: https://secure.phabricator.com/D19129
Summary: Depends on D19126. Ref T13090. For datasource custom fields, this proxies the datasource and provides "none()" and "any()" functions to allow you to search for objects with no values or any values.
Test Plan:
- Created a custom "Owning Group" field in Maniphest using a Projects datasource.
- For a task with no owner assigned, searched for "none()" (hit) and "any()" (miss).
- Assigned the task to an owning project.
- Searched for "none()" (miss), "any()" (hit), the project it is now a member of (hit) and some random other project (miss).
Maniphest Tasks: T13090
Differential Revision: https://secure.phabricator.com/D19127
Summary:
Depends on D19121. Ref T13083. Group transactions and show groups in the debugging view.
Fix some of the most obvious issues with fact generation:
- No more 0-point facts.
- Engine can now generate at least one of every type of fact.
Test Plan: Generated facts, viewed them in the debugging view, fact generation largely appeared to align with reality. No more "no facts in storage" facts.
Subscribers: yelirekim
Maniphest Tasks: T13083
Differential Revision: https://secure.phabricator.com/D19122
Summary:
Depends on D19120. Ref T13083. When you write a fact engine, it's currently somewhat difficult to figure out exactly what it's doing. It would also be difficult to diagnose bugs or report them to the upstream.
To ease this, add a page which shows all the facts an object generates. This allows you to iterate on an engine quickly without needing to reanalyze facts, take a screenshot, easily compare the timeline to the fact view, etc.
Test Plan: Viewed the object fact page for several objects.
Subscribers: yelirekim
Maniphest Tasks: T13083
Differential Revision: https://secure.phabricator.com/D19121
Summary:
Depends on D19119. Ref T13083. This is probably still very buggy, but I'm planning to build support tools to make debugging facts easier shortly.
This generates a large number of datapoints, at least, and can render some charts which aren't all completely broken in an obvious way.
Test Plan: Ran `bin/fact analyze --all`, got some charts with lines that went up and down in the web UI.
Subscribers: yelirekim
Maniphest Tasks: T13083
Differential Revision: https://secure.phabricator.com/D19120
Summary:
Ref T13083. Facts has a fair amount of weird hardcoding and duplication of responsibilities. Reduce this somewhat: no more hard-coded fact aggregates, no more database-driven list of available facts, etc. Generally, derive all objective truth from FactEngines. This is more similar to how most other modern applications work.
For clarity, hopefully: rename "FactSpec" to "Fact". Rename "RawFact" to "Datapoint".
Split the fairly optimistic "RawFact" table into an "IntDatapoint" table with less stuff in it, then dimension tables for the object PHIDs and key names. This is primarily aimed at reducing the row size of each datapoint. At the time I originally wrote this code we hadn't experimented much with storing similar data in multiple tables, but this is now more common and has worked well elsewhere (CustomFields, Edges, Ferret) so I don't anticipate this causing issues. If we need more complex or multidimension/multivalue tables later we can accommodate them. The queries a single table supports (like "all facts of all kinds in some time window") don't make any sense as far as I can tell and could likely be UNION ALL'd anyway.
Remove all the aggregation stuff for now, it's not really clear to me what this should look like.
Test Plan: Ran `bin/fact analyze` and viewed web UI. Nothing exploded too violently.
Subscribers: yelirekim
Maniphest Tasks: T13083
Differential Revision: https://secure.phabricator.com/D19119
Summary:
Depends on D19106. Fixes T5941. Ref T13077. Allows you to find Phriction documents as suggestions from global quick search.
Also supports `w` to jump to Phriction and `w query` to query Phriction.
The actual query logic for the datasource may need some tweaking after it collides with reality, but seems to produce fairly reasonable results in local testing against synthetic data.
Test Plan: Searched for "Porcupine Facts", "Travel Companions", and other useful local pages. Searched for `w`. Searched for `w zebra facts`.
Maniphest Tasks: T13077, T5941
Differential Revision: https://secure.phabricator.com/D19107
Summary:
See PHI370. Support the "Affected packages" and "Affected package owners" Herald fields in pre-commit hooks.
I believe there's no technical reason these fields aren't supported and this was just overlooked.
Test Plan: Wrote a rule which makes use of the new fields, pushed commits through it. Checked transcripts and saw sensible-looking values.
Differential Revision: https://secure.phabricator.com/D19104
Summary: Depends on D19100. Ref T13077. Adds a "content" attachment to get the actual page text. This works on both "phriction.document.search" and "phriction.content.search".
Test Plan: Called both API methods with the attachment, saw proper text content returned.
Maniphest Tasks: T13077
Differential Revision: https://secure.phabricator.com/D19103
Summary:
Depends on D19098. Ref T13077.
Phriction status constants currently use the "bag of statuses" approach typical of older code, and store integers in the database.
This fixes the "bag of statuses" stuff; a future change will fix the integers.
Also adds a skeleton for `phriction.document.search`, but doesn't implement the Conduit interface yet.
Test Plan: Searched for documents with various status constraints. Grepped for removed status constants. Viewed document list.
Maniphest Tasks: T13077
Differential Revision: https://secure.phabricator.com/D19099
Summary: Depends on D19096. Ref T13077. Adds a new "v3" API method for Phriction document content, to replace the existing "phriction.history" call.
Test Plan: Made various calls via web API console.
Maniphest Tasks: T13077
Differential Revision: https://secure.phabricator.com/D19097
Summary:
Depends on D19093. Ref T13077. Although content objects normally don't have any edges today, they may in the future.
Also implement Policy stuff properly.
Test Plan: Used `bin/remove destroy` to destroy a document, verified it also loaded and destroyed the correspoding Content correctly by looking at `--trace` and the database rows.
Maniphest Tasks: T13077
Differential Revision: https://secure.phabricator.com/D19094
Summary:
Depends on D19092. Ref T13077. This modernizes markup rendering for PhrictionContent.
This is a little messy because table of contents generation isn't straightforward.
Test Plan: Viewed Phriction documents with and without 3+ headers, saw ToC vs no ToC. Edited/previewed documents. Grepped for affected symbols. Checked DarkConsole for sensible cache behavior.
Maniphest Tasks: T13077
Differential Revision: https://secure.phabricator.com/D19093
Summary: Ref T13077. Prepares for modern API access to document history using standard "v3" APIs.
Test Plan: Ran migration, verified PHIDs appeared in the database. Created/edited a document, got even more PHIDs in the database.
Maniphest Tasks: T13077
Differential Revision: https://secure.phabricator.com/D19092
