Summary:
Ref T13625. See that task for discussion.
Currently, the Viewer when performing revision updates in response to commits may be an arbitrary low-privilege user (an Application, a disabled User, a bot, a mailing list, etc).
Today, this leads to an exception when trying to make API calls.
Ideally, we probably would not perform the update in these cases. However, performing the update isn't a policy violation and is generally less surprising than not performing it, so continue performing it for now: just use the omnipotent user to interact with the API.
Test Plan:
- Authored a commit as a bot user without permission to view the repository or revision.
- Commented out a couple of caches, and used `bin/repository reparse --publish ...` to republish the commit.
- Before: exception when trying to interact with the API.
- After: clean publish.
Maniphest Tasks: T13625
Differential Revision: https://secure.phabricator.com/D21582
Summary:
Ref T13623. When paginating notifications, we may currently construct a query which:
- loads from non-unique rows; and
- returns multiple results.
In particular, `chronologicalKey` isn't unique across the whole table (only for a given viewer). We can get away with this because no user-facing view of notifications is truly "every notification for every viewer" today.
One fix would be to implicitly force the paging query to include `withUserPHIDs(viewerPHID)`, but puruse a slightly more general fix:
- Load only unique stories.
- Explictly limit the pagination subquery to one result.
Test Plan:
- Set page size to 1, inserted duplicate notifications of all stories for another user, clicked "Next", got the GROUP BY error.
- Applied the "only load unique stories" part of the change, got a "expected one row" error instead.
- Applied the "limit 1" part of the change, got a second page of notifications.
Maniphest Tasks: T13623
Differential Revision: https://secure.phabricator.com/D21577
Summary: Ref T13624. Depends on D21578. In "sshd" subprocess contexts, use "PhutilErrorLog" to direct errors to both stderr and, if configured, a logfile on disk.
Test Plan:
- Confiugured an error log.
- Forced `ssh-auth` to fatal.
- Saw errors on stderr and in log.
Maniphest Tasks: T13624
Differential Revision: https://secure.phabricator.com/D21579
Summary: Ref T13611. This property worked correctly when implemented in D19357. The behavior was broken by D20775, which tested node-level routing but did not specifically re-test the "writable" property. This was difficult to spot because ref query outcomes weren't observable in the UI, and the ref itself had the correct property value.
Test Plan:
See D21575. After this change, the UI shows the correct state, rather than showing a read-only service ref as writable:
{F8465865}
Maniphest Tasks: T13611
Differential Revision: https://secure.phabricator.com/D21576
Summary: Ref T13611. Currently, the "writable" property on service bindings has no effect because of a trivial bug. Provide more information in the UI to make this kind of problem observable.
Test Plan:
Viewed "Storage" section of management UI, saw a more-obvious problem with ref management (a non-writable ref is listed as writable).
{F8465851}
Maniphest Tasks: T13611
Differential Revision: https://secure.phabricator.com/D21575
Summary:
Ref T13620.
- Make generic edge stories render links with hovercards. Other story types (like subscriptions) already do this so I'm fairly certain this is just old code from before hovercards.
- Include a longer commit message snippet in hovercards.
Test Plan: {F8465645}
Maniphest Tasks: T13620
Differential Revision: https://secure.phabricator.com/D21574
Summary:
Ref T13617. When an inline comment is added inside a block of added lines, it currently ends up off-by-one when porting forward.
This is a disagreement between the mapping engine and the display engine about what "offset" means. Choose the simpler of the two interpretations.
Test Plan:
- Created a revision with the diff in T13617.
- Added an inline in the middle of the added block.
- Updated the revision with the same diff.
- Before: inline incorrectly moves up by one line.
- After: inline maps correctly.
Maniphest Tasks: T13617
Differential Revision: https://secure.phabricator.com/D21572
Summary: Ref T13586. In the footsteps of D21563, make Herald rule results more formal and structured to support meaningful exception reporting.
Test Plan:
Ran various Herald rules and viewed transcripts, including rules with recursive dependencies and condition exceptions.
{F8447894}
Maniphest Tasks: T13586
Differential Revision: https://secure.phabricator.com/D21565
Summary: Ref T13586. Lift the behavioral core of "HeraldConditionResult" into a new abstract base "HeraldTranscriptResult", with the intent to introduce a "HeraldRuleResult".
Test Plan:
- Ran Herald rules, reviewed transcripts.
- This change should have no behavioral effect.
Maniphest Tasks: T13586
Differential Revision: https://secure.phabricator.com/D21564
Summary:
Ref T13586. Currently, Herald condition logs encode "pass" or "fail" robustly, "forbidden" through a sort of awkward side channel, and can not properly encode "invalid" or "exception" outcomes.
Structure the condition log so results are represented unambiguously and all possible outcomes (pass, fail, forbidden, invalid, exception) are clearly encoded.
Test Plan:
{F8446102}
{F8446103}
Maniphest Tasks: T13586
Differential Revision: https://secure.phabricator.com/D21563
Summary: Ref T13586. The Herald transcript page has become more and more complicated over time, and recently added "Transactions" and "Profiler" sections. Split these across separate navigation tabs to limit the maximum complexity of any single view and make it easier to navigate to particular sections, like the profiler section.
Test Plan: Viewed various transcripts, saw nice digestible sections.
Maniphest Tasks: T13586
Differential Revision: https://secure.phabricator.com/D21493
Summary: Ref T13615. This property was removed from the Facebook API at some point, perhaps November 2020. Stop relying no it.
Test Plan: Created a local Facebook OAuth app, registered a new account locally.
Maniphest Tasks: T13615
Differential Revision: https://secure.phabricator.com/D21571
Summary: Ref T13613. Improve the performance of this migration by using a temporary table and an "UPDATE x JOIN y ..." pattern.
Test Plan:
- Ran on `secure`, got exit after a few seconds since the migration is idempotent and changesets already had PHIDs.
- Ran on `secure` with the `continue;` commented out, got valid new PHIDs in 53s (from 153s).
- Tried a larger page size (16K), didn't see any improvement.
- From "--trace", client PHID generation seems to be the limiting factor.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13613
Differential Revision: https://secure.phabricator.com/D21570
Summary: Ref T13609. Add the Object PHID (object being built), Container PHID (container of the object being built), Build PHID, and Buildable PHID to Harbormaster build variables.
Test Plan:
{F8448191}
{F8448192}
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13609
Differential Revision: https://secure.phabricator.com/D21569
Summary:
Ref T13608. When searching for bare URIs in remarkup text, don't look for URIs with a protocol string longer than 32 characters.
This avoids a case where the regexp engine may be tricked into executing at `O(N^2)` or some similar complexity.
Test Plan:
- Applied remarkup to "AAAA..." (512KB).
- Before: 64 seconds to process.
- After: <10ms to process.
- Ran unit tests.
Maniphest Tasks: T13608
Differential Revision: https://secure.phabricator.com/D21562
Summary:
Ref T13587. D21495 has significant changes to the ngram indexer, which might possibly contain bugs.
Make it easier to reindex a subset of documents (based on the date when the index was built, and/or the software version which generated the index).
This is in addition to the existing versioning, which is focused on object versions.
Test Plan: Ran `bin/search index` with various old and new arguments. Spot-checked the `IndexVersion` table.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13587
Differential Revision: https://secure.phabricator.com/D21560
Summary:
Ref T13587. Currently, when a document is reindexed by Ferret, the old document is completely discarded and a new version is inserted to replace it.
This approach is simple to implement, but can lead to exhaustion of the ngram AUTO_INCREMENT id column in reasonable circumstances.
Conceptually, this approach "should" be fine and this exhaustion is an awkard implementation detail. However, since it's easy to be less wasteful when performing document updates and all the other approaches are awkward or leaky in other ways that are probably worse, use a more complex implementation to avoid executing unnecessary INSERT statements.
Test Plan:
- Created and indexed a new document, searched for it.
- Updated a document, indexed it with `bin/search index ... --force --trace`, saw only modifications updated in the index.
- Searched for newly added terms (got hits) and removed terms (no longer got hits) to verify add/delete index behavior.
Maniphest Tasks: T13587
Differential Revision: https://secure.phabricator.com/D21495
Summary: Ref T13607. Add some time-oriented constraints to this API method to support compiling build statistics.
Test Plan:
- Called "harbormaster.target.search" with all new constraints.
- Viewed documentation in API console.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13607
Differential Revision: https://secure.phabricator.com/D21559
Summary: Ref T13605. Support selecting a diff's changesets (to get a list of affected file paths) via the API.
Test Plan: Called API with no arguments, diffPHIDs, PHIDs, IDs. Got sensible output.
Maniphest Tasks: T13605
Differential Revision: https://secure.phabricator.com/D21558
Summary:
Ref T13605. Changesets currently have no PHID, which limits their ability to use standard API infrastructure.
Give them a PHID, since there's no reason they don't have one other than their age.
Test Plan:
- Ran migrations, saw PHIDs populated.
- Created new changesets, saw PHIDs.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13605
Differential Revision: https://secure.phabricator.com/D21557
Summary: Ref T13602. Warn when a reivison has at least one human reviewer, no non-human reviewers, and no human reviewers can view it.
Test Plan: {F8430683}
Maniphest Tasks: T13602
Differential Revision: https://secure.phabricator.com/D21556
Summary: Ref T13602. Similar to subscriber and mention treatments, make it clear when a user doesn't have view permission.
Test Plan: {F8430595}
Maniphest Tasks: T13602
Differential Revision: https://secure.phabricator.com/D21555
Summary:
Ref T13602. When rendering a user hovercard, pass the object on which the reference appears. If the user can't see the object, make it clear on the hovecard.
Restyle the "nopermission" markup in mentions to make it more obvious what the style means: instead of grey text, use red with an explicit icon.
Test Plan: {F8430398}
Maniphest Tasks: T13602
Differential Revision: https://secure.phabricator.com/D21554
Summary:
Ref T13602. Currently, Hovercards are functions only of the object they represent (and the viewer, etc).
Recent changes to how users who can't see an object are rendered motivate making them a function of both the object they represent //and// the context in which they are being viewed. In particular, this enables a hovecard for a user to explain "This user can't see the thing you're lookign at right now.", so visual "exiled" markers can have a path forward toward discovery.
Test Plan:
- This change isn't expected to affect any behavior.
- Viewed hovercards, moused over/out, resized windows, viewed standalone cards, viewed debug cards, saw no behavioral changes.
Maniphest Tasks: T13602
Differential Revision: https://secure.phabricator.com/D21553
Summary:
Ref T13602. Currently, timeline comment rendering does not (by default) propagate the context object to the rendering layer.
This means that `@mentions` of users who can't see the object aren't rendered properly (currently: they show up as blue, but should show up as grey).
Pass the context down the stack and into the remarkup engine.
Test Plan: {F8382905}
Maniphest Tasks: T13602
Differential Revision: https://secure.phabricator.com/D21548
Summary:
Ref T13602. When a subscriber can't see an object, it's currently hard to figure it out.
Show this status clearly in the curtain UI.
Test Plan: {F8382865}
Maniphest Tasks: T13602
Differential Revision: https://secure.phabricator.com/D21547
Summary:
Ref T13602. Currently, the policy framework can not execute "test if many users can see one object" particluarly efficiently. This test must be executed more broadly to implement the changes in T13602.
To avoid making this any worse than it already is, lift this block into a wrapper class that has a bulk queue + fetch API and could eventually be optimized.
Test Plan: Viewed a task with an `@mention` of a user without permission to see it in the summary, saw it rendered in a disabled style.
Maniphest Tasks: T13602
Differential Revision: https://secure.phabricator.com/D21546
Summary:
Ref T13395. Libphutil has merged into Arcanist and no longer needs to be installed or upgraded. Additionally:
- The minimum PHP version is now PHP 5.5.
- Although older versions of PHP should still install APC, modern versions come with Opcache and do not need APC. Setup issues guide administrators thorugh the correct install procedure now.
Test Plan: Read documentation.
Maniphest Tasks: T13395
Differential Revision: https://secure.phabricator.com/D21550
Summary:
Ref T13600. When migrating observed repositories between cluster services, impact can be better controlled by fetching a copy of the repository on the target host before clusterizing it.
In particular, in the Phacility cluster, migrations are generally from one shared shard to one dedicated shard. It's helpful to perform these migrations synchronously without waiting for the cluster to sync in the background (helpful in the sense that there are fewer steps and fewer commands to run).
This supports an "--observe" mode to the internal "bin/services load-repository" workflow, which transfers repository data by refetching it from the remote rather than by getting it from the older host. This fetch occurs before cluster configuration is adjusted.
Test Plan: Ran locally as a sanity check, will apply in production.
Maniphest Tasks: T13600
Differential Revision: https://secure.phabricator.com/D21544
Summary:
See <https://discourse.phabricator-community.org/t/i-cant-create-almanac-space/4424/>.
Almanac namespaces have never really had property support, but they implemented the interface in the original implementation.
At the time, this had no effect. Later changes integrated properties into the edit flows and broke this no-op integration.
Remove the interface for now. They could be given property support later, but need a bit of support code.
This feature is very rarely used and primarily useful for Phacility instances.
Test Plan: Created new namespaces and edited namespaces, browsed namespace UI.
Differential Revision: https://secure.phabricator.com/D21543
Summary:
Ref T13588. This has never been meaningful, but a "final private" method is specifically forbidden in PHP8.
Remove meaningless "final" from these methods, per new lint checks.
Test Plan: Ran `arc lint --everything` to identify affected methods, then `... | xargs -n1 arc lint --apply-patches`.
Maniphest Tasks: T13588
Differential Revision: https://secure.phabricator.com/D21540
Summary: Ref T13591. Provide some guidance on the most common cases for wanting to interact with the worker queue.
Test Plan: Read documentation.
Maniphest Tasks: T13591
Differential Revision: https://secure.phabricator.com/D21536
Summary: Ref T13591. Support delaying selected tasks until a later time and bulk-adjustment of task priority.
Test Plan: Ran `bin/worker delay` and `bin/worker priority` to delay and reprioritize tasks. Confirmed outcomes with daemon console.
Maniphest Tasks: T13591
Differential Revision: https://secure.phabricator.com/D21535
Summary:
Ref T13591. Add more selector flags to let "bin/worker" commands operate on tasks by container PHID, object PHID, priority, etc.
This anticipates adding "bin/worker reprioritize" and "bin/worker delay" workflows, to provide more tools for handling repository imports.
Test Plan:
- Ran `bin/worker execute`, `cancel`, `retry`, and `free` with various sets of selector flags.
- Used `--min-priority`, `--max-priority`, `--object`, `--container`, `--archived`, `--max-failure-count` to select tasks.
- Specified invalid, duplicate, aliased objects with "--object".
- Specified invalid range priority selectors.
Maniphest Tasks: T13591
Differential Revision: https://secure.phabricator.com/D21534
Summary: Ref T13591. This is a minor consistency change to use PHIDs instead of IDs in the commit import processing pipeline. PHIDs are generally more powerful in more contexts and it would be unusual for a modern worker to use an ID here.
Test Plan:
- Made the "accept either ID or PHID" part of the change only.
- Pushed a commit, parsed and reparsed it step by step (this tests that "commitID" tasks can still process normally).
- Made the "write PHIDs" part of the change.
- Pushed a commit, parsed and reparsed it step by step.
- Looked at the task row in the database, saw PHID data.
Maniphest Tasks: T13591
Differential Revision: https://secure.phabricator.com/D21533
Summary:
Ref T13591. Improve how parameters are passed between commit worker tasks:
- Always pass "via", to track where tasks came from.
- Always provide "objectPHID" (with the commit PHID).
- Always provide "containerPHID" (with the repository PHID).
Test Plan:
- Pushed a new commit.
- Ran `bin/repository pull` + `bin/repository discover`, saw commit with all parameters.
- Ran `bin/worker execute ...`, saw a Change worker and then a Publish worker with appropriate parameters.
- Ran `bin/repository reparse ... --background`, saw workers queue with appropriate parameters.
Maniphest Tasks: T13591
Differential Revision: https://secure.phabricator.com/D21532
Summary:
Ref T13591. Worker queue tasks which affect commits currently (mostly) store the commit as an "objectPHID", but do not directly reference the repository the commit belongs to.
This can make certain operations (like "change the priority of all tasks affecting repository Y") more difficult than it needs to be.
Support a "containerPHID", similar to the field of the same name on builds, that can store a parent object like a repository and better support operations against subsets of tasks.
See also D11044 for the genesis of "objectPHID".
This depends on the introduction of storage patch phases (in D21529) so that earlier migrations which queue worker tasks don't try to insert this column before it actually exists.
Test Plan:
- Ran `bin/storage upgrade`.
- No callers yet, see further changes for usage.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13591
Differential Revision: https://secure.phabricator.com/D21531
Summary:
Ref T13591. Some storage patches queue worker tasks, currently always to rebuild search indexes.
These patches can not execute in creation order if a later patch modifies the worker task table, since they'll try to perform a modern INSERT against an out-of-date table schema. Such a modification is desirable in the context of T13591, but making it causes these patches to fail.
Patches have an existing "after" mechanism which allows them to have explicit dependencies. This mechanism could be used to resolve this issue, but all patches with a dependency like this would need to be updated every time the queue table changes.
Instead, introduce "phases" to provide broader ordering rules. There are now two phases: "default" and "worker". Patches in the "worker" phase execute after patches in the "default" phase.
Phases may eventually be further separated, but
Test Plan:
- Ran `bin/storage status`, saw patches annotated with phases.
- Will apply `containerPHID` changes on top of this.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13591
Differential Revision: https://secure.phabricator.com/D21529
Summary:
Ref T13596. See that task for discussion. Executing "INSERT ... SELECT" at default isolation levels requires more locking than executing "SELECT" + "INSERT" separately.
Decompose this "INSERT ... SELECT" into "SELECT + INSERT", and reformat it to execute a minimal set of changes instead of wiping everything out and then writing all of it back. In most cases, this means we write 1 row instead of `O(number of project members)` rows.
Test Plan:
- Created a project. Added and removed members, looked at database and saw a consistent membership/materialization list.
- Created a subproject. Added and removed members, looked at database and saw a consistent membership/materialization list.
I wasn't successful in reproducing the LOCK WAIT issue locally by trying various concurrent SELECT / INSERT / INSERT ... SELECT strategies. It may depend on the "DELETE + INSERT ... SELECT" structure used here, or versions/config/etc, so we'll have to see how that fares in production.
Maniphest Tasks: T13596
Differential Revision: https://secure.phabricator.com/D21527
Summary:
See PHI1983. Ref T13599. Ref T13589. Currently, if you browse to a path browse URI in Diffusion without a trailing slash (`/browse/master/src`), you get a nonsensical view (the directory as a single item).
Be more precise in how "git ls-tree" arguments are constructed.
Test Plan: Visited files and directories in the browse view, with and without trailing slashes. Saw improved behavior for directories with no trailing slash and reasonable behavior in all other cases.
Maniphest Tasks: T13599, T13589
Differential Revision: https://secure.phabricator.com/D21528
Summary:
Ref T13590. By default, PHP kills execution after web scripts run for 30 seconds. If this occurs in the locked section of a repository write while we're holding the durable write lock, the lock will get stuck.
Use "set_time_limit(0)" to prevent this mechanism from interrupting execution while the durable lock is held.
Test Plan:
- Added "set_time_limit(1)" before the lock and "while (1);" in the critical section of the lock.
- Pushed, got the lock stuck.
- Cleared the lock, applied this patch, pushed.
- Got an infinite hang instead. (Normally, we expect the script to take more than 30 seconds to execute because there is a large push that executes in finite time, not because there's an infinte loop.)
Maniphest Tasks: T13590
Differential Revision: https://secure.phabricator.com/D21526
Summary:
Ref T13590. Currently, errors arising from cluster locking (like the "stuck write lock" exception) are not caught and converted into VCS responses on the HTTP VCS workflow.
Catch a broader range of exceptions and convert them into appropriate responses.
Test Plan:
- Forced a "stuck write lock" exception, pushed to a Git repository over HTTP.
- Before: generic fatal.
- After: VCS-specific fatal with a useful message in the "X-Phabricator-Message" response header.
Maniphest Tasks: T13590
Differential Revision: https://secure.phabricator.com/D21525
Summary:
Ref T13590. User objects have some inline caches that don't do readthrough generation by default because it may be indicative of high-impact performance problems in code.
During a VCS request, these caches are normally unnecessary, but they may be hit on some unusual pathways (like error handling).
Flag VCS users as okay for inline generation. This does not indicate a performance problem and access to these caches is very rare, at least today.
Test Plan:
- Executed a Git HTTP request which hit an unhandled exception (stuck write lock).
- Before: got a second-level exception while handling the first exception, when trying to access user preferences to render a standard uncaught exception page.
- After: no second-level exception.
Maniphest Tasks: T13590
Differential Revision: https://secure.phabricator.com/D21524
Summary:
Ref T13590. Currently, when you encounter a HTTP error in Git, there is no apparent way to make the client show any additional useful information. In particular, the response body is ignored.
We can partially get around this by putting the information in an "X-Phabricator-Message: ..." HTTP header, which is visible with "GIT_CURL_VERBOSE=1 git ...". Users won't normally know to look here, but it's still better than nothing.
Test Plan:
- Ran "GIT_CURL_VERBOSE=1 git fetch" against a Phabricator HTTP URI that returned a HTTP/500 error.
- Before: no clue what happened on the client.
- After: client shows useful message in the "X-Phabricator-Message" header in debug output.
Maniphest Tasks: T13590
Differential Revision: https://secure.phabricator.com/D21523
Summary:
Ref T13593. The commit cache in this Engine has a maximum fixed size (currently 65,535 entries).
If we execute discovery in a repository with more refs than this (e.g., 180K), we get fast lookups for the first 65,535 refs and slow lookups for the remaining refs.
Instead, divide the refs into chunks no larger than the cache size, and perform an explicit cache fill before each chunk is processed.
Test Plan:
- Created a repository with 1K refs. Set cache size to 256. Ran discovery.
- Before patch: saw one large cache fill and then ~750 single-gets.
- After patch: saw four large cache fills.
- Compared `bin/repository discover ... --verbose` output before and after patch for overall effect; saw no differences.
Maniphest Tasks: T13593
Differential Revision: https://secure.phabricator.com/D21521
