Summary:
Ref T13603. On common edit pathways, extract explicit file attachments from Remarkup. These pathways are affected:
- Objects that use EditEngine and expose a remarkup area via "RemarkupEditField".
- Objects that use EditEngine to generate a comment area.
This is the vast majority of pathways, but not entirely exhaustive.
Test Plan: Created and commented on a task, explicitly attaching images. Saw images attach properly.
Maniphest Tasks: T13603
Differential Revision: https://secure.phabricator.com/D21830
Summary:
Ref T13603. This removes the file attachment edge types, which no longer have readers or writers.
It does not delete the underlying data.
This indirectly removes API access to this edge. As far as I know, this was only used by one customer (see D21480) who has migrated away from Phabricator.
Test Plan: Grepped for edge constants.
Maniphest Tasks: T13603
Differential Revision: https://secure.phabricator.com/D21823
Summary: Ref T13603. Removes all reads and writes to the other half of the legacy edge.
Test Plan:
- Verified I could still see file content accessible only through object attachment.
- This fixes a unit test broken by removal of only half the edge access in the previous change.
Maniphest Tasks: T13603
Differential Revision: https://secure.phabricator.com/D21820
Summary: Ref T13603. Migrate all code which interacts with the "ObjectHasFile" edge to use the "Attachments" table instead.
Test Plan:
- Edited a paste's view policy, confirmed associated file secret was scrambled.
- Verified I could still view paste content as a user who could not naturally view the underlying file.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13603
Differential Revision: https://secure.phabricator.com/D21819
Summary: Ref T13603. Copy all existing file attachment data into the new storage table.
Test Plan: Ran migration, checked attachment table for general reasonableness.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13603
Differential Revision: https://secure.phabricator.com/D21818
Summary: Ref T13603. New transactional file attachment extracts PHIDs earlier than the older mechanism did. Allow either pathway to save content and extract a PHID.
Test Plan:
- Created and modified a paste.
- Confirmed both pathways do old and new storage writes.
Maniphest Tasks: T13603
Differential Revision: https://secure.phabricator.com/D21817
Summary: Ref T13603. This adds a second write to new "attachment" storage to all writers except one in Paste, which creates the file inline.
Test Plan:
- Updated a macro image, confirmed a write to "attachment" storage (transaction pathway).
- Updated a blog profile image, confirmed a write to "attachment" storage (legacy pathway).
Maniphest Tasks: T13603
Differential Revision: https://secure.phabricator.com/D21816
Summary:
Ref T13603. Currently, files are sometimes detached from objects. For example, when you change the image for a Macro, the old image is detached.
This is wrong: the image should remain attached so users who can view the macro can view the complete "alice change the image from X to Y" transaction. To be able to understand the change that was applied, you need to be able to view both files.
All workflows which currently detach files aren't conistent with the modern way applications behave, except maybe one callsite in a unit test, and that one's kind of moot.
Get rid of this stuff and just use PHID extraction to perform file attachment in all cases.
Test Plan: Created and edited macros, verified files were properly attached and remained attached across edits.
Maniphest Tasks: T13603
Differential Revision: https://secure.phabricator.com/D21815
Summary: Ref T13603. Prepare to move this relationship out of edge storage into dedicated storage so it is easier to formalize better in the UI.
Test Plan: Ran `bin/storage upgrade`.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13603
Differential Revision: https://secure.phabricator.com/D21813
Summary: Ref T13603. When processing transactions that include remarkup blocks, stop extracting file references from them for attachment.
Test Plan: Submitted a remarkup block with a file reference, no longer saw automatic file attachment.
Maniphest Tasks: T13603
Differential Revision: https://secure.phabricator.com/D21812
Summary: Ref T13588. This allows "arc unit --everything" to more or less run cleanly on PHP 8.1.
Test Plan: Ran "arc unit --everything".
Maniphest Tasks: T13588
Differential Revision: https://secure.phabricator.com/D21821
Summary:
Ref T13603. Allow "VersionedDraft" to persist remarkup comment area metadata from stacked actions controls.
When files are dragged and dropped, record them as explicit uploads in comment metadata.
Test Plan: Dragged and dropped files into Remarkup stacked action text areas (e.g., in Maniphest), reloaded page, saw metadata persist across reloads.
Maniphest Tasks: T13603
Differential Revision: https://secure.phabricator.com/D21828
Summary: Ref T13603. This will support explicit handling of attached files.
Test Plan: Adjusted new input to have "text" input type, used it alongside additional upcoming changes, saw sensible metadata behavior.
Maniphest Tasks: T13603
Differential Revision: https://secure.phabricator.com/D21827
Summary: Ref T13680. Conpherence may pass values with an integer type to this layer of the stack. These are "supposed" to be strings, but just be accepting.
Test Plan:
- Wrote 100+ messages to a Conpherence room.
- Clicked "Show Older Messages".
- Before: exception, int passed to "phutil_nonempty_string()".
- After: older messages loaded.
Maniphest Tasks: T13680
Differential Revision: https://secure.phabricator.com/D21824
Summary:
Ref T13681. This was introduced in D593, never used, and doesn't make sense or have any uses in modern Phabricator.
It also does some pretty direct writes that can technically do things that at least //look// like they violate policies, so remove it.
Test Plan:
- Checked the API console, no longer saw "feed.publish".
- Grepped for "feed.publish", no hits.
Maniphest Tasks: T13681
Differential Revision: https://secure.phabricator.com/D21826
Summary:
Ref T13679. In D16983, global settings objects were given an exception to let logged-out users see them, even on installs with no "public" user role.
This exception is too broad and grants everyone all capabilities, not just "CAN_VIEW". In particular, it incorrectly grants "CAN_EDIT", so any user can edit global settings defaults.
Restrict this grant to "CAN_VIEW".
Test Plan:
- As a non-administrator, tried to edit global settings.
- Before: could.
- After: could not.
Maniphest Tasks: T13679
Differential Revision: https://secure.phabricator.com/D21811
Summary: Ref T13678. When remarkup content embeds other remarkup content, detect and degrade if the references have nesting depth greater than 1. This is a coarse cycle detector, since rendering shallow (but technically non-cycling) trees doesn't seem valuable.
Test Plan: Created various objects with self-references, saw everything degrade properly (after one level of embedding) when embedded in itself and in other contexts. See attached screenshot.
Maniphest Tasks: T13678
Differential Revision: https://secure.phabricator.com/D21810
Summary: Ref T13677. This was an accidental change in D21807: when reclaiming a resource, wait for it to be completely destroyed before allowing a lease to reclaim another resource.
Test Plan: Reverts accidental behavior change in D21807.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13677
Differential Revision: https://secure.phabricator.com/D21809
Summary:
Ref T13677. Drydock has a hard-coded and fairly arbitrary limit which prevents a resource pool from growing more than 25% at once.
This is vaguely reasonable for resources which allocate quickly, but suffocating for slower resources. It's also wholly arbitrary, and the "one per lease" limit introduced in D21807 should do a better job of covering the same ground while generally being more reasonable, expected, and predictable.
Test Plan: Ran Drydock allocations without the throttle, saw faster pool growth.
Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13677
Differential Revision: https://secure.phabricator.com/D21808
Summary:
Ref T13677. Currently, one lease may cause multiple resources to allocate simultaneously if it starts allocating one, then wakes up from a yield later on and still sees no available resources.
This is never desired -- or, at least, produces desirable behavior only entirely by accident. Normally, it causes an excess of resources to allocate.
This is not a catastrophic problem: the extra resources usually get used sooner or later or cleaned up; and the total amount of badness is limited by overall resource allocation limits.
However, this behavior is also suppressed by an artificial "25% of current pool size" growth limit throttle which I intend to remove. Removing this throttle without fixing the allocator behavior could make this "too many resources" problem worse.
Change the allocator so that a lease that has started allocating a resource won't allocate another resource until the first resource leaves the "pending" state.
This also fixes some general oddness with the allocator and attempts to simplify the structure.
Test Plan:
- Ran 8 taskmasters.
- Destroyed all resources and leases.
- Leased 4 working copies.
- Saw exactly 4 resources build and lease, all simultaneously.
- Destroyed all resources and leases.
- Leased 32 working copies.
- Saw exactly 32 resources build and lease, approximately 8 at a time (limited by taskmasters).
- Destroyed all leases (but not resources).
- Leased 32 working copies, saw them satisfied by existing resources.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13677
Differential Revision: https://secure.phabricator.com/D21807
Summary:
Ref T13677. Track which resources a given lease has begun allocating or reclaiming in a more formal way, and add logging for waiting actions.
The "allocating" mechanism is new. This will replace an existing similar "reclaiming" mechanism in a future change.
Test Plan: See followup changes.
Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13677
Differential Revision: https://secure.phabricator.com/D21806
Summary: Ref T13677. These flags increase the convenience of testing in a development environment.
Test Plan: Used new "--all" flags to release all resources and leases.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13677
Differential Revision: https://secure.phabricator.com/D21805
Summary:
Ref T13676. The 3-minute grace period when a resource can not be reclaimed after its leases are released currently doesn't work reliably because the Resource object usually isn't actually updated when a lease is released.
Add an additional check for recently-destroyed leases, and extend the grace period if we find any.
Test Plan:
- See T13676. Ran reproduction sequence there, observed immediate resource reclamation.
- Applied patch.
- Ran sequence again, observed repository B wait 3 minutes to reclaim a repository A resource.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13676
Differential Revision: https://secure.phabricator.com/D21803
Summary:
Ref T13676. Currently, "bin/drydock lease" just creates a lease that permits any blueprint.
To prepare for "use specific blueprint X", unify the logic between this workflow and LeaseUpdateWorker so we select only blueprints which at least have coarse compatibility (e.g., if we're leasing a host, only select enabled blueprints of classes that can allocate hosts).
Test Plan: Used `bin/drydock lease` to try to lease a nonsense type, got sensible error. Leased a host.
Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13676
Differential Revision: https://secure.phabricator.com/D21801
Summary: Ref T13676. This makes it easier to create resource pressure without juggling a big pile of terminals.
Test Plan: Used `bin/drydock lease --count 5 ...` to acquire 5 leases.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13676
Differential Revision: https://secure.phabricator.com/D21800
Summary: Ref T13676. When the required "repositories.map" attribute is omitted, `bin/drydock lease` currently fatals in an unhelpful way when trying to lease a working copy.
Test Plan:
Ran `bin/drydock lease --type working-copy` with no attributes, after following steps in T13676.
```
<Allocation Failed> One or more blueprints promised a new resource, but failed when allocating: [PhutilAggregateException] All blueprints failed to allocate a suitable new resource when trying to allocate lease ("PHID-DRYL-orbtwtlinksm3xqpyhmw").
- Exception: Working copy lease is missing required attribute "repositories.map".
Attribute "repositories.map" should be a map of repository specifications.
```
Subscribers: yelirekim, PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13676
Differential Revision: https://secure.phabricator.com/D21796
Summary: Ref T13676. Ref T13588. Fix some issues that prevent "bin/phd" and "bin/drydock" from executing under PHP 8.1, broadly because `null` is being passed to `strlen()`.
Test Plan: Ran `bin/phd debug task` and `bin/drydock ...` under PHP 8.1.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13676, T13588
Differential Revision: https://secure.phabricator.com/D21795
Summary:
Ref T5479. Ref T13658. This was a contributed application from the early days of Phabricator which never had customers or users in the wild. The contributor moved on from the project many years ago.
Any capabilities in this general role would look different today. It also has one or two product name literal strings, so this is as good a time as any to remove it.
This change does not remove storage; I'll issue upgrade guidance and do that separately after some time.
Test Plan: Grepped for "phragment", got no relevant hits.
Subscribers: hach-que, PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13658, T5479
Differential Revision: https://secure.phabricator.com/D21793
Summary:
Ref T9530. Ref T13658. The "Releeph" application was never useful outside of Facebook and any application providing release support would not resemble it much.
It has some product name literal strings, so now is as good a time as any to get rid of it.
This application never left prototype and I'm not aware of any install in the wild that uses it (or has ever used it).
I did not destroy the database itself. I'll issue upgrade guidance and destroy the database in some future release, just in case.
Test Plan: Grepped for "releeph", found no relevant/removable hits.
Subscribers: PHID-OPKG-gm6ozazyms6q6i22gyam
Maniphest Tasks: T13658, T9530
Differential Revision: https://secure.phabricator.com/D21792
Summary: Ref T13658.
Test Plan: Static checks only, these are all reachable only by breaking your install in weird ways.
Maniphest Tasks: T13658
Differential Revision: https://secure.phabricator.com/D21781
Summary: Ref T13658.
Test Plan: Static checks only, these are mostly obscure or require breaking the install in unusual ways.
Maniphest Tasks: T13658
Differential Revision: https://secure.phabricator.com/D21780
Summary: Ref T13658.
Test Plan:
This is non-exhaustive.
- Started a JIRA auth provider creation workflow.
- Same for LDAP.
Maniphest Tasks: T13658
Differential Revision: https://secure.phabricator.com/D21779
Summary: Ref T13658.
Test Plan:
This is non-exhuastive.
- Poked around "Auth": edited a provider, toggled a provider.
Maniphest Tasks: T13658
Differential Revision: https://secure.phabricator.com/D21778
Summary: Ref T13658. This one is a little silly since I used `split` to divide the list of files up, and only ended up with one file in the last partition.
Test Plan: Viewed Phabricator.
Maniphest Tasks: T13658
Differential Revision: https://secure.phabricator.com/D21774
Summary: Ref T13658.
Test Plan:
This test plan is non-exhaustive.
- Ran `bin/storage databases`.
- Viewed Badges UI exmaples page.
- Used eval rule for `strings.platform.server.name`, got "Phabricator".
Maniphest Tasks: T13658
Differential Revision: https://secure.phabricator.com/D21773
Summary: Ref T13658.
Test Plan:
This test plan is non-exhaustive.
- Viewed "remarkup.process" Conduit method API page.
- Viewed URIs in a Diffusion repository.
- Viewed editor protocol configuration in Settings.
Maniphest Tasks: T13658
Differential Revision: https://secure.phabricator.com/D21772
