Summary: Ref T10537. Minor updates to simplify and modernize these codepaths.
Test Plan: Searched for queues and sources.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10537
Differential Revision: https://secure.phabricator.com/D15429
Summary:
Ref T10449. Currently, we store classes (like "AlmanacClusterRepositoryServiceType") in the database.
Instead, store types (like "cluster.repository").
This is a small change, but types are a little more flexible (they let us freely reanme classes), a little cleaner (fewer magic strings in the codebase), and a little better for API usage (they're more human readable).
Make this minor usability change now, before we unprototype.
Also make services searchable by type.
Also remove old Almanac API endpoints.
Test Plan:
- Ran migration, verified all data migrated properly.
- Created, edited, rebound, and changed properties of services.
- Searched for services by service type.
- Reviewed available Conduit methods.
Reviewers: chad
Reviewed By: chad
Subscribers: yelirekim
Maniphest Tasks: T10449
Differential Revision: https://secure.phabricator.com/D15346
Summary:
Fixes T6741. Ref T10246. Broadly, we want to protect Almanac cluster services:
- Today, against users in the Phacility cluster accidentally breaking their own instances.
- In the future, against attackers compromising administrative accounts and adding a new "cluster database" which points at hardware they control.
The way this works right now is really complicated: there's a global "can create cluster services" setting, and then separate per-service and per-device locks.
Instead, change "Can Create Cluster Services" into "Can Manage Cluster Services". Require this permission (in addition to normal permissions) to edit or create any cluster service.
This permission can be locked to "No One" via config (as we do in the Phacility cluster) so we only need this one simple setting.
There's also zero reason to individually lock //some// of the cluster services.
Also improve extended policy errors.
The UI here is still a little heavy-handed, but should be good enough for the moment.
Test Plan:
- Ran migrations.
- Verified that cluster services and bindings reported that they belonged to the cluster.
- Edited a cluster binding.
- Verified that the bound device was marked as a cluster device
- Moved a cluster binding, verified the old device was unmarked as a cluster device.
- Tried to edit a cluster device as an unprivileged user, got a sensible error.
{F1126552}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T6741, T10246
Differential Revision: https://secure.phabricator.com/D15339
Summary: Fixes T10411. Ref T10246. There are probably still some rough edges with this, but replace the old-school endpoints with modern ones so we don't unprototype with deprecated stuff.
Test Plan:
- Made a bunch of calls to the new endpoints with various constraints/attachments.
- Created and edited services, devices, interfaces, bindings, and properties on everything.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10246, T10411
Differential Revision: https://secure.phabricator.com/D15329
Summary:
Fixes T10410. Immediate impact of this is that you can now actually delete properties from Almanac services, devices and bindings.
The meat of the change is switching from CustomField to EditEngine for most of the actual editing logic. CustomField creates a lot of problems with using EditEngine for everything else (D15326), and weird, hard-to-resolve bugs like this one (not being able to delete stuff).
Using EditEngine to do this stuff instead seems like it works out much better -- I did this in ProfilePanel first and am happy with how it looks.
This also makes the internal storage for properties JSON instead of raw text.
Test Plan:
- Created, edited and deleted properties on services, devices and bindings.
- Edited and reset builtin properties on repository services.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10410
Differential Revision: https://secure.phabricator.com/D15327
Summary:
Ref T10246. Ref T6741.
When you have a namespace like "phacility.net", require users creating services and devices within it to have edit permission on the namespace.
This primarily allows us to lock down future device names in the cluster, so instances can't break themselves once they get access to Almanac.
Test Plan:
- Configured a `phacility.net` namespace, locked myself out of it.
- Could not create new `stuff.phacility.net` services/devices.
- Could still edit existing devices I had permission for.
- Configured a `free.phacility.net` namespace with more liberal policies.
- Could create `me.free.phacility.net`.
- Still could not create `other.phacility.net`.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T6741, T10246
Differential Revision: https://secure.phabricator.com/D15325
Summary:
Ref T6741. Ref T10246.
Root problem: to provide Drydock in the cluster, we need to expose Almanac, and doing so would let users accidentally or intentionally create a bunch of `repo006.phacility.net` devices/services which could conflict with the real ones we manage.
There's currently no way to say "you can't create anything named `*.blah.net`". This adds "namespaces", which let you do that (well, not yet, but they will after the next diff).
After the next diff, if you try to create `repo003.phacility.net`, but the namespace `phacility.net` already exists and you don't have permission to edit it, you'll be asked to choose a different name.
Also various modernizations and some new docs.
Test Plan:
- Created cool namespaces like `this.computer`.
- Almanac namespaces don't actually enforce policies yet.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T6741, T10246
Differential Revision: https://secure.phabricator.com/D15324
Summary: Ref T6741. Ref T10246. This is largely modernization, but will partially support namespace locking in Almanac.
Test Plan:
Searched for Almanac networks by name substring.
{F1121740}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T6741, T10246
Differential Revision: https://secure.phabricator.com/D15322
Summary: Ref T10246. Build an ngram index for Almanac services, and use it to support improved search.
Test Plan: {F1121725}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10246
Differential Revision: https://secure.phabricator.com/D15321
Summary:
Fixes T10205. Ref T10246. Previously, the issue was that the result set was not ordered, so "More Results" would not have been able to work in a reasonable way if there were more than 100 matching interfaces.
You would have seen 100 interfaces more or less at random, then clicked "more" and gotten 100 more random interfaces.
Now, you would see 100 "a" interfaces, then click more to get the next 100 alphabetical interfaces (say, "b" and "c" interfaces).
Test Plan:
- Clicked browse when binding an interface.
- Got a browse dialog.
- Artificially set query limit to 1, paged through "local" interfaces in an ordered, consistent way.
{F1121313}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10205, T10246
Differential Revision: https://secure.phabricator.com/D15320
Summary:
Ref T10205. Ref T10246. This is general modernization, but also supports fixing the interface datasource in T10205.
- Update Query.
- Update SearchEngine.
- Use an ngrams index for searching names efficiently.
Test Plan:
- Ran migrations.
- Searched Almanac devices by name.
- Created a new device, searched for it by name.
{F1121303}
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T10205, T10246
Differential Revision: https://secure.phabricator.com/D15319
Summary: Ref T8686. This method has the wrong name and calls a nonexistent method on `$service`.
Test Plan: Set page size to 5, paged locally, hit error, applied changes, got clean paging.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T8686
Differential Revision: https://secure.phabricator.com/D13460
Summary: Ref T8099. This adds a new class which all search engines return for layout. I thought about this a number of ways, and I think this is the cleanest path. Each Engine can return whatever UI bits they needs, and AppSearch or Dashboard picks and lays the bits out as needed. In the AppSearch case, interfaces like Notifications, Calendar, Legalpad all need more custom layouts. I think this also leaves a resonable path forward for NUX as well. Also, not sure I implemented the class correctly, but assume thats easy to fix?
Test Plan: Review and do a search in each application changed. Grep for all call sites.
Reviewers: btrahan, epriestley
Reviewed By: epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T8099
Differential Revision: https://secure.phabricator.com/D13332
Summary: See also D13186.
Test Plan: Ran `arc unit --everything`
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D13201
Summary:
Ref T8441. Ref T7715. Automatically generate a modern "Order" control in ApplicationSearch for engines which fully support SearchField.
Notably, this allows the standard "Order" control to automatically support custom field orders. We do this in Maniphest today, but in an ad-hoc way.
Test Plan: Performed order-by queries in Almanac (Services), Pholio, Files, People, Projects, and Paste.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T7715, T8441
Differential Revision: https://secure.phabricator.com/D13193
Summary:
Ref T4100. Ref T5595. These functions are trivial for now, but move us toward being able to define more default query behavior by default.
Future changes will give these methods meaningful, nontrivial behaviors.
Test Plan: `arc unit --everything`
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5595, T4100
Differential Revision: https://secure.phabricator.com/D12454
Summary:
Ref T4100. Ref T5595.
To support a unified "Projects:" query across all applications, a future diff is going to add a set of "Edge Logic" capabilities to `PolicyAwareQuery` which write the required SELECT, JOIN, WHERE, HAVING and GROUP clauses for you.
With the addition of "Edge Logic", we'll have three systems which may need to build components of query claues: ordering/paging, customfields/applicationsearch, and edge logic.
For most clauses, queries don't currently call into the parent explicitly to get default components. I want to move more query construction logic up the class tree so it can be shared.
For most methods, this isn't a problem, but many subclasses define a `buildWhereClause()`. Make all such definitions protected and consistent.
This causes no behavioral changes.
Test Plan: Ran `arc unit --everything`, which does a pretty through job of verifying this statically.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: yelirekim, hach-que, epriestley
Maniphest Tasks: T4100, T5595
Differential Revision: https://secure.phabricator.com/D12453
Summary: Ref T5750. Update the Almanac service query to be browsable.
Test Plan:
- Browsed and reordered Diffusion.
- Browsed and reordered services in Almanac.
{F373735}
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5750
Differential Revision: https://secure.phabricator.com/D12433
Summary: Fixes T7118. This does the basic "filter the list" thing, though it ends up being a little manual since I guess this hasn't come up before? There is also potential weird behavior if the user was using an app and lost access to it - they will have nothing selected on edit - but I think this is actually correct behavior in this circumstance.
Test Plan:
used a user who couldn't get access to the "quick create" apps and noted that the dropdown list on dashboard panel create was missing the expected engines
ran `arc unit --everything` to verify abstract method implemented everywhere
Reviewers: epriestley
Reviewed By: epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7118
Differential Revision: https://secure.phabricator.com/D11687
Summary: Ref T6822.
Test Plan: Visual inspection. These methods are only called from within the `PhabricatorApplicationSearchEngine` class.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T6822
Differential Revision: https://secure.phabricator.com/D11242
Summary: Ref T5833. Ref T6238. These are general capabilities which are particularly useful for synchronizing cluster specifications to instances.
Test Plan:
- Synchronized networks, devices, interfaces, services, bindings and properties to a managed instance.
- Used typeahead.
- Destroyed networks, devices, and services. Saw interfaces and bindings destroyed.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6238, T5833
Differential Revision: https://secure.phabricator.com/D11024
Summary: Ref T5833. Ref T6238. This will probably be useful somewhere in the upstream eventually, and is materially useful in the Instances application right now.
Test Plan: tippy typey typey
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6238, T5833
Differential Revision: https://secure.phabricator.com/D11016
Summary:
Fixes T6741. This allows Almanac services to be locked from the CLI. Locked services (and their bindings, interfaces and devices) can not be edited. This serves two similar use cases:
- For normal installs, you can protect cluster configuration from an attacker who compromises an account (or generally harden services which are intended to be difficult to edit).
- For Phacility, we can lock externally-managed instance cluster configuration without having to pull any spooky tricks.
Test Plan:
- Locked and unlocked services.
- Verified locking a service locks connected properties, bindings, binding properties, interfaces, devices, and device properties.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T6741
Differential Revision: https://secure.phabricator.com/D11006
Summary:
Ref T5833. This allows Almanac ServiceTypes to define default properties for a service, which show up in the UI and are more easily editable.
Overall, this makes it much easier to make structured/usable/consistent service records: you can check a checkbox that says "prevent new allocations" instead of needing to know the meaning of a key.
Test Plan: {F251593}
Reviewers: chad, btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5833
Differential Revision: https://secure.phabricator.com/D10996
Summary:
Ref T5833. This allows services to be typed, to distinguish between different kinds of services. This makes a few things easier:
- It's easier for clients to select the services they're interested in (see note in T5873 about Phacility). This isn't a full-power solution, but gets is some of the way there.
- It's easier to set appropriate permissions around when modifications to the Phabricator cluster are allowed. These service nodes need to be demarcated as special in some way no matter what (see T6741). This also defines a new policy for users who are permitted to create services.
- It's easier to browse/review/understand services.
- Future diffs will allow ServiceTypes to specify more service structure (for example, default properties) to make it easier to configure services correctly. Instead of a free-for-all, you'll get a useful list of things that consumers of the service expect to read.
The "custom" service type allows unstructured/freeform services to be created.
Test Plan:
- Created a new service (and hit error cases).
- Edited an existing service.
- Saw service types on list and detail views.
- Poked around new permission stuff.
- Ran `almanac.queryservices` with service class specification.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5833
Differential Revision: https://secure.phabricator.com/D10995
Summary:
Ref T2783. This is primarily exploratory and just figuring out what we're blocked on:
- Allow a Repository to be bound to a Service. The Service may eventually define multiple read/write nodes, etc.
- There's no UI to do this binding yet, you have to touch the database manually.
- If a repository is bound to a Service, effect Conduit calls via calls to the remote service instead of executing them in-process.
- These don't actually work yet since there's no authentication (see T5955).
Test Plan:
- Made a nice Service with a nice Binding to a nice Interface on a nice Device.
- Force-associated a repository with the service using a raw MySQL query.
- Saw Phabricator try to make a remote call to the service (on localhost) and fail because of missing auth stuff.
- Also ran `almanac.queryservices`.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T2783
Differential Revision: https://secure.phabricator.com/D10982
Summary:
Ref T5833. Adds support for arbitrary properites to Almanac devices and bindings.
- For Devices, this allows you to maybe mark what `rack` a server is on, the `serial` number of a router, etc.
- For Bindings, this allows you to maybe mark that a bound device is `active`, provide `credentials`, expose it as `readonly`, etc.
Test Plan: Added properties to Devices and Bindings.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5833
Differential Revision: https://secure.phabricator.com/D10781
Summary:
Ref T5833. Currently, we have an `AlmanacDeviceProperty`, but it doesn't use CustomFields and is specific to devices. Make this more generic:
- Reuse most of the CustomField infrastructure (so we can eventually get easy support for nice editor UIs, etc).
- Make properties more generic so Services, Bindings and Devices can all have them.
The major difference between this implementation and existing CustomField implementations is that all other implementations are application-authoritative: the application code determines what the available list of fields is.
I want Almanac to be a bit more freeform (basically: you can write whatever properties you want, and we'll put nice UIs on them if we have a nice UI available). For example, we might have some sort of "ServiceTemplate" that says "a database binding should usually have the fields 'writable', 'active', 'credential'", which would do things like offer these as options and put a nice UI on them, but you should also be able to write whatever other properties you want and add services without building a specific service template for them.
This involves a little bit of rule bending, but ends up pretty clean. We can adjust CustomField to accommodate this a bit more gracefully later on if it makes sense.
Test Plan: {F229172}
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5833
Differential Revision: https://secure.phabricator.com/D10777
Summary: Ref T5833. Allows you to bind a service (like `db.example.com`) to one or more interfaces (for example, to specify a pool with one read/write host and two read-only hosts). You can't configure which hosts have which properties yet, but you can add all the relevant interfaces to the service. Next diff will start supporting service, binding, and device properties like "is writable", "is active", etc., so that Almanac will be able to express operations like "change which database is writable", "disable writes", "bring a device down", etc.
Test Plan: See screenshots.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5833
Differential Revision: https://secure.phabricator.com/D10745
Summary: Ref T5833. An interface is an IP (maybe v4, maybe v6) and port on a specified network (public internet, VPN, NAT block, etc).
Test Plan: See screenshots.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5833
Differential Revision: https://secure.phabricator.com/D10718
Summary: Ref T5833. This differentiates address spaces like the public internet from VPNs, so when a service is available at `192.168.0.1`, we'll know it's on some specific NAT block or whatever.
Test Plan: See screenshots.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5833
Differential Revision: https://secure.phabricator.com/D10715
Summary: Ref T5833. The "uninteresting" part of this object is virtually identical to AlmanacService.
Test Plan: See screenshots.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5833
Differential Revision: https://secure.phabricator.com/D10714
Summary: Ref T5833. See that task for functional goals and some discussion of design.
Test Plan: See screenshots.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: epriestley
Maniphest Tasks: T5833
Differential Revision: https://secure.phabricator.com/D10713
Summary:
Ref T4209. This creates storage for public keys against authorized hosts, such that servers can be authorized to make Conduit calls as the omnipotent user.
Servers are registered into this system by running the following command once:
```
bin/almanac register
```
NOTE: This doesn't implement authorization between servers, just the storage of public keys.
Placing this against Almanac seemed like the most sensible place, since I'm imagining in future that the `register` command will accept more information (like the hostname of the server so it can be found in the service directory).
Test Plan: Ran `bin/almanac register` and saw the host (and public key information) appear in the database.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T4209
Differential Revision: https://secure.phabricator.com/D10400
