1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-02 11:42:42 +01:00
Commit graph

1020 commits

Author SHA1 Message Date
Chad Little
83a89166ee Add profile images to Repositories
Summary: Builds out some images to use to identify repositories. Fixes T12825.

Test Plan:
Try setting custom, built in, and null images.

{F4998175}

{F4998192}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12825

Differential Revision: https://secure.phabricator.com/D18116
2017-06-12 07:51:39 -07:00
epriestley
8ef9490f66 Add a retroactive migration to expand the contentHash field
Summary:
See D18037. The migration there may cause us to write new file records as a side effect.

Ideally, we would rewrite that migration to not ever have this kind of side effect. However, that would make it much more complicated, and it's already very complicated.

Instead, retroactively expand the size of this field before `storage adjust` does it, so it has the right size by the time we hit the migration in D18037.

Test Plan:
@chad, can you `arc patch` this and see if it works?

It's possible that it will get us about five lines deeper and then we'll just hit another similar exception, and that this isn't really a viable way forward.

Reviewers: chad, amckinley

Reviewed By: amckinley

Subscribers: amckinley, chad

Differential Revision: https://secure.phabricator.com/D18107
2017-06-08 07:42:40 -07:00
Austin McKinley
fb9d036e57 Show task duplicates as related objects in Maniphest and migrate old duplicates
Summary: Does the UI work that's part of T12234 and adds migrations for both of the old-style duplicate transactions.

Test Plan:
 - Started with a clean DB.
 - Checked out really old code that marks tasks as dupes using comments.
 - Made a bunch of tasks and closed some as dupes. Made a bunch of additional comments.
 - Checked out D10427 and did a `storage upgrade`.
 - Made a bunch more new tasks and dupes.
 - Snapshotted DB.
 - Ran migration repeatedly until all expected edges showed up in the `phabricator_maniphest.edge`table.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T12234

Differential Revision: https://secure.phabricator.com/D18037
2017-06-07 13:30:20 -07:00
Austin McKinley
04fd93e51e Drop DifferentialDraft storage
Summary: Fixes T12104.

Test Plan: Ran `bin/storage upgrade` and observed table dun got dropped.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Maniphest Tasks: T12104

Differential Revision: https://secure.phabricator.com/D18034
2017-05-26 13:59:26 -07:00
Austin McKinley
b27c2ed6d1 Index Project milestones to accurately reflect milestone membership
Summary: Fixes T12505. `PhabricatorProjectsMembershipIndexEngineExtension->materializeProject()` was incorrectly bailing early for milestone objects, which prevented milestone members from being calculated correctly. This was causing problems where (for example) an Owners package owned by a milestone wasn't being satisfied when a member of the milestone approved a revision.

Test Plan: Invoked migration, observed that a user's milestones correctly showed up when searched for. Also observed that accepting a revision on behalf of a milestone now satisfies Owners rules.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12505

Differential Revision: https://secure.phabricator.com/D18033
2017-05-26 13:10:41 -07:00
epriestley
7e91f42b02 Issue commands to Nuance items, at least roughly
Summary:
Ref T12738. This makes clicking "Throw In Trash" technically do something, sort of.

In Nuance, the default mode of operation for actions is asynchronous -- so you don't have to wait for a response from Twitter or GitHub after you mash the "send default reply tweet" / "close this pull request with a nice response" button and can move directly to the next item instead.

In the future, some operations will attempt to apply synchronously (e.g., local actions like "ignore this item forever"). This fakes our way through that for now.

There's also no connection to the action actually doing anything yet, but I'll probably rig that up next.

Test Plan: {F4975227}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12738

Differential Revision: https://secure.phabricator.com/D18010
2017-05-24 11:04:57 -07:00
epriestley
272a5d668f Fix a handful of Nuance fatals
Summary: Ref T12738. Some of the Nuance "form" workflows currently fatal after work on the GitHub stuff. Try to make everything stop fataling, at least.

Test Plan: Using "Complaints Form" no longer fatals, and now lodges a complaint instead.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12738

Differential Revision: https://secure.phabricator.com/D18007
2017-05-24 11:02:55 -07:00
Austin McKinley
f92059d84c Migrate Project status to modular transactions
Test Plan: Unit tests pass. Archived/activated some projects a couple times; observed expected transactions on timeline.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D17953
2017-05-18 11:36:13 -07:00
Austin McKinley
1e3c8df1c8 Migrate Project names to modular transactions
Summary: Also changes access modifiers on `PhabricatorProjectTransactionEditor` and sets up `storage` for `applyExternalEffects`.

Test Plan: Created new projects, attempted to create without name, with too long of a name, and with a name that conflicts with other projects and observed expected errors.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Maniphest Tasks: T12673

Differential Revision: https://secure.phabricator.com/D17947
2017-05-17 17:12:22 -07:00
Chad Little
d6a620be45 Update Maniphest for modular transactions
Summary: Ref T12671. This modernized Maniphest transactions to modular transactions.

Test Plan:
- Create Task
- Edit Task
- Raise Priority
- Change Status
- Merge as a duplicate
- Create Subtask
- Claim Task
- Assign Project
- Move on Workboard
- Set a cover image
- Assign story points
- Change story points
- Generate lots via lipsum
- Bulk edit tasks
- Leave comments
- Award Token

I'm sure I'm missing something.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: hazelyang, Korvin

Maniphest Tasks: T12671

Differential Revision: https://secure.phabricator.com/D17844
2017-05-15 10:29:06 -07:00
Austin McKinley
0dc90b891a Reimplement Slowvote transactions using modular transactions
Summary:
Fixes T12623. Adds new modular transactions to Slowvote. Also converts
the `shuffle` column to `bool` for consistency with other boolean-ish columns.

Test Plan:
Create a new vote, modified everything that could be modified from the web UI,
observed expected timeline.

Example timeline: {F4938843}

Example transaction values in DB: {F4938850}

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T12623

Differential Revision: https://secure.phabricator.com/D17830
2017-05-04 20:20:00 -07:00
epriestley
d2baa88171 Allow Owners packages to have arbitrarily long names
Summary:
  - Change column type from `sort128` to `sort`.
  - Remove `originalName`. This column is unused. Long ago, we used it to generate a `Thread-Topic` header for mail, but just use PHIDs now (the value just needs to be stable for a given object, users normally don't see it).

Test Plan:
  - Created a package with a beautifully long name. Magnificent!
  - Grepped for `originalName` / `getOriginalName()`, found no Owners hits.
  - Verified that there isn't any name-length validation code to remove.

{F4925637}

Reviewers: chad, amckinley

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17798
2017-04-27 18:04:03 -07:00
epriestley
e2a94019b1 Migrate accounts to correct user email verification state flag
Summary:
Depends on D17785. Fixes T12635. There was a bug where users could verify their primary email without getting the "isEmailVerified" flag set on their accounts.

D17785 fixes this bug. This change migrates affected account to fix their state, now that they can't get in trouble any more (hopefully).

Test Plan:
  - Explicitly removed this flag from a bunch of accounts.
  - Ran migration, saw the accounts get fixed.
  - Ran migration again (`storage upgrade --apply ...`), saw the accounts not get touched.
  - We have 117 affected accounts on `secure`, so I'll verify that this fixes them.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12635

Differential Revision: https://secure.phabricator.com/D17786
2017-04-24 10:24:51 -07:00
epriestley
bef68ccdd6 Regenrate Quickstart SQL
Summary:
The quickstart SQL hasn't been regenrated in a while, mildly impacting unit test and instance startup times.

  - Use `bin/storage quickstart` to regenerate quickstart.
  - Manually set the FULLTEXT tables back to `MyISAM` until we deal with T11741.

Test Plan:
  - Saw database setup drop from ~10,500ms to ~7,500ms locally.
  - Visually inspected diff, changes looked expected.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17775
2017-04-23 11:04:02 -07:00
epriestley
52c4715bbc No-op the Conpherence thumbnail resizing migration
Summary:
Fixes T12628. After later changes to `PhabricatorFile`, this migration no longer runs if you upgrade through it to a recent `HEAD` while your data has some room images.

Since this isn't critical and has been available for ~6 months, I just nuked it as a first pass. I can find a more careful approach which lets us continue to run this migration instead if you're hesitant to skip this step, although it may be a little involved.

In 95% of cases we avoid this by updating the storage table as it existed at the time the migraiton ran, but Files are much too complicated for that to be realistic.

Test Plan: Ran `bin/storage upgrade -f --apply phabricator:20161005.conpherence.image.2.php`, saw it do nothing.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12628

Differential Revision: https://secure.phabricator.com/D17770
2017-04-23 10:46:09 -07:00
epriestley
95dd9dbf43 Make Applications extend LiskDAO
Summary:
Ref T11476. This is a bit hacky, but makes `Application` extend `LiskDAO` so we can apply transactions to it with an `Editor` class.

Also fixes schema stuff so builds should produce a clean bill of health again.

This might only get you slightly further, yell if you run into more trouble.

Test Plan:
  - Ran `bin/storage upgrade -f` and got no warnings.
  - Browsed around, nothing exploded?

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11476

Differential Revision: https://secure.phabricator.com/D17738
2017-04-19 16:06:14 -07:00
Austin McKinley
febd68039f Add initial infrastructure for adding ModularTransaction support to Application config changes
Summary: Part of the groundwork for T11476.

Test Plan: ran `./bin/storage upgrade` and observed expected DB tables

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11476

Differential Revision: https://secure.phabricator.com/D17736
2017-04-19 15:44:57 -07:00
epriestley
76d0b67d91 Remove "dateTouched" from ConpherenceParticipant
Summary:
Pathway to D17685. This column is (mostly) a denormalization of `dateModified` on the thread.

Just use a JOIN instead.

This isn't //exactly// the same: we'll bump threads to the top now for non-message changes (e.g., a topic or title change). That seems fine, but we could put a `lastMessageDate` on Thread later if we want to refine it.

Also got rid of a lot of other unused stuff. There's a big garbage TODO here, I'll fix that in the next change.

Test Plan:
  - Grepped for `dateTouched`.
  - Grepped for `participantCursor`.
  - Grepped for `ConpherenceParticipantQuery::LIMIT`.
  - Looked for callsites to `setOrder()`, found none.
  - Added a message to an older thread, saw it bump up to the top.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17731
2017-04-19 13:58:54 -07:00
epriestley
0a335f91cd Remove "participationStatus" from ConpherenceParticipant
Summary:
Pathway to D17685. This column is a very complicated cache of: is participant.messageCount equal to thread.messageCount?

We can just ask this question with a JOIN instead and simplify things dramatically.

Test Plan:
  - Ran migration.
  - Browsed around.
  - Sent a message, saw unread count go up.
  - Read the message, saw unread count go down.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17730
2017-04-19 13:58:42 -07:00
epriestley
c96e17f5ea Remove "behindTransactionPHID" from ConpherenceParticipant
Summary: Pathway to D17685. Nothing reads this field and it has no use or value.

Test Plan:
  - Ran migration.
  - Grepped for `behindTransactionPHID`.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17729
2017-04-19 13:58:29 -07:00
Austin McKinley
be00264ae7 Make daemons perform file deletion
Summary:
Deletion is a possibly time-intensive process, especially with large
files that are backed by high-latency, chunked storage (such as
S3). Even ~200mb objects take minutes to delete, which makes for an
unhappy experience. Fixes T10828.

Test Plan:
Delete a large file, and stare in awe of the swiftness with
which I am redirected to the main file application.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: thoughtpolice, Korvin

Maniphest Tasks: T10828

Differential Revision: https://secure.phabricator.com/D15743
2017-04-18 11:09:41 -07:00
Austin McKinley
976fbee877 Implement ngram search for File objects
Summary: Follows the outline in D15656 for implementing ngram search for names of File objects. Also created FileFullTextEngine, because without implementing `PhabricatorFulltextInterface`, `./bin/search` complains that `File` is not an indexable type.

Test Plan:
  - ran `./bin/storage upgrade` to apply the schema change
  - confirmed the presence of a new `file_filename_ngrams` table
  - added a couple file objects
  - ran `bin/search index --type file --force`
  - confirmed the presence of rows in `file_filename_ngrams`
  - did a few keyword searches and saw expected results

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T8788

Differential Revision: https://secure.phabricator.com/D17702
2017-04-17 17:37:20 -07:00
Chad Little
5587abf04c Remove recentParticipants from ConpherenceThread
Summary: We no longer display this any more in the UI, so go ahead and remove the callsites and db column.

Test Plan: New Room, with and without participants.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17683
2017-04-13 13:55:08 -07:00
epriestley
3245e74f16 Show users how fulltext search queries are parsed and executed; don't query stopwords or short tokens
Summary:
Depends on D17670. Fixes T12137. Fixes T12003. Ref T2632.

This shows users a readout of which terms were actually searched for.

This also drops those terms from the query we submit to the backend, dodging the weird behaviors / search engine bugs in T12137.

This might need some design tweaking.

Test Plan: {F4899825}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12137, T12003, T2632

Differential Revision: https://secure.phabricator.com/D17672
2017-04-12 19:07:54 -07:00
Chad Little
a9845b0b1d Remove picture crop transaction from Conpherence
Summary: Fixes T11730. Removes an old transaction that hasn't been used in a year.

Test Plan: Run sql, check various rooms.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11730

Differential Revision: https://secure.phabricator.com/D17666
2017-04-12 14:18:54 -07:00
epriestley
d4d46501ed Add a repair migration to fix legacy Calendar events without "utcInstanceEpoch"
Summary: Fixes T12488. Some events appear to have survived earlier migrations without getting completely fixed. Fix them.

Test Plan:
  - Ran migration locally with `bin/storage upgrade` (but: I could not reproduce this problem locally).
  - Ran migration in production and saw ICS import stop fataling.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12488

Differential Revision: https://secure.phabricator.com/D17642
2017-04-10 08:47:07 -07:00
epriestley
3d816e94df Rename "PhabricatorHash::digest()" to "weakDigest()"
Summary: Ref T12509. This encourages code to move away from HMAC+SHA1 by making the method name more obviously undesirable.

Test Plan: `grep`, browsed around.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D17632
2017-04-06 15:43:33 -07:00
epriestley
d450a08890 Support HMAC+SHA256 with automatic key generation and management
Summary:
Ref T12509. This adds support for HMAC+SHA256 (instead of HMAC+SHA1). Although HMAC+SHA1 is not currently broken in any sense, SHA1 has a well-known collision and it's good to look at moving away from HMAC+SHA1.

The new mechanism also automatically generates and stores HMAC keys.

Currently, HMAC keys largely use a per-install constant defined in `security.hmac-key`. In theory this can be changed, but in practice essentially no install changes it.

We generally (in fact, always, I think?) don't use HMAC digests in a way where it matters that this key is well-known, but it's slightly better if this key is unique per class of use cases. Principally, if use cases have unique HMAC keys they are generally less vulnerable to precomputation attacks where an attacker might generate a large number of HMAC hashes of well-known values and use them in a nefarious way. The actual threat here is probably close to nonexistent, but we can harden against it without much extra effort.

Beyond that, this isn't something users should really have to think about or bother configuring.

Test Plan:
  - Added unit tests.
  - Used `bin/files integrity` to verify, strip, and recompute hashes.
  - Tampered with a generated HMAC key, verified it invalidated hashes.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D17630
2017-04-06 15:42:59 -07:00
epriestley
aea46e55da Fix an issue where "Request Review" of a fully-accepted revision would transition to "Accepted"
Summary:
Ref T10967. This is explained in more detail in T10967#217125

When an author does "Request Review" on an accepted revision, void (in the sense of "cancel out", like a bank check) any "accepted" reviewers on the current diff.

Test Plan:
  - Create a revision with author A and reviewer B.
  - Accept as B.
  - "Request Review" as A.
  - (With sticky accepts enabled.)
  - Before patch: revision swithced back to "accepted".
  - After patch: the earlier review is "voided" by te "Request Review", and the revision switches to "Review Requested".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967

Differential Revision: https://secure.phabricator.com/D17566
2017-03-28 11:50:15 -07:00
Mukunda Modell
e41c25de50 Support multiple fulltext search clusters with 'cluster.search' config
Summary:
The goal is to make fulltext search back-ends more extensible, configurable and robust.

When this is finished it will be possible to have multiple search storage back-ends and
potentially multiple instances of each.

Individual instances can be configured with roles such as 'read', 'write' which control
which hosts will receive writes to the index and which hosts will respond to queries.

These two roles make it possible to have any combination of:

* read-only
* write-only
* read-write
* disabled

This 'roles' mechanism is extensible to add new roles should that be needed in the future.

In addition to supporting multiple elasticsearch and mysql search instances, this refactors
the connection health monitoring infrastructure from PhabricatorDatabaseHealthRecord and
utilizes the same system for monitoring the health of elasticsearch nodes. This will
allow Wikimedia's phabricator to be redundant across data centers (mysql already is,
elasticsearch should be as well).

The real-world use-case I have in mind here is writing to two indexes (two elasticsearch clusters
in different data centers) but reading from only one. Then toggling the 'read' property when
we want to migrate to the other data center (and when we migrate from elasticsearch 2.x to 5.x)

Hopefully this is useful in the upstream as well.

Remaining TODO:

* test cases
* documentation

Test Plan:
(WARNING) This will most likely require the elasticsearch index to be deleted and re-created due to schema changes.

Tested with elasticsearch versions 2.4 and 5.2 using the following config:

```lang=json
  "cluster.search": [
    {
      "type": "elasticsearch",
      "hosts": [
        {
          "host": "localhost",
          "roles": { "read": true, "write": true }
        }
      ],
      "port": 9200,
      "protocol": "http",
      "path": "/phabricator",
      "version": 5
    },
    {
      "type": "mysql",
      "roles": { "write": true }
     }
  ]

Also deployed the same changes to Wikimedia's production Phabricator instance without any issues whatsoever.
```

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Tags: #elasticsearch, #clusters, #wikimedia

Differential Revision: https://secure.phabricator.com/D17384
2017-03-26 08:16:47 +00:00
epriestley
aa91dc992e Record which user accepted on behalf of packages/owners reviewers
Summary:
Ref T12271. Don't do anything with this yet, but store who accepted/rejected/whatever on behalf of reviewers.

In the future, we could use this to render stuff like "Blessed Committers (accepted by epriestley)" or whatever. I don't know that this is necessarily super useful, but it's easy to track, seems likely to be useful, and would be a gigantic pain to backfill later if we decide we want it.

Test Plan: Accepted/rejected a revision, saw reviewers update appropriately.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12271

Differential Revision: https://secure.phabricator.com/D17537
2017-03-22 14:26:37 -07:00
epriestley
8ad5d28686 Migrate old reviewer edges to new storage
Summary:
Ref T10967. We still have double writes, so all reviewers are being written to both old and new storage. This migrates all the data in the old storage to the new storage, so both storage tables should have a complete set of data and be getting identical updates as we move forward.

After this, I can move readers over one at a time and eventually get rid of the old writes and old storage.

This loads all of the edge data into memory in a big chunk. I reached out to one install to get some more information about their data size. Ours is quite manageable and I think even large installs will probably fit into memory, but we can do this in chunks if not.

However, because the Edge table doesn't have an `id` column, we can't use either the `RawMigrationIterator` or the `MigrationIterator`, and would need to write a new `EdgeMigrationIterator`. This isn't tons of work but might not be necessary.

Test Plan: Ran the migration locally, spot-checked the results in the database for sanity and correctness.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967

Differential Revision: https://secure.phabricator.com/D17515
2017-03-20 16:44:40 -07:00
epriestley
794b456530 Store "last comment" and "last action" diffs on reviewers
Summary:
Ref T10967. We have a "commented" state to help reviewers get a better sense of who is part of a discussion, and a "last action" state to help distinguish between "accept" and "accepted an older version", for the purposes of sticky accepts and as a UI hint.

Currently, these are first-class states, partly beacuse we were somewhat limited in what we could do with edges. However, a more flexible way to represent them is as flags separate from the primary state flag.

In the new storage, write them as separate state information: `lastActionDiffPHID` stores the Diff PHID of the last review action (accept, reject, etc). `lastCommentDiffPHID` stores the Diff PHID of the last comment (top-level or inline).

Test Plan: Applied storage changes, commented and acted on a revision. Saw appropriate state reflected in the database.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967

Differential Revision: https://secure.phabricator.com/D17514
2017-03-20 16:44:05 -07:00
epriestley
7626ec0ce1 Correct an issue where "View Raw File" in Differential generated a file with overbroad permissions
Summary:
Via HackerOne. When you view a raw file in Differential, we currently generate a permanent file with default permissions. This may be incorrect: default permissions may be broader than the diff's permissions.

The other three methods of downloading/viewing raw files ("Download" in Diffusion and Differential, "View Raw" in Diffusion and Differential) already apply policies correctly and generate temporary files. However, this workflow was missed when other workflows were updated.

Beyond updating the workflow, delete any files we've generated in the past. This wipes the slate clean on any security issues and frees up a little disk space.

Test Plan:
  - Ran migration script, saw existing files get purged.
  - Did "View Raw File", got a new file.
  - Verified that the file was temporary and properly attached to the diff, with "NO ONE" permissions.
  - Double-checked that Diffusion already runs policy logic correctly and applies appropriate policies.
  - Double-checked that "Download Raw Diff" in Differential already runs policy logic correctly.
  - Double-chekced that "Download Raw Diff" in Diffusion already runs policy logic correctly.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17504
2017-03-16 09:51:48 -07:00
epriestley
251ee9b660 Add dedicated "reviewers" storage to Differential and do double writes
Summary:
Ref T10967. This is an incremental step toward removing "reviewers" back to a dedicated storage table so we can handle changes like T11050.

This adds the storage table, and starts doing double writes to it (so new or updated reviewers write to both the old edge table and the new "reviewers" table).

Then we can do a migration, swap readers over one at a time, and eventually remove the old write and old storage and then implement new features.

This change has no user-facing impact, it just causes us to write new data to two places instead of one.

This is not completely exhaustive: the Herald "Add Reviewers" action is still doing a manual EDGE transaction. I'll clean that up next and do another pass to look for anything else I missed.

This is also a bit copy/pastey for now but the logic around "RESIGN" is a little different in the two cases until T11050. I'll unify it in future changes.

Test Plan:
  - Did a no-op edit.
  - Did a no-op comment.
  - Added reviewers.
  - Removed reviewers.
  - Accepted and rejected revisions.

After all of these edits, did a `SELECT * FROM differential_reviewer` manually and saw consistent-looking rows in the database.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967

Differential Revision: https://secure.phabricator.com/D17495
2017-03-14 11:51:51 -07:00
Chad Little
f2e013c2e9 Prep user table for default images
Summary: Ref T10319. Adds in database columns for upcoming default generated avatar support.

Test Plan: Ran storage upgrade, log into local site to verify it didn't blow up.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T10319

Differential Revision: https://secure.phabricator.com/D17459
2017-03-04 08:18:07 -08:00
epriestley
dc7ecf5875 Add "subtype" storage to Maniphest tasks
Summary: Ref T12314. Provides a field on tasks for storing subtypes. Does nothing interesting yet.

Test Plan:
  - Ran storage upgrade.
  - Created some tasks.
  - Looked in the database.
  - Used Conduit to query some tasks.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12314

Differential Revision: https://secure.phabricator.com/D17441
2017-03-02 04:17:47 -08:00
epriestley
91ef237290 Add a "subtype" field to EditEngine forms
Summary:
Ref T12314. This adds storage so EditEngine forms can later be marked as edit fields for particular types of objects (like an "animal edit form" vs a "plant edit form").

We'll take you to the right edit form when you click "Edit" by selecting among forms with the same subtype as the task.

This doesn't do anything very interesting on its own.

Test Plan:
  - Ran `bin/storage upgrade`.
  - Verified database got the field with proper values.
  - Created a new form, checked the database.
  - Ran unit tests.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12314

Differential Revision: https://secure.phabricator.com/D17439
2017-03-02 04:16:27 -08:00
epriestley
ad01e26af7 Redesign Home/Profile/Projects side navigation
Summary: Ref T11957. Needs some more polish, but I think everything here is square.

Test Plan: Add personal/global items to home, test mobile. Test workboards / colors.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: 20after4, rfreebern, Korvin

Maniphest Tasks: T11957

Differential Revision: https://secure.phabricator.com/D17259
2017-01-31 08:59:01 -08:00
Chad Little
58c857a681 Remove motivator panel
Summary: Removes the often funny, but never really used but will cause us bug reports someday.... cat facts.

Test Plan: Install cat facts, run storage upgrade, see no cat facts in menu.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12126

Differential Revision: https://secure.phabricator.com/D17233
2017-01-19 14:55:19 -08:00
epriestley
11f3a8cfca Force 'changes' to a string in hunk migration
Summary: Fixes T12090. In obscure situations lost to the mists of time, the `changes` column could be `null`. Force a string cast so the migration finishes, even though these changesets are likely meaningless.

Test Plan:
I did a force-reapply as a sanity check:

```
$ ./bin/storage upgrade -f --apply phabricator:20161213.diff.01.hunks.php
```

That went cleanly; it would only have caught dramatic errors, but I didn't completely butcher things.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12090

Differential Revision: https://secure.phabricator.com/D17168
2017-01-10 12:28:09 -08:00
epriestley
2dfe79cfc7 When updating revisions in response to commits, reuse previously generated diffs
Summary:
Fixes T10968. In rare situations, we can generate a diff, then hit an error which causes this update to fail.

When it does, we tend to get stuck in a loop creating diffs, which can fill the database up with garbage. We saw this once in the Phacility cluster, and one instance hit it, too.

Instead: when we create a diff, keep track of which commit we generated it from. The next time through, reuse it if we already built it.

Test Plan:
  - Used `bin/differential attach-commit <commit> <revision>` to hit this code.
  - Simulated a filesystem write failure, saw the diff get reused.
  - Also did a normal update, which worked properly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10968

Differential Revision: https://secure.phabricator.com/D17164
2017-01-09 12:13:44 -08:00
Chad Little
8a85ee7c15 Add CustomPHID to PhabricatorProfileMenuEngineConfiguration
Summary: Ref T5867, adds a customPHID field, nullable, and lets you query by it... i think? Not fully able to grok all the EditEngine stuff, but I think this is the right place for the query.

Test Plan: Not wired to anything, but pulling up project menu, editing, all still works.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T5867

Differential Revision: https://secure.phabricator.com/D17149
2017-01-07 10:49:54 -08:00
Sébastien Santoro
178466b826 Fix 20160122.project.1.boarddefault.php database upgrade patch
Summary: Fixes T12031.

Test Plan: Upgraded a December 2015 Phabricator installation to current schema.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T12031

Differential Revision: https://secure.phabricator.com/D17094
2016-12-19 12:52:55 -08:00
Chad Little
c6bdd2c56b Add Ngram support to Dashboards / Panels
Summary: Build ngram indexs, adds search by name capability.

Test Plan: Search for a dashboard by partial name, search for a panel by partial name.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17090
2016-12-16 12:09:06 -08:00
epriestley
3d8d98bd8d Probably (?) fix two very old project slug migrations
Summary: Fixes T12020. These callsites to `getPhrictionSlug()` were missed when that method was removed. They're very old (early 2014, late 2011).

Test Plan:
These are tricky to test because the migrations are so ancient, but `bin/storage upgrade --force --apply phabricator:20140521.projectslug.2.mig.php` gave me //plausible// results.

The other migration is so ancient that it can't apply to a modern database so I'm just kind of winging that one. We probably have essentially no installs which will ever apply it again, though.

Reviewers: chad, avivey

Reviewed By: avivey

Maniphest Tasks: T12020

Differential Revision: https://secure.phabricator.com/D17060
2016-12-14 14:14:21 -08:00
epriestley
c67e87c809 Force hunk storage migration to the modern format
Summary:
Ref T8475. This forces installs to migrate hunks to the modern format.

We stopped writing to the legacy format a very long time ago (2+ years?) without issues.

This doesn't destroy any data. T8623 has guidance and I'll publish more changelog guidance.

Test Plan: Faked some legacy data and migrated it.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8475

Differential Revision: https://secure.phabricator.com/D17039
2016-12-13 14:34:08 -08:00
epriestley
9d2feacfab Rename a missed Profile -> MenuItem
Summary: Didn't grep this good enough.

Test Plan: `bin/storage upgrade -f --apply ..`, got a clean apply.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17046
2016-12-13 14:27:00 -08:00
Chad Little
26127b9c5f Allow Dashboards to set an icon
Summary: Allows users set an icon (for reuse on upcoming home) for their dashboard based on 16 descriminating choices.

Test Plan: Create a new dashboard, set new icon. Edit an existing dashboard, set icon.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17042
2016-12-13 11:30:22 -08:00
Chad Little
c03a412d5c Add authorPHID to Dashboard Panels
Summary: Adds authorPHID to panels so we can default to the panels you made.

Test Plan: Run upgrade, visit manage panels, see my panels. Create a new panel. Edit a panel.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17036
2016-12-13 10:07:16 -08:00
epriestley
776a1f3aec Rename dashboard author patches so that the backfill patch happens
after the column it is adding is created

Auditors: chad
2016-12-12 15:44:28 -08:00
Chad Little
59f3b5125d Add authorPHID to Dashboards
Summary: Adds an authorPHIDs, populates olds ones.

Test Plan: Make a new Dashboard, see that I created it.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17022
2016-12-12 15:26:43 -08:00
epriestley
8480776ccd Rename "ProfilePanelConfiguration" to "ProfileMenuItemConfiguration"
Summary:
Ref T11957. This renames the Configuration storage, transaction, query, and PHID type.

No rename on the actual menu item types yet, that's next (and should be the end of this, I think).

Test Plan:
  - Viewed projects.
  - Viewed profiles.
  - Edited a project menu.
  - Grepped for all renamed symbols, I think?

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11957

Differential Revision: https://secure.phabricator.com/D17027
2016-12-11 11:44:22 -08:00
epriestley
29a3cd5121 Add "Manual Activities", to tell administrators to rebuild the search index
Summary:
Ref T11922. After updating to HEAD of `master`, you need to manually rebuild the index. We don't do this during `bin/storage upgrade` because it can take a very long time (`secure.phabricator.com` took roughly an hour) and can happen while Phabricator is running.

However, if we don't warn users about this they'll just get a broken index unless they go read the changelog (or file an issue, then we tell them to go read the changelog).

This adds a very simple table for notes to administrators so we can write a "you need to go rebuild the index" note, then adds one.

Administrators clear the note by completing the activity and running `bin/config done reindex`. This isn't automatic because there are various strategies you can use to approach the issue, which I'll discuss in greater detail in the linked documentation.

Also, fix an issue where `bin/storage upgrade --apply <patch>` could try to re-mark an already-applied patch as applied.

Test Plan:
  - Ran storage ugrades.
  - Got instructions to rebuild search index.
  - Cleared instructions with `bin/config done reindex`.

Reviewers: chad

Reviewed By: chad

Subscribers: avivey

Maniphest Tasks: T11922

Differential Revision: https://secure.phabricator.com/D16965
2016-11-30 11:23:54 -08:00
epriestley
7c5b5327c8 Use stemming in the MySQL fulltext search engine
Summary:
Ref T6740. When we index a document, also save a copy of the stemmed version.

When querying, search the combined corpus for the terms.

(We may need to tune this a bit later since it's possible for literal, quoted terms to match in the stemmed section, but I think this wil rarely cause issues in practice.)

A downside here is that search sort of breaks if you upgrade into this and don't reindex. I wasn't able to find a way to issue the query that remained compatible with older indexes and didn't have awful performance, so my plan is:

  - Put this on `secure`.
  - Rebuild the index.
  - If things look good after a couple of days, add a way that we can tell people they need to rebuild the search index with a setup warning.

We might get some reports between now and then, but if this is super awful we should know by the end of the weekend.

Test Plan:
WOW AMAZING

{F2021466}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T6740

Differential Revision: https://secure.phabricator.com/D16947
2016-11-25 15:30:50 -08:00
epriestley
ff3333548f Create and populate a stopwords table for InnoDB fulltext indexes to use in the future
Summary:
Ref T11741. InnoDB uses a stopwords table instead of a stopwords file.

During `storage upgrade`, synchronize the table from the stopwords file on disk.

Test Plan:
  - Ran `storage upgrade`.
  - Ran `select * from stopwords`, saw stopwords.
  - Added some garbage to the table.
  - Ran `storage upgrade`, saw it remove it.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11741

Differential Revision: https://secure.phabricator.com/D16940
2016-11-25 15:13:08 -08:00
epriestley
4da74166fe When storage is partitioned, refuse to serve requests unless web and databases agree on partitioning
Summary:
Ref T11044. One popular tool in a modern operations environment is Puppet. The primary purpose of this tool is to randomly revert hosts to older or different configurations.

Introducing an element of chaotic unpredictability into operations trains staff to be on high alert at all times, rather than lulled into complacency by predictability or consistency.

When Puppet reverts a Phabricator host's configuration to an older version, we might start writing data to a lot of crazy places where it shouldn't go. This will create a big sticky mess that is virtually impossible to undo, mostly because we'll get two files with ID 123 or two tasks with ID 456 or whatever else and good luck with that.

Instead, after changing the partition layout, require `bin/storage partition` to be run. This writes a copy of the config everywhere.

Then, when we start serving web requests, make sure every database has the exact same config. This will foil Puppet by refusing to run requests on hosts it has reverted.

Test Plan:
  - Changed partition configuration.
  - Ran Phabricator.
  - FOILED!
  - Ran `bin/storage partition` to sync config.
  - Things worked again.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11044

Differential Revision: https://secure.phabricator.com/D16910
2016-11-22 04:15:46 -08:00
Chad Little
ce0cb115ca Add Hero Image to Phame Post
Summary: Adds a headerimage and lets you set it on posts for added reverence. Is that a word?

Test Plan:
Add an image, see an image.

{F1923010}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16873
2016-11-15 13:44:47 -08:00
Chad Little
015ead6e7b Add subtitle to PhamePost
Summary: No view engine yet (adding header image next), but adds subtitle to display like PhameBlog

Test Plan: Add a subtitle, remove a subtitle.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16871
2016-11-15 12:02:03 -08:00
epriestley
e337029769 Allow users to mark themselves as "Available", "Busy" or "Away" while attending an event
Summary:
Ref T11816.

  - Now that we can do something meaningful with them, bring back the yellow dots for "busy".
  - Default to "busy" when attending events (we could make this "busy" for short events and "away" for long events or something).
  - Let users pick how to display their attending status on the event page.
  - Also show which event the user is attending since I had to mess with the cache code anyway. We can get rid of this again if it doesn't feel good.

Test Plan:
{F1904179}

{F1904180}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D16802
2016-11-04 16:55:44 -07:00
epriestley
3f2f81a1c8 Remove obsolete Calendar event date storage fields
Summary: Ref T11809. These have been replaced with more flexible storage that accommodates a wider range of behaviors, including those in the ICS format and RRULEs.

Test Plan:
  - Ran migration.
  - Viewed, created, edited events.
  - Grepped for all removed names/symbols.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11809

Differential Revision: https://secure.phabricator.com/D16789
2016-11-02 09:49:08 -07:00
epriestley
6982bded71 Remove ancient "Holiday" storage
Summary:
Ref T11809. This came out of Facebook many years ago for computing the number of business days that revisions had been stale.

We removed the little staleness marker a few months ago and haven't seen complaints about it.

If we did holidays now it would make sense to integrate them more directly with Calendar as real events, but I have no plans to pursue this anytime soon. It's easy enough to add the federal holidays manually (~5 minutes of work per year?) if you want them, and they're commentable/editable and you can add local holidays if you're not in the US.

Test Plan:
  - Ran `bin/storage upgrade -f`.
  - Grepped for `CalendarHoliday`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11809

Differential Revision: https://secure.phabricator.com/D16788
2016-11-01 16:03:44 -07:00
epriestley
6e6ae36dcf Add a skeleton for Calendar notifications
Summary:
Ref T7931. I'm going to do this separate from existing infrastructure because:

  - events start at different times for different users;
  - I like the idea of being able to batch stuff (send one email about several upcoming events);
  - triggering on ghost/recurring events is a real complicated mess.

This puts a skeleton in place that finds all the events we need to notify about and writes some silly example bodies to stdout, marking that we notified users so they don't get notified again.

Test Plan:
Ran `bin/calendar notify`, got a "great" notification in the command output.

{F1891625}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7931

Differential Revision: https://secure.phabricator.com/D16783
2016-11-01 10:41:15 -07:00
epriestley
b084efb362 Record a "series parent PHID" on Calendar events that retains relationships after forks
Summary:
When you edit "X and all future events", X becomes the new parent of an event series.

Currently, it loses its relationship to its original parent. Instead, retain that relationship -- it's separate from the normal "parent", but we can use it to make the UI more clear or tweak behaviors later.

This mostly just keeps us from losing/destroying data that we might need/want later.

Test Plan:
  - Ran migrations.
  - Cancelled "X and all future events", saw sensible-appearing beahvior in the database for "seriesParentPHID".

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16780
2016-10-31 15:30:34 -07:00
Chad Little
5fd79479ec Add a basic invoice view for printing to Phortune
Summary: Makes a more complete PDF looking invoice form for printing in Phortune.

Test Plan: Make an invoice, click print view, print.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16762
2016-10-29 17:46:47 -07:00
Chad Little
c3809b0d59 Add a Merchant logo to Phortune
Summary: Is a logo. For merchants.

Test Plan: Set a new logo, remove it. See on list.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T7607

Differential Revision: https://secure.phabricator.com/D16751
2016-10-28 13:56:35 -07:00
epriestley
09775279a9 Support arbitary event invitees when importing events
Summary:
Ref T10747. When we import a ".ics" file, represent any attendees as simple external references.

For consistency with other areas of the product, I've avoided disclosing email addresses. We'll try to get a real name if we can.

(We store addresses and could expose or use them later, or do some kind of masking junk like "epr...ley@g...l.com" which is utterly impossible to figure out.)

Test Plan: {F1888367}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10747

Differential Revision: https://secure.phabricator.com/D16759
2016-10-27 13:27:58 -07:00
epriestley
2d7f574b9d Allow Calendar imports to be configured with hourly or daily auto-updates
Summary:
Ref T10747. For URI-based (and, in the future, Google-based) imports, we can automatically refresh them periodically.

(In the general case there's no way to get a push notification for an ICS file, so we just have to do this every-so-often.)

Test Plan:
  - Set an ICS file to update hourly.
  - Used `bin/trigger fire --id ...` to fire it artificially.
  - Saw Calendar update.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10747

Differential Revision: https://secure.phabricator.com/D16752
2016-10-26 12:19:14 -07:00
Chad Little
a69ac888b3 Add Contact Information to Phortune Merchants
Summary: Part of making this look/feel/be more professional is having decent receipts for billing, including contact information (whatever we want to put in there). I'm not using this anywhere at the moment, but will.

Test Plan: Add Contact Info, see Contact Info. Also, why is Remarkup not rendering with line breaks? Seems to be a OneOff thing... anywho... bears!

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T7607

Differential Revision: https://secure.phabricator.com/D14125
2016-10-25 17:11:02 -07:00
Chad Little
dd25b2b48b Remove imagePHIDs column from ConpherenceThread
Summary: Ref T11730. Removes the unused column, seen no issues during past week migrations.

Test Plan: Run migration, check database no longer contains column.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11730

Differential Revision: https://secure.phabricator.com/D16711
2016-10-17 15:53:22 -07:00
epriestley
3d98558593 Add import log messages to Calendar imports
Summary: Ref T10747. When stuff goes wrong (or right) let the user know what happened.

Test Plan: {F1870139}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10747

Differential Revision: https://secure.phabricator.com/D16704
2016-10-14 07:57:48 -07:00
epriestley
2ab07ed29b Prepare for event imports in Calendar
Summary:
Ref T10747. Adds a bunch of stuff so we can keep track of which events we've imported from external sources.

This doesn't do anything yet: you can't actually import anything.

Test Plan:
  - Ran `bin/storage upgrade`.
  - Clicked "Imports", saw an empty wasteland.
  - Created/edited events.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10747

Differential Revision: https://secure.phabricator.com/D16696
2016-10-12 10:45:31 -07:00
Chad Little
0244ec3115 Add Room typeahead for Conpherence Search
Summary: Ref T3165. Builds an ngram table for Conpherence Room titles, allowing a tokenizer for searching a subset of rooms.

Test Plan: Say `Gabbert` in two different rooms, search all, see two rooms returned. Search specific room, see specific result.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T3165

Differential Revision: https://secure.phabricator.com/D16692
2016-10-12 08:38:45 -07:00
epriestley
49448a87c1 Rough in most of Calendar exports
Summary:
Ref T10747. Rough flow is:

  - Run a query.
  - Select a new "Export Events..." action.
  - This lets you define an "Export", which has a unique URL you can paste into Google Calendar or Calendar.app or whatever.

Most of this does nothing yet but here's the boilerplate.

Test Plan: Doesn't do anything yet.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10747

Differential Revision: https://secure.phabricator.com/D16675
2016-10-06 04:06:35 -07:00
epriestley
3164ff68db Convert Calendar Events to use RRULE frequency constants in various other places
Summary:
Ref T10747.

  - Store recurrence as RRULEs internally.
  - Use RRULE constants.
  - Migrate existing rules to RRULEs.

Test Plan: Ran migration, nothing seemed broken?

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10747

Differential Revision: https://secure.phabricator.com/D16670
2016-10-06 04:02:27 -07:00
epriestley
d3fc1800f8 Migrate Calendar away from stored-epoch fields
Summary:
Ref T10747. This deprecates "dateFrom", "dateTo", "allDayDateFrom", "allDayDateTo", and "recurrenceEndDate".

They are replaced with "utc*Epoch" fields (for querying) and CalendarDateTime objects (for start, end, until). These objects can represent the full range of dates and times expressible in ICS format, allowing us to import a wider range of ICS events.

Test Plan:
Ran migrations, viewed/edited Calendar, didn't catch anything catastrophcially broken.

This likely needs some followups, I'll keep it local for a bit until I'm confident I didn't break anything too catastrophically. I'm retaining the old data for now so we can likely fix things if it turns out there is some sort of issue.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10747

Differential Revision: https://secure.phabricator.com/D16664
2016-10-06 03:55:59 -07:00
epriestley
e042533375 Store "start", "end", and "until" event dates as CalendarDateTime objects
Summary:
Ref T10747. This does double-writes and starts generating/writing CalendarDateTimes.

This greater flexibility is necessary to support the full range of ICS-specifiable events, including "floating" events.

This doesn't do anything yet.

Test Plan: Created and edited events, verified sensible representations of corresponding datetimes appeared in the database.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10747

Differential Revision: https://secure.phabricator.com/D16661
2016-10-06 03:51:57 -07:00
epriestley
0ce7eacaf1 Introduce Calendar "UTC Epoch" columns for query windowing
Summary:
Ref T10747. Currently, Calendar events are mostly epoch-based and cheat a little bit for all-day events.

This already felt a little flimsy, and can't reasonably accommodate the full range of `.ics` events, which include "floating" events (e.g., occurs at 3PM regardless of timezone, like "Tea Time").

As a secondary issue, we identify instances of a recurring event by instance number (1, 2, 3, etc.). This can't accommodate the full range of `.ics` events, which include arbitrary additional "RDATE" events (e.g., recurrs every week, and also on these specific extra days).

However, we do need to store some epoch information so we can do query windowing: when the user looks at "October 2016", we want to select the smallest number of events that we can from the database initially, before refining them down to generate instances. We can't reasonably query the actual dates no matter how we store them because this depends on computing things like UNTIL, COUNT, initial dates, whether events are recurring or not, timezones, etc.

Instead, when we save an event compute the earliest second it occurs on in UTC and the latest second it occurs on in UTC. We can then query for a small superset of possible events in "October 2016" for any viewer pretty easily.

Also, start laying the groundwork for using fewer epochs in the rest of the code, and for reducing the role of sequence indexes (I plan to keep some sequences indexes around, probably, since they're nice in the UI, but not all child events will have indexes since there's no index for an RDATE event).

This doesn't migrate existing events yet or actually read these new columns -- that will come later once the new code is a little more solid.

Test Plan:
  - Ran `bin/storage upgrade`.
  - Created a new event.
  - Saved an existing event.
  - Viewed database, saw sensible-looking "UTC Epoch" values.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10747

Differential Revision: https://secure.phabricator.com/D16652
2016-10-06 03:46:03 -07:00
Chad Little
cc230ed98a Fix migration on conpherence images 2016-10-05 12:29:14 -07:00
Chad Little
d68c444ffa Convert Conpherence to use normal picture setting flows
Summary: This moves room pictures out of the dialog and into it's own PictureController. Also adds a standard image (and removes the "last person to chat" picture (though we could add that back. My plan is though that direct messages use auto use the other person's photo, after we have editengine and room pictures will have a plain, replaceable image.

Test Plan: Set a new room picture, remove a picture. Run migration, see old images properly set with new image.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11730

Differential Revision: https://secure.phabricator.com/D16669
2016-10-05 12:05:36 -07:00
epriestley
5d1359d78f Fix an issue where repository message counts would never reset
Summary:
Fixes T11705. I did not realize that `ON DUPLICATE KEY UPDATE` was order-dependent, so the "reset" clause of this `IF(...)` never actually worked.

Reorder it so we check if we're changing the message type //first//, then actually change the message type.

This makes the count reset properly when a failing repository succeeds, or a working repository fails.

Test Plan:
  - On `master`, forced a working repository to fail a `bin/repository update`, saw the message change types (expected) but keep the old count (wrong!).
  - With this patch, repeated the process and saw the count reset properly.
  - Ran the patch, verified counts reset to 0.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11705

Differential Revision: https://secure.phabricator.com/D16623
2016-09-28 15:02:26 -07:00
Josh Cox
32d660c08f Added a token_token table in anticipation of some data-driven tokens
Summary: Ref T11217. This just adds the table that we'll store tokens in. It doesn't make use of the table at all yet. This is mostly pulled from this diff (D16178). Specifically I mostly followed Evan's instructions related to the token table here: D16178#189120.

Test Plan: I ran `./bin/storage upgrade` successfully and there were no schema errors.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley, yelirekim

Maniphest Tasks: T11217

Differential Revision: https://secure.phabricator.com/D16621
2016-09-27 11:12:34 -04:00
Josh Cox
fc82118848 Expose conduit API methods for Phurl URLs
Summary: Fixes T10681. Adds a search API endpoint and an edit API endpoint for Phurl URLs. I still need to add the ability to search by name, alias, URL, and maybe description.

Test Plan: Test the methods through `/conduit/method/phurls.search/` and `/conduit/method/phurls.edit/`

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley, yelirekim

Maniphest Tasks: T10681

Differential Revision: https://secure.phabricator.com/D16600
2016-09-27 00:21:49 -04:00
Josh Cox
eea540c5e4 Endpoint+controller for a remarkup image proxy
Summary:
Ref T4190. Currently only have the endpoint and controller working. I added caching so subsequent attempts to proxy the same image should result in the same redirect URL. Still need to:

- Write a remarkup rule that uses the endpoint

Test Plan: Hit /file/imageproxy/?uri=http://i.imgur.com/nTvVrYN.jpg and are served the picture

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley, yelirekim

Maniphest Tasks: T4190

Differential Revision: https://secure.phabricator.com/D16581
2016-09-23 10:28:24 -04:00
epriestley
51f8ec4487 Add a default value for messageCount so writes from old tiers survive the
update query

Auditors: chad
2016-09-19 19:43:24 -07:00
epriestley
d3280c406d When repositories hit pull errors, stop updating them as frequently
Summary:
Ref T11665. Currently, when a repository hits an error, we retry it after 15s. This is correct if the error was temporary/transient/config-related (e.g., bad network or administrator setting up credentials) but not so great if the error is long-lasting (completely bad authentication, invalid URI, etc), as it can pile up to a meaningful amount of unnecessary load over time.

Instead, record how many times in a row we've hit an error and adjust backoff behavior: first error is 15s, then 30s, 45s, etc.

Additionally, when computing the backoff for an empty repository, use the repository creation time as though it was the most recent commit. This is a good proxy which gives us reasonable backoff behavior.

This required removing the `CODE_WORKING` messages, since they would have reset the error count. We could restore them (as a different type of message), but I think they aren't particularly useful since cloning usually doesn't take too long and there's more status information avilable now than there was when this stuff was written.

Test Plan:
  - Ran `bin/phd debug pull`.
  - Saw sensible, increasing backoffs selected for repositories with errors.
  - Saw sensible backoffs selected for empty repositories.

Reviewers: chad

Maniphest Tasks: T11665

Differential Revision: https://secure.phabricator.com/D16575
2016-09-19 17:29:56 -07:00
Chad Little
c1c5fbce21 Rebuild Conpherence
Summary:
Minor rebuild / redesign of Conpherence. Most of this is new UX and tossing out things like widgets, device fallbacks. I expect some of the UI to get more polished after next pass, but most everything here is in place.

 - Removed "Widgets", now just a single Participants pane
 - Added "Topic"
 - New header
 - Settings, Edit are action icons
 - Removed a lot of JS
 - Simplified CSS as much as I could

Test Plan:
Desktop, Tablet, Mobile. Adding and removing people. Setting new topics, new rooms.

{F1828662}

{F1828669}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16550
2016-09-14 18:51:32 -07:00
Josh Cox
9422596ebf Converted Paste language selection to a typeahead
Summary: Fixes T11532. The language selection for pastes is now a typeahead that is backed by `pygments.dropdown-choices`. There is still a bit of weirdness around making "auto-detection" the default state. To actually select a different language, you first need to remove the "auto detect" option that is pre-populated in a new paste. Other than that, it works as intended.

Test Plan:
Create a new paste with a file extension that can be auto-detected.
Created a new paste and manually selected the language
Edited a paste and changed the language.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley, yelirekim

Maniphest Tasks: T11532

Differential Revision: https://secure.phabricator.com/D16463
2016-08-30 00:41:47 +00:00
Josh Cox
9a52492f1b Added autopatch to remove ponder vote data
Summary: Fixes T9117. Adds a migration to remove ponder vote data.

Test Plan: I added a bunch of lines to phabricator_user.edge with type 18 and they were successfully removed by this patch

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley, yelirekim

Maniphest Tasks: T9117

Differential Revision: https://secure.phabricator.com/D16452
2016-08-25 15:49:42 -04:00
epriestley
e4c4724afd Migrate the "badcommit" table to use the less-hacky "hint" mechanism
Summary: Ref T11522. This migrates any "badcommit" data (which probably only exists at Facebook and on 1-2 other installs in the wild) to the new "hint" table.

Test Plan:
  - Wrote some bad commit annotations to the badcommit table.
  - Viewed them in the web UI and used `bin/repository reparse --change ...` to reparse them. Saw "this is bad" messages.
  - Ran migration, verified that valid "badcommit" rows were successfully migrated to become "hint" rows.
  - Viewed the new web UI and re-parsed the change, saw "unreadable commit" messages.
  - Viewed a good commit; reparsed a good commit.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11522

Differential Revision: https://secure.phabricator.com/D16435
2016-08-24 09:32:59 -07:00
epriestley
8a4fbcd8c0 Provide a new "hint" table for weird commits (rewritten, unreadable)
Summary:
Ref T11522. This provides storage for tracking rewritten commits (new feature) and unreadable commits (existing feature, but really hacky).

This doesn't do anything yet, just adds a table and a CLI tool for updating it. I'll document the tool once it works. You just pipe in some JSON, but I need to document the format.

Test Plan:
  - Piped JSON for "none", "rewritten" and "unreadable" hints into `bin/repository hint`.
  - Examined the database to see that the table was written properly.
  - Tried to pipe bad JSON in, invalid hint types, etc. Got reasonable human-readable error messages.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11522

Differential Revision: https://secure.phabricator.com/D16434
2016-08-24 09:31:46 -07:00
Josh Cox
3c62be6956 Add patch to remove conduit_connectionlog table (Fixes T9982)
Summary: Adds a schema patch that removes conduit_connectionlog. This table hasn't been used in 8ish months so it's probably safe to get rid of.

Test Plan: Apply the patch locally and confirm that the table does indeed get dropped.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: epriestley

Differential Revision: https://secure.phabricator.com/D16438
2016-08-23 03:57:14 -05:00
epriestley
e8083ad63a Increase the storage size for commit summaries
Summary:
Fixes T11453. Currently, commit message summaries are limited to 80 bytes. This may only be 20-40 characters for CJK languages or langauges with Cyrillic script.

Increase storage size to 255, then truncate to the shorter of 255 bytes or 80 glyphs. This preserves the same behavior for latin languages, but is less tight for Russian, etc.

Some minor additional changes:

  - Provide a way to ask "how much data fits in this column?" so we don't have to duplicate column lengths across summary checks or UI errors like "title too long".
  - Remove the `text80` datatype, since no other columns use it and we have no use cases (or likely use cases) for it.

Test Plan:
  - Made a commit with a Cyrillic title, saw reasonable summarization in UI:

{F1757522}

  - Added and ran unit tests.
  - Grepped for removed `SUMMARY_MAX_LENGTH` constant.
  - Grepped for removed `text80` data type.

Reviewers: avivey, chad

Reviewed By: avivey

Subscribers: avivey

Maniphest Tasks: T11453

Differential Revision: https://secure.phabricator.com/D16385
2016-08-10 11:12:45 -07:00
epriestley
08a19f35f0 Add basic search capabilities to Packages
Summary: Ref T8116. Add search-by-name and per-package / per-publisher search to Packages.

Test Plan: Searched publishers, packages, versions by name. Searched packages by publisher. Searched versions by package.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8116

Differential Revision: https://secure.phabricator.com/D16320
2016-07-27 12:27:12 -07:00
epriestley
ee37eca2e4 Add PackagesVersion
Summary:
Ref T8116. A version has:

  - a package (like "Arcanist") which it belongs to;
  - a name (like "v3.1.5").

The name is immutable and unique, like the package key and publisher key.

Policy stuff:

  - Versions have the exact same policies as their packages.
  - You must be able to edit a package to create new versions of it.

This is still entirely uninteresting.

Test Plan: {F1731703}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8116

Differential Revision: https://secure.phabricator.com/D16316
2016-07-27 12:24:32 -07:00
epriestley
704afea281 Add PackagesPackage
Summary:
Ref T8116. A package has:

  - a publisher (like "Phacility"), from the previous revision;
  - a name (like "Arcanist");
  - a package key (like "arcanist").

The package key is immutable, like the publisher key.

This gives a package a full key like "phacility/arcanist".

Policy stuff:

  - You must be able to view a publisher to view a package (currently, everyone can always see all publishers).
  - You must be able to edit a publisher to create a new package inside it.
  - Packages have separate view/edit permissions.

This still does nothing interesting.

Test Plan: {F1731663}

Reviewers: chad

Reviewed By: chad

Subscribers: eadler

Maniphest Tasks: T8116

Differential Revision: https://secure.phabricator.com/D16315
2016-07-27 12:22:57 -07:00
epriestley
9160da1afb Add a Packages application and PackagePublisher
Summary:
Ref T8116. Partially scavenged from D14152. This roughs in a new Packages application for Arcanist extensions and third-party applications, and adds a "Publisher" object.

A "Publisher" represents an individual or entity who is publishing a package, like "Phacility". It's explicitly //not// necessarily the original author -- just the primary entity vouching for the safety of the code.

A publisher just has a name and a unique key for now. For example, Phacility might have "Phacility" and "phacility", respectively.

Unique keys are immutable, e.g., the package "phacility/arcanist" will always be exactly the same package by exactly the same publisher.

Test Plan: {F1731621}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8116

Differential Revision: https://secure.phabricator.com/D16314
2016-07-27 12:21:57 -07:00
Aviv Eyal
b6bf0f6a3b Re-implement calendar.invite transactions
Summary:
Fix T11339.
Now, old and new are both simple lists of phids, and the rendering should make sense.

Test Plan: Viewed existing transaction with all 3 states.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin

Maniphest Tasks: T11339

Differential Revision: https://secure.phabricator.com/D16311
2016-07-22 18:03:28 +00:00
epriestley
e2b6912b9d Store "All Day" events in a way that is compatible with EditEngine
Summary:
Ref T11326. Normally, events occur at a specific epoch, independent of the viewer. For example, if we're having a meeting in 35 hours, every user who looks at the event will see that it starts 35 hours from now.

But when an event is "All Day", the start time and end time depend on the //viewer//. A day like "Christmas" does not start at the same time for everyone: it starts sooner if you're in a more-eastern timezone. Baiscally, an event on "July 15th" starts whenever "July 15th" starts for whoever is looking at it.

Previously, we stored these events by using the western-most and eastern-most timezones as the start and end times (the earliest possible start and latest possible end).

This worked OK, but we get into a bunch of trouble with EditEngine, mostly because each field can be updated individually now. We can't easily tell if an event is all-day or not when reading or updating the start time and end time, and making that easier would introduce a huge amount of complexity.

Instead, when we update the start or end time, we write //two// times:

  - The epoch timestamp of the time the user entered, which is the start time we will use if the event is a normal event.
  - The epoch timestamp of 12:00 AM in UTC on the same date as the //local// date the user entered. This is pretty much like just storing the date the user actually typed. This is what w'ell use if the event is an all-day event.

Then, no matter whether the event is later made all-day or not, we have all the information we need to display it correctly.

Test Plan:
  - Created and edited all-day events.
  - Migrated existing all-day events, which appeared to survive without problems. (Note that events all-day which were created or edited in the last couple of days `master` won't survive this mutation correctly and will need to be fixed.)
  - Created and edited normal, recurring, and recurring all-day events.
  - Swapped back to `stable`, created an event, specifically migrated it forward, made sure it survived with times intact.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11326

Differential Revision: https://secure.phabricator.com/D16305
2016-07-15 12:24:01 -07:00
epriestley
b6daa049de Rename Event "userPHID" to "hostPHID"
Summary: Ref T10909. Ref T9224. We label this field "Host" in the UI; make the storage format consistent.

Test Plan:
  - Viewed month view, day view, detail view of an event.
  - Created a new event, saw myself as the host.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T9224, T10909

Differential Revision: https://secure.phabricator.com/D16291
2016-07-13 15:39:55 -07:00