1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-16 10:30:56 +01:00
Commit graph

955 commits

Author SHA1 Message Date
epriestley
95dd9dbf43 Make Applications extend LiskDAO
Summary:
Ref T11476. This is a bit hacky, but makes `Application` extend `LiskDAO` so we can apply transactions to it with an `Editor` class.

Also fixes schema stuff so builds should produce a clean bill of health again.

This might only get you slightly further, yell if you run into more trouble.

Test Plan:
  - Ran `bin/storage upgrade -f` and got no warnings.
  - Browsed around, nothing exploded?

Reviewers: amckinley

Reviewed By: amckinley

Maniphest Tasks: T11476

Differential Revision: https://secure.phabricator.com/D17738
2017-04-19 16:06:14 -07:00
Austin McKinley
febd68039f Add initial infrastructure for adding ModularTransaction support to Application config changes
Summary: Part of the groundwork for T11476.

Test Plan: ran `./bin/storage upgrade` and observed expected DB tables

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11476

Differential Revision: https://secure.phabricator.com/D17736
2017-04-19 15:44:57 -07:00
epriestley
76d0b67d91 Remove "dateTouched" from ConpherenceParticipant
Summary:
Pathway to D17685. This column is (mostly) a denormalization of `dateModified` on the thread.

Just use a JOIN instead.

This isn't //exactly// the same: we'll bump threads to the top now for non-message changes (e.g., a topic or title change). That seems fine, but we could put a `lastMessageDate` on Thread later if we want to refine it.

Also got rid of a lot of other unused stuff. There's a big garbage TODO here, I'll fix that in the next change.

Test Plan:
  - Grepped for `dateTouched`.
  - Grepped for `participantCursor`.
  - Grepped for `ConpherenceParticipantQuery::LIMIT`.
  - Looked for callsites to `setOrder()`, found none.
  - Added a message to an older thread, saw it bump up to the top.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17731
2017-04-19 13:58:54 -07:00
epriestley
0a335f91cd Remove "participationStatus" from ConpherenceParticipant
Summary:
Pathway to D17685. This column is a very complicated cache of: is participant.messageCount equal to thread.messageCount?

We can just ask this question with a JOIN instead and simplify things dramatically.

Test Plan:
  - Ran migration.
  - Browsed around.
  - Sent a message, saw unread count go up.
  - Read the message, saw unread count go down.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17730
2017-04-19 13:58:42 -07:00
epriestley
c96e17f5ea Remove "behindTransactionPHID" from ConpherenceParticipant
Summary: Pathway to D17685. Nothing reads this field and it has no use or value.

Test Plan:
  - Ran migration.
  - Grepped for `behindTransactionPHID`.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17729
2017-04-19 13:58:29 -07:00
Austin McKinley
be00264ae7 Make daemons perform file deletion
Summary:
Deletion is a possibly time-intensive process, especially with large
files that are backed by high-latency, chunked storage (such as
S3). Even ~200mb objects take minutes to delete, which makes for an
unhappy experience. Fixes T10828.

Test Plan:
Delete a large file, and stare in awe of the swiftness with
which I am redirected to the main file application.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: thoughtpolice, Korvin

Maniphest Tasks: T10828

Differential Revision: https://secure.phabricator.com/D15743
2017-04-18 11:09:41 -07:00
Austin McKinley
976fbee877 Implement ngram search for File objects
Summary: Follows the outline in D15656 for implementing ngram search for names of File objects. Also created FileFullTextEngine, because without implementing `PhabricatorFulltextInterface`, `./bin/search` complains that `File` is not an indexable type.

Test Plan:
  - ran `./bin/storage upgrade` to apply the schema change
  - confirmed the presence of a new `file_filename_ngrams` table
  - added a couple file objects
  - ran `bin/search index --type file --force`
  - confirmed the presence of rows in `file_filename_ngrams`
  - did a few keyword searches and saw expected results

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T8788

Differential Revision: https://secure.phabricator.com/D17702
2017-04-17 17:37:20 -07:00
Chad Little
5587abf04c Remove recentParticipants from ConpherenceThread
Summary: We no longer display this any more in the UI, so go ahead and remove the callsites and db column.

Test Plan: New Room, with and without participants.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17683
2017-04-13 13:55:08 -07:00
epriestley
3245e74f16 Show users how fulltext search queries are parsed and executed; don't query stopwords or short tokens
Summary:
Depends on D17670. Fixes T12137. Fixes T12003. Ref T2632.

This shows users a readout of which terms were actually searched for.

This also drops those terms from the query we submit to the backend, dodging the weird behaviors / search engine bugs in T12137.

This might need some design tweaking.

Test Plan: {F4899825}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12137, T12003, T2632

Differential Revision: https://secure.phabricator.com/D17672
2017-04-12 19:07:54 -07:00
Chad Little
a9845b0b1d Remove picture crop transaction from Conpherence
Summary: Fixes T11730. Removes an old transaction that hasn't been used in a year.

Test Plan: Run sql, check various rooms.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T11730

Differential Revision: https://secure.phabricator.com/D17666
2017-04-12 14:18:54 -07:00
epriestley
d4d46501ed Add a repair migration to fix legacy Calendar events without "utcInstanceEpoch"
Summary: Fixes T12488. Some events appear to have survived earlier migrations without getting completely fixed. Fix them.

Test Plan:
  - Ran migration locally with `bin/storage upgrade` (but: I could not reproduce this problem locally).
  - Ran migration in production and saw ICS import stop fataling.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12488

Differential Revision: https://secure.phabricator.com/D17642
2017-04-10 08:47:07 -07:00
epriestley
3d816e94df Rename "PhabricatorHash::digest()" to "weakDigest()"
Summary: Ref T12509. This encourages code to move away from HMAC+SHA1 by making the method name more obviously undesirable.

Test Plan: `grep`, browsed around.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D17632
2017-04-06 15:43:33 -07:00
epriestley
d450a08890 Support HMAC+SHA256 with automatic key generation and management
Summary:
Ref T12509. This adds support for HMAC+SHA256 (instead of HMAC+SHA1). Although HMAC+SHA1 is not currently broken in any sense, SHA1 has a well-known collision and it's good to look at moving away from HMAC+SHA1.

The new mechanism also automatically generates and stores HMAC keys.

Currently, HMAC keys largely use a per-install constant defined in `security.hmac-key`. In theory this can be changed, but in practice essentially no install changes it.

We generally (in fact, always, I think?) don't use HMAC digests in a way where it matters that this key is well-known, but it's slightly better if this key is unique per class of use cases. Principally, if use cases have unique HMAC keys they are generally less vulnerable to precomputation attacks where an attacker might generate a large number of HMAC hashes of well-known values and use them in a nefarious way. The actual threat here is probably close to nonexistent, but we can harden against it without much extra effort.

Beyond that, this isn't something users should really have to think about or bother configuring.

Test Plan:
  - Added unit tests.
  - Used `bin/files integrity` to verify, strip, and recompute hashes.
  - Tampered with a generated HMAC key, verified it invalidated hashes.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12509

Differential Revision: https://secure.phabricator.com/D17630
2017-04-06 15:42:59 -07:00
epriestley
aea46e55da Fix an issue where "Request Review" of a fully-accepted revision would transition to "Accepted"
Summary:
Ref T10967. This is explained in more detail in T10967#217125

When an author does "Request Review" on an accepted revision, void (in the sense of "cancel out", like a bank check) any "accepted" reviewers on the current diff.

Test Plan:
  - Create a revision with author A and reviewer B.
  - Accept as B.
  - "Request Review" as A.
  - (With sticky accepts enabled.)
  - Before patch: revision swithced back to "accepted".
  - After patch: the earlier review is "voided" by te "Request Review", and the revision switches to "Review Requested".

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967

Differential Revision: https://secure.phabricator.com/D17566
2017-03-28 11:50:15 -07:00
Mukunda Modell
e41c25de50 Support multiple fulltext search clusters with 'cluster.search' config
Summary:
The goal is to make fulltext search back-ends more extensible, configurable and robust.

When this is finished it will be possible to have multiple search storage back-ends and
potentially multiple instances of each.

Individual instances can be configured with roles such as 'read', 'write' which control
which hosts will receive writes to the index and which hosts will respond to queries.

These two roles make it possible to have any combination of:

* read-only
* write-only
* read-write
* disabled

This 'roles' mechanism is extensible to add new roles should that be needed in the future.

In addition to supporting multiple elasticsearch and mysql search instances, this refactors
the connection health monitoring infrastructure from PhabricatorDatabaseHealthRecord and
utilizes the same system for monitoring the health of elasticsearch nodes. This will
allow Wikimedia's phabricator to be redundant across data centers (mysql already is,
elasticsearch should be as well).

The real-world use-case I have in mind here is writing to two indexes (two elasticsearch clusters
in different data centers) but reading from only one. Then toggling the 'read' property when
we want to migrate to the other data center (and when we migrate from elasticsearch 2.x to 5.x)

Hopefully this is useful in the upstream as well.

Remaining TODO:

* test cases
* documentation

Test Plan:
(WARNING) This will most likely require the elasticsearch index to be deleted and re-created due to schema changes.

Tested with elasticsearch versions 2.4 and 5.2 using the following config:

```lang=json
  "cluster.search": [
    {
      "type": "elasticsearch",
      "hosts": [
        {
          "host": "localhost",
          "roles": { "read": true, "write": true }
        }
      ],
      "port": 9200,
      "protocol": "http",
      "path": "/phabricator",
      "version": 5
    },
    {
      "type": "mysql",
      "roles": { "write": true }
     }
  ]

Also deployed the same changes to Wikimedia's production Phabricator instance without any issues whatsoever.
```

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley, #blessed_reviewers

Subscribers: Korvin, epriestley

Tags: #elasticsearch, #clusters, #wikimedia

Differential Revision: https://secure.phabricator.com/D17384
2017-03-26 08:16:47 +00:00
epriestley
aa91dc992e Record which user accepted on behalf of packages/owners reviewers
Summary:
Ref T12271. Don't do anything with this yet, but store who accepted/rejected/whatever on behalf of reviewers.

In the future, we could use this to render stuff like "Blessed Committers (accepted by epriestley)" or whatever. I don't know that this is necessarily super useful, but it's easy to track, seems likely to be useful, and would be a gigantic pain to backfill later if we decide we want it.

Test Plan: Accepted/rejected a revision, saw reviewers update appropriately.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12271

Differential Revision: https://secure.phabricator.com/D17537
2017-03-22 14:26:37 -07:00
epriestley
8ad5d28686 Migrate old reviewer edges to new storage
Summary:
Ref T10967. We still have double writes, so all reviewers are being written to both old and new storage. This migrates all the data in the old storage to the new storage, so both storage tables should have a complete set of data and be getting identical updates as we move forward.

After this, I can move readers over one at a time and eventually get rid of the old writes and old storage.

This loads all of the edge data into memory in a big chunk. I reached out to one install to get some more information about their data size. Ours is quite manageable and I think even large installs will probably fit into memory, but we can do this in chunks if not.

However, because the Edge table doesn't have an `id` column, we can't use either the `RawMigrationIterator` or the `MigrationIterator`, and would need to write a new `EdgeMigrationIterator`. This isn't tons of work but might not be necessary.

Test Plan: Ran the migration locally, spot-checked the results in the database for sanity and correctness.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967

Differential Revision: https://secure.phabricator.com/D17515
2017-03-20 16:44:40 -07:00
epriestley
794b456530 Store "last comment" and "last action" diffs on reviewers
Summary:
Ref T10967. We have a "commented" state to help reviewers get a better sense of who is part of a discussion, and a "last action" state to help distinguish between "accept" and "accepted an older version", for the purposes of sticky accepts and as a UI hint.

Currently, these are first-class states, partly beacuse we were somewhat limited in what we could do with edges. However, a more flexible way to represent them is as flags separate from the primary state flag.

In the new storage, write them as separate state information: `lastActionDiffPHID` stores the Diff PHID of the last review action (accept, reject, etc). `lastCommentDiffPHID` stores the Diff PHID of the last comment (top-level or inline).

Test Plan: Applied storage changes, commented and acted on a revision. Saw appropriate state reflected in the database.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967

Differential Revision: https://secure.phabricator.com/D17514
2017-03-20 16:44:05 -07:00
epriestley
7626ec0ce1 Correct an issue where "View Raw File" in Differential generated a file with overbroad permissions
Summary:
Via HackerOne. When you view a raw file in Differential, we currently generate a permanent file with default permissions. This may be incorrect: default permissions may be broader than the diff's permissions.

The other three methods of downloading/viewing raw files ("Download" in Diffusion and Differential, "View Raw" in Diffusion and Differential) already apply policies correctly and generate temporary files. However, this workflow was missed when other workflows were updated.

Beyond updating the workflow, delete any files we've generated in the past. This wipes the slate clean on any security issues and frees up a little disk space.

Test Plan:
  - Ran migration script, saw existing files get purged.
  - Did "View Raw File", got a new file.
  - Verified that the file was temporary and properly attached to the diff, with "NO ONE" permissions.
  - Double-checked that Diffusion already runs policy logic correctly and applies appropriate policies.
  - Double-checked that "Download Raw Diff" in Differential already runs policy logic correctly.
  - Double-chekced that "Download Raw Diff" in Diffusion already runs policy logic correctly.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17504
2017-03-16 09:51:48 -07:00
epriestley
251ee9b660 Add dedicated "reviewers" storage to Differential and do double writes
Summary:
Ref T10967. This is an incremental step toward removing "reviewers" back to a dedicated storage table so we can handle changes like T11050.

This adds the storage table, and starts doing double writes to it (so new or updated reviewers write to both the old edge table and the new "reviewers" table).

Then we can do a migration, swap readers over one at a time, and eventually remove the old write and old storage and then implement new features.

This change has no user-facing impact, it just causes us to write new data to two places instead of one.

This is not completely exhaustive: the Herald "Add Reviewers" action is still doing a manual EDGE transaction. I'll clean that up next and do another pass to look for anything else I missed.

This is also a bit copy/pastey for now but the logic around "RESIGN" is a little different in the two cases until T11050. I'll unify it in future changes.

Test Plan:
  - Did a no-op edit.
  - Did a no-op comment.
  - Added reviewers.
  - Removed reviewers.
  - Accepted and rejected revisions.

After all of these edits, did a `SELECT * FROM differential_reviewer` manually and saw consistent-looking rows in the database.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10967

Differential Revision: https://secure.phabricator.com/D17495
2017-03-14 11:51:51 -07:00
Chad Little
f2e013c2e9 Prep user table for default images
Summary: Ref T10319. Adds in database columns for upcoming default generated avatar support.

Test Plan: Ran storage upgrade, log into local site to verify it didn't blow up.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T10319

Differential Revision: https://secure.phabricator.com/D17459
2017-03-04 08:18:07 -08:00
epriestley
dc7ecf5875 Add "subtype" storage to Maniphest tasks
Summary: Ref T12314. Provides a field on tasks for storing subtypes. Does nothing interesting yet.

Test Plan:
  - Ran storage upgrade.
  - Created some tasks.
  - Looked in the database.
  - Used Conduit to query some tasks.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12314

Differential Revision: https://secure.phabricator.com/D17441
2017-03-02 04:17:47 -08:00
epriestley
91ef237290 Add a "subtype" field to EditEngine forms
Summary:
Ref T12314. This adds storage so EditEngine forms can later be marked as edit fields for particular types of objects (like an "animal edit form" vs a "plant edit form").

We'll take you to the right edit form when you click "Edit" by selecting among forms with the same subtype as the task.

This doesn't do anything very interesting on its own.

Test Plan:
  - Ran `bin/storage upgrade`.
  - Verified database got the field with proper values.
  - Created a new form, checked the database.
  - Ran unit tests.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12314

Differential Revision: https://secure.phabricator.com/D17439
2017-03-02 04:16:27 -08:00
epriestley
ad01e26af7 Redesign Home/Profile/Projects side navigation
Summary: Ref T11957. Needs some more polish, but I think everything here is square.

Test Plan: Add personal/global items to home, test mobile. Test workboards / colors.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: 20after4, rfreebern, Korvin

Maniphest Tasks: T11957

Differential Revision: https://secure.phabricator.com/D17259
2017-01-31 08:59:01 -08:00
Chad Little
58c857a681 Remove motivator panel
Summary: Removes the often funny, but never really used but will cause us bug reports someday.... cat facts.

Test Plan: Install cat facts, run storage upgrade, see no cat facts in menu.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T12126

Differential Revision: https://secure.phabricator.com/D17233
2017-01-19 14:55:19 -08:00
epriestley
11f3a8cfca Force 'changes' to a string in hunk migration
Summary: Fixes T12090. In obscure situations lost to the mists of time, the `changes` column could be `null`. Force a string cast so the migration finishes, even though these changesets are likely meaningless.

Test Plan:
I did a force-reapply as a sanity check:

```
$ ./bin/storage upgrade -f --apply phabricator:20161213.diff.01.hunks.php
```

That went cleanly; it would only have caught dramatic errors, but I didn't completely butcher things.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T12090

Differential Revision: https://secure.phabricator.com/D17168
2017-01-10 12:28:09 -08:00
epriestley
2dfe79cfc7 When updating revisions in response to commits, reuse previously generated diffs
Summary:
Fixes T10968. In rare situations, we can generate a diff, then hit an error which causes this update to fail.

When it does, we tend to get stuck in a loop creating diffs, which can fill the database up with garbage. We saw this once in the Phacility cluster, and one instance hit it, too.

Instead: when we create a diff, keep track of which commit we generated it from. The next time through, reuse it if we already built it.

Test Plan:
  - Used `bin/differential attach-commit <commit> <revision>` to hit this code.
  - Simulated a filesystem write failure, saw the diff get reused.
  - Also did a normal update, which worked properly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10968

Differential Revision: https://secure.phabricator.com/D17164
2017-01-09 12:13:44 -08:00
Chad Little
8a85ee7c15 Add CustomPHID to PhabricatorProfileMenuEngineConfiguration
Summary: Ref T5867, adds a customPHID field, nullable, and lets you query by it... i think? Not fully able to grok all the EditEngine stuff, but I think this is the right place for the query.

Test Plan: Not wired to anything, but pulling up project menu, editing, all still works.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Maniphest Tasks: T5867

Differential Revision: https://secure.phabricator.com/D17149
2017-01-07 10:49:54 -08:00
Sébastien Santoro
178466b826 Fix 20160122.project.1.boarddefault.php database upgrade patch
Summary: Fixes T12031.

Test Plan: Upgraded a December 2015 Phabricator installation to current schema.

Reviewers: #blessed_reviewers, epriestley

Reviewed By: #blessed_reviewers, epriestley

Subscribers: Korvin, epriestley

Maniphest Tasks: T12031

Differential Revision: https://secure.phabricator.com/D17094
2016-12-19 12:52:55 -08:00
Chad Little
c6bdd2c56b Add Ngram support to Dashboards / Panels
Summary: Build ngram indexs, adds search by name capability.

Test Plan: Search for a dashboard by partial name, search for a panel by partial name.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17090
2016-12-16 12:09:06 -08:00
epriestley
3d8d98bd8d Probably (?) fix two very old project slug migrations
Summary: Fixes T12020. These callsites to `getPhrictionSlug()` were missed when that method was removed. They're very old (early 2014, late 2011).

Test Plan:
These are tricky to test because the migrations are so ancient, but `bin/storage upgrade --force --apply phabricator:20140521.projectslug.2.mig.php` gave me //plausible// results.

The other migration is so ancient that it can't apply to a modern database so I'm just kind of winging that one. We probably have essentially no installs which will ever apply it again, though.

Reviewers: chad, avivey

Reviewed By: avivey

Maniphest Tasks: T12020

Differential Revision: https://secure.phabricator.com/D17060
2016-12-14 14:14:21 -08:00
epriestley
c67e87c809 Force hunk storage migration to the modern format
Summary:
Ref T8475. This forces installs to migrate hunks to the modern format.

We stopped writing to the legacy format a very long time ago (2+ years?) without issues.

This doesn't destroy any data. T8623 has guidance and I'll publish more changelog guidance.

Test Plan: Faked some legacy data and migrated it.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T8475

Differential Revision: https://secure.phabricator.com/D17039
2016-12-13 14:34:08 -08:00
epriestley
9d2feacfab Rename a missed Profile -> MenuItem
Summary: Didn't grep this good enough.

Test Plan: `bin/storage upgrade -f --apply ..`, got a clean apply.

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D17046
2016-12-13 14:27:00 -08:00
Chad Little
26127b9c5f Allow Dashboards to set an icon
Summary: Allows users set an icon (for reuse on upcoming home) for their dashboard based on 16 descriminating choices.

Test Plan: Create a new dashboard, set new icon. Edit an existing dashboard, set icon.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17042
2016-12-13 11:30:22 -08:00
Chad Little
c03a412d5c Add authorPHID to Dashboard Panels
Summary: Adds authorPHID to panels so we can default to the panels you made.

Test Plan: Run upgrade, visit manage panels, see my panels. Create a new panel. Edit a panel.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17036
2016-12-13 10:07:16 -08:00
epriestley
776a1f3aec Rename dashboard author patches so that the backfill patch happens
after the column it is adding is created

Auditors: chad
2016-12-12 15:44:28 -08:00
Chad Little
59f3b5125d Add authorPHID to Dashboards
Summary: Adds an authorPHIDs, populates olds ones.

Test Plan: Make a new Dashboard, see that I created it.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D17022
2016-12-12 15:26:43 -08:00
epriestley
8480776ccd Rename "ProfilePanelConfiguration" to "ProfileMenuItemConfiguration"
Summary:
Ref T11957. This renames the Configuration storage, transaction, query, and PHID type.

No rename on the actual menu item types yet, that's next (and should be the end of this, I think).

Test Plan:
  - Viewed projects.
  - Viewed profiles.
  - Edited a project menu.
  - Grepped for all renamed symbols, I think?

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11957

Differential Revision: https://secure.phabricator.com/D17027
2016-12-11 11:44:22 -08:00
epriestley
29a3cd5121 Add "Manual Activities", to tell administrators to rebuild the search index
Summary:
Ref T11922. After updating to HEAD of `master`, you need to manually rebuild the index. We don't do this during `bin/storage upgrade` because it can take a very long time (`secure.phabricator.com` took roughly an hour) and can happen while Phabricator is running.

However, if we don't warn users about this they'll just get a broken index unless they go read the changelog (or file an issue, then we tell them to go read the changelog).

This adds a very simple table for notes to administrators so we can write a "you need to go rebuild the index" note, then adds one.

Administrators clear the note by completing the activity and running `bin/config done reindex`. This isn't automatic because there are various strategies you can use to approach the issue, which I'll discuss in greater detail in the linked documentation.

Also, fix an issue where `bin/storage upgrade --apply <patch>` could try to re-mark an already-applied patch as applied.

Test Plan:
  - Ran storage ugrades.
  - Got instructions to rebuild search index.
  - Cleared instructions with `bin/config done reindex`.

Reviewers: chad

Reviewed By: chad

Subscribers: avivey

Maniphest Tasks: T11922

Differential Revision: https://secure.phabricator.com/D16965
2016-11-30 11:23:54 -08:00
epriestley
7c5b5327c8 Use stemming in the MySQL fulltext search engine
Summary:
Ref T6740. When we index a document, also save a copy of the stemmed version.

When querying, search the combined corpus for the terms.

(We may need to tune this a bit later since it's possible for literal, quoted terms to match in the stemmed section, but I think this wil rarely cause issues in practice.)

A downside here is that search sort of breaks if you upgrade into this and don't reindex. I wasn't able to find a way to issue the query that remained compatible with older indexes and didn't have awful performance, so my plan is:

  - Put this on `secure`.
  - Rebuild the index.
  - If things look good after a couple of days, add a way that we can tell people they need to rebuild the search index with a setup warning.

We might get some reports between now and then, but if this is super awful we should know by the end of the weekend.

Test Plan:
WOW AMAZING

{F2021466}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T6740

Differential Revision: https://secure.phabricator.com/D16947
2016-11-25 15:30:50 -08:00
epriestley
ff3333548f Create and populate a stopwords table for InnoDB fulltext indexes to use in the future
Summary:
Ref T11741. InnoDB uses a stopwords table instead of a stopwords file.

During `storage upgrade`, synchronize the table from the stopwords file on disk.

Test Plan:
  - Ran `storage upgrade`.
  - Ran `select * from stopwords`, saw stopwords.
  - Added some garbage to the table.
  - Ran `storage upgrade`, saw it remove it.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11741

Differential Revision: https://secure.phabricator.com/D16940
2016-11-25 15:13:08 -08:00
epriestley
4da74166fe When storage is partitioned, refuse to serve requests unless web and databases agree on partitioning
Summary:
Ref T11044. One popular tool in a modern operations environment is Puppet. The primary purpose of this tool is to randomly revert hosts to older or different configurations.

Introducing an element of chaotic unpredictability into operations trains staff to be on high alert at all times, rather than lulled into complacency by predictability or consistency.

When Puppet reverts a Phabricator host's configuration to an older version, we might start writing data to a lot of crazy places where it shouldn't go. This will create a big sticky mess that is virtually impossible to undo, mostly because we'll get two files with ID 123 or two tasks with ID 456 or whatever else and good luck with that.

Instead, after changing the partition layout, require `bin/storage partition` to be run. This writes a copy of the config everywhere.

Then, when we start serving web requests, make sure every database has the exact same config. This will foil Puppet by refusing to run requests on hosts it has reverted.

Test Plan:
  - Changed partition configuration.
  - Ran Phabricator.
  - FOILED!
  - Ran `bin/storage partition` to sync config.
  - Things worked again.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11044

Differential Revision: https://secure.phabricator.com/D16910
2016-11-22 04:15:46 -08:00
Chad Little
ce0cb115ca Add Hero Image to Phame Post
Summary: Adds a headerimage and lets you set it on posts for added reverence. Is that a word?

Test Plan:
Add an image, see an image.

{F1923010}

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16873
2016-11-15 13:44:47 -08:00
Chad Little
015ead6e7b Add subtitle to PhamePost
Summary: No view engine yet (adding header image next), but adds subtitle to display like PhameBlog

Test Plan: Add a subtitle, remove a subtitle.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16871
2016-11-15 12:02:03 -08:00
epriestley
e337029769 Allow users to mark themselves as "Available", "Busy" or "Away" while attending an event
Summary:
Ref T11816.

  - Now that we can do something meaningful with them, bring back the yellow dots for "busy".
  - Default to "busy" when attending events (we could make this "busy" for short events and "away" for long events or something).
  - Let users pick how to display their attending status on the event page.
  - Also show which event the user is attending since I had to mess with the cache code anyway. We can get rid of this again if it doesn't feel good.

Test Plan:
{F1904179}

{F1904180}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11816

Differential Revision: https://secure.phabricator.com/D16802
2016-11-04 16:55:44 -07:00
epriestley
3f2f81a1c8 Remove obsolete Calendar event date storage fields
Summary: Ref T11809. These have been replaced with more flexible storage that accommodates a wider range of behaviors, including those in the ICS format and RRULEs.

Test Plan:
  - Ran migration.
  - Viewed, created, edited events.
  - Grepped for all removed names/symbols.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11809

Differential Revision: https://secure.phabricator.com/D16789
2016-11-02 09:49:08 -07:00
epriestley
6982bded71 Remove ancient "Holiday" storage
Summary:
Ref T11809. This came out of Facebook many years ago for computing the number of business days that revisions had been stale.

We removed the little staleness marker a few months ago and haven't seen complaints about it.

If we did holidays now it would make sense to integrate them more directly with Calendar as real events, but I have no plans to pursue this anytime soon. It's easy enough to add the federal holidays manually (~5 minutes of work per year?) if you want them, and they're commentable/editable and you can add local holidays if you're not in the US.

Test Plan:
  - Ran `bin/storage upgrade -f`.
  - Grepped for `CalendarHoliday`.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T11809

Differential Revision: https://secure.phabricator.com/D16788
2016-11-01 16:03:44 -07:00
epriestley
6e6ae36dcf Add a skeleton for Calendar notifications
Summary:
Ref T7931. I'm going to do this separate from existing infrastructure because:

  - events start at different times for different users;
  - I like the idea of being able to batch stuff (send one email about several upcoming events);
  - triggering on ghost/recurring events is a real complicated mess.

This puts a skeleton in place that finds all the events we need to notify about and writes some silly example bodies to stdout, marking that we notified users so they don't get notified again.

Test Plan:
Ran `bin/calendar notify`, got a "great" notification in the command output.

{F1891625}

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T7931

Differential Revision: https://secure.phabricator.com/D16783
2016-11-01 10:41:15 -07:00
epriestley
b084efb362 Record a "series parent PHID" on Calendar events that retains relationships after forks
Summary:
When you edit "X and all future events", X becomes the new parent of an event series.

Currently, it loses its relationship to its original parent. Instead, retain that relationship -- it's separate from the normal "parent", but we can use it to make the UI more clear or tweak behaviors later.

This mostly just keeps us from losing/destroying data that we might need/want later.

Test Plan:
  - Ran migrations.
  - Cancelled "X and all future events", saw sensible-appearing beahvior in the database for "seriesParentPHID".

Reviewers: chad

Reviewed By: chad

Differential Revision: https://secure.phabricator.com/D16780
2016-10-31 15:30:34 -07:00
Chad Little
5fd79479ec Add a basic invoice view for printing to Phortune
Summary: Makes a more complete PDF looking invoice form for printing in Phortune.

Test Plan: Make an invoice, click print view, print.

Reviewers: epriestley

Reviewed By: epriestley

Subscribers: Korvin

Differential Revision: https://secure.phabricator.com/D16762
2016-10-29 17:46:47 -07:00