1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-12-11 08:06:13 +01:00
Commit graph

78 commits

Author SHA1 Message Date
epriestley
d2e5afb095 Use application PHIDs in Releeph, plus more
Summary:
Ref T2715. Ref T3551. Ref T603. This does a few things, but they're all sort of small:

  - We commonly use a `getX()` / `attachX()` pattern, but have very similar code in the `getX()` method every time. Provide a convenience method to make this pattern easier to write.
  - We use `willFilterPage()` in many queries, but it currently is called with zero or more results. This means we have a lot of "if no results, return nothing" boilerplate. Make it call only for one or more results.
  - Implement `PhabricatorPolicyInterface` on `ReleephBranch`. A branch has the same policy as its project.
  - Implement `ReleephBranchQuery`.
  - Move the branch PHID type to application PHID infrastructure.

Test Plan: Browsed Releeph. Used `phid.query` to query branch PHIDs.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T603, T2715, T3551

Differential Revision: https://secure.phabricator.com/D6512
2013-07-22 12:17:32 -07:00
Levi Jackson
d27e7c52b2 Add explicit mysql.port configuration
See: https://github.com/facebook/phabricator/pull/356

Reviewed by: epriestley
2013-07-14 16:06:23 -07:00
epriestley
8e8057c6fe Add LiskRawMigrationIterator
Summary: Ref T2222. I need this to migrate the Differential comment tables, because they are large. We have the similar `LiskMigrationIterator` already, but it won't work here because I intend to destroy the original objects after migrating them.

Test Plan:
Wrote a script to iterate over the `differential_comment` table, got reasonable output:

{P869}

Reviewers: btrahan

Reviewed By: btrahan

CC: chad, aran

Maniphest Tasks: T2222

Differential Revision: https://secure.phabricator.com/D6264
2013-06-21 12:55:07 -07:00
Jakub Vrana
9da6c9ff15 Delete LiskDAO::IDS_PHID
Test Plan: Searched for `IDS_PHID` and loaded homepage.

Reviewers: epriestley, edward

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D5619
2013-04-08 20:46:54 -07:00
epriestley
176ee9a889 Fix {Fnnn} rule in Remarkup
Summary:
Remarkup rule callbacks now get SafeHTML matches instead of string matches. If they call:

  $some_lisk_dao->load($matches[1]);

..as is the case with the `{F123}` rule, we reject the SafeHTML as an invalid ID and return null.

Allow load() to string convert any object (which will either succeed or fatal in an obviously-broken way).

(Long ago we threw instead of returning null here, but it meant we had to do a lot of redundant checks.)

Test Plan: `{F123}` shows an image again. `{C1}` embeds a countdown.

Reviewers: vrana, chad

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D4961
2013-02-14 13:09:19 -08:00
Nick Pellegrino
3e6fa43658 getConfigEnv fails fast when key is not found and no default value is given.
Summary:
T2345
getConfig throws an Exception when the key does not exist.
Also removes dead code that throws an Exception.

Test Plan:
Reloaded the Phabricator home page.  In the process, found
2 Exceptions thrown due to nonexistent keys.  After addressing these problems,
the home page loads without Exceptions.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D4541
2013-01-19 12:11:28 -08:00
epriestley
0902543fc8 Port MySQL settings to PHP
Summary:
  - Ports MySQL settings to PHP.
  - Removes "mysql.retries" -- this existed only because Magic Numbers Are Bad, but there is no concievable reason it should ever be set to anything other than 3.
  - Introduced "Hidden" config, which isn't visible from the web (for SaaS, we'll just mark anything with secret keys as "hidden").
  - Introduced "Masked" config, which will be masked in darkconsole once that gets updated.
  - "Hidden" implies "Masked" and "Locked".
  - Moved "storage.default-namespace" here -- it probably makes more sense than core; this was my bad in T2255.
  - Put cancel button back for hidden/locked config.
  - Introduce 'class' config type.

Test Plan: Viewed MySQL options. None are editable.

Reviewers: codeblock, btrahan

Reviewed By: codeblock

CC: aran

Maniphest Tasks: T2255

Differential Revision: https://secure.phabricator.com/D4326
2013-01-03 06:01:14 -08:00
epriestley
aae5f9efd3 Implement a more compact, general database-backed key-value cache
Summary:
See discussion in D4204. Facebook currently has a 314MB remarkup cache with a 55MB index, which is slow to access. Under the theory that this is an index size/quality problem (the current index is on a potentially-384-byte field, with many keys sharing prefixes), provide a more general index with fancy new features:

  - It implements PhutilKeyValueCache, so it can be a component in cache stacks and supports TTL.
  - It has a 12-byte hash-based key.
  - It automatically compresses large blocks of data (most of what we store is highly-compressible HTML).

Test Plan:
  - Basics:
    - Loaded /paste/, saw caches generate and save.
    - Reloaded /paste/, saw the page hit cache.
  - GC:
    - Ran GC daemon, saw nothing.
    - Set maximum lifetime to 1 second, ran GC daemon, saw it collect the entire cache.
  - Deflate:
    - Selected row formats from the database, saw a mixture of 'raw' and 'deflate' storage.
    - Used profiler to verify that 'deflate' is fast (12 calls @ 220us on my paste list).
  - Ran unit tests

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D4259
2012-12-21 14:17:56 -08:00
epriestley
7332599e03 Provide an IDS_COUNTER mechanism for ID assignment
Summary: See D3912 for discussion. InnoDB may reuse autoincrement IDs after restart; provide a way to avoid it.

Test Plan: Unit tests. Scheduled and executed tasks through `drydock lease --type host` and `phd debug taskmaster`.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D3914
2012-11-07 13:33:07 -08:00
vrana
ef85f49adc Delete license headers from files
Summary:
This commit doesn't change license of any file. It just makes the license implicit (inherited from LICENSE file in the root directory).

We are removing the headers for these reasons:

- It wastes space in editors, less code is visible in editor upon opening a file.
- It brings noise to diff of the first change of any file every year.
- It confuses Git file copy detection when creating small files.
- We don't have an explicit license header in other files (JS, CSS, images, documentation).
- Using license header in every file is not obligatory: http://www.apache.org/dev/apply-license.html#new.

This change is approved by Alma Chao (Lead Open Source and IP Counsel at Facebook).

Test Plan: Verified that the license survived only in LICENSE file and that it didn't modify externals.

Reviewers: epriestley, davidrecordon

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2035

Differential Revision: https://secure.phabricator.com/D3886
2012-11-05 11:16:51 -08:00
Jakub Vrana
d506ffbd30 Add regression test for LiskDAO::__call()
Summary: D3606

Test Plan: Deleted `call()`, ran the test.

Reviewers: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3814
2012-10-24 14:00:43 -07:00
vrana
fd87a88d71 Fix static variable usage in Lisk
Summary:
See comment.
This can reveal some pretty bad bugs but HPHP handles this correctly so we already know about them.

Test Plan:
Added `phlog()` to `__call()` and observed what is defined for each method (under PHP). Also:

  class C {
    function __call($name, $args) {
      static $class;
      if (!$class) {
        $class = get_class($this);
      }
      return $class;
    }
  }

  class D extends C {
  }

  class E extends C {
  }

  $d = new D;
  $e = new E;
  var_dump($d->x());
  var_dump($e->x()); // Prints D under PHP!

See also D3754.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T1261

Differential Revision: https://secure.phabricator.com/D3753
2012-10-20 05:04:38 -07:00
epriestley
cd9d78c107 Allow Fact analysis of commits
Summary: For immutable objects, just use the ID as a cursor.

Test Plan:
  - Analyzed commits from an empty cursor.
  - Checked that cursor was good.
  - Pulled some more commits.
  - Analyzed commits again, verified it only hit the new ones.
  - Verified the graph of "Count of CMIT" looked reasonable.

Reviewers: vrana

Reviewed By: vrana

CC: aran

Maniphest Tasks: T1866

Differential Revision: https://secure.phabricator.com/D3656
2012-10-08 13:27:06 -07:00
epriestley
ea32c541e4 Remove all optimistic lock code from Lisk
Summary: We never use this and almost certainly never will. It's been in Lisk for ~7 years but is a solution in search of a problem. It causes a conflict with any DAO that has a `version` column.

Test Plan: Browsed around, performed inserts and updates. Edited a Phriction document.

Reviewers: vrana, btrahan

Reviewed By: btrahan

CC: aran, leslie.chong

Differential Revision: https://secure.phabricator.com/D3625
2012-10-04 13:55:43 -07:00
epriestley
fe6022cd97 Load only valid properties in LiskDAO::loadFromArray()
Summary:
Fixes a TODO, and silences a warning introduced by D3601.

There are several cases where we load data like:

  SELECT *, ... AS extraData FROM ...

...and then pass it to `loadAllFromArray()`. Currently, this causes us to set an `extraData` property on the object.

This idiom seems fairly useful and non-dangerous, so I made `loadFromArray()` just drop extra keys.

Since we hit this loop a potentially huge number of times (10,000+ for full Maniphest pages) I did some microoptimization. Lisk is hot enough that it's one of the few places where it's worthwhile (see D1291).

Test Plan: Loaded homepage, no longer got warnings about `viewerIsMember` from Project queries. Browsed ~10 apps, didn't see any issues.

Reviewers: vrana

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D3606
2012-10-03 15:42:44 -07:00
vrana
4682e0c104 Warn against writing to undeclared properties
Summary:
I make this error quite often: I forget to declare a property I am writing to or I make a typo in it.
PHP implicitly creates a public property which I don't like.

I would much rather see a linter warning me against this than this runtime check but writing it is very difficult:

- We need to explore all parents of the class we are checking.
- It is even possible that children will declare that property but it's OK to treat this as error anyway.
- We can extend also builtin or external classes.
- It's somewhat doable for `$this` but even more complex for any `$obj` because we don't know the class of it.

This should catch significant part of these errors and I'm fine with that.

I don't plan escalating to exception because this error is not fatal and should not stop the application from working.

Test Plan: Loaded homepage, checked log.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3601
2012-10-03 11:54:03 -07:00
vrana
7c39c4ca7d Declare common Lisk properties
Summary:
Calling `->setPHID()` or other common Lisk setters creates an implicit public property `$phid`.
I don't like implicit properties and I see them as errors.
Its public visibility also makes me nervous and is vulnerable to bypassing any setters we may create.

Test Plan: Loaded homepage, checked log.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3600
2012-10-03 11:53:37 -07:00
vrana
8e7ae7b33a Optimize reindex_everything.php
Summary:
We have two troubles with this script:

# Our revisions and commits don't fit in the memory. (Our tasks do :-).)
# Reindexing revisions is slow.

Test Plan: Ran it.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3483
2012-09-17 13:34:41 -07:00
vrana
45e93495e4 Add method for loading relative edges
Summary:
More and more relations are going under edges and I can't work with them from Relatives framework.

This doesn't have the nice transitive property of normal relatives (loading relative objects from relatives loads all of them at once) but I can add it when I need it.

I plan to use it in D3085 (after converting relationships to edges).

Test Plan:
  $task = id(new ManiphestTask())
    ->loadOneWhere('phid = %s', $phid);
  print_r($task->loadRelativeEdges(4));

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3344
2012-08-20 21:11:55 -07:00
vrana
1379876db7 Remove subsets when cleaning Lisk set 2012-08-13 11:39:59 -07:00
vrana
f841491524 Use Lisk sets in fact update iterator
Summary:
Fact engines loading dependent objects are super slow because they load them one by one.
This diff put each page in a Lisk set allowing engines to use `loadRelatives()`.

It also introduces `clearSet()` method which is somewhat neccessary in PHP < 5.3 or with disabled cyclic [[ http://php.net/gc | GC ]].

Test Plan:
  $iterator = new PhabricatorFactUpdateIterator(new DifferentialRevision());
  foreach ($iterator as $revision) {
    $diffs = $revision->loadRelatives(new DifferentialDiff(), 'revisionID');
    echo memory_get_usage() . "\n";
  }

Experienced not-steadily-increasing memory usage and much faster loading.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3247
2012-08-13 10:26:17 -07:00
vrana
b2c9edd17d Fix doc links 2012-08-10 14:21:55 -07:00
epriestley
ff61dba7ac Extend LiskMigrationIterator from PhutilBufferedIterator
Summary: PhutilBufferedIterator now implements all the nonspecific logic here.

Test Plan:
Created a test script like this:

  $iterator = new LiskMigrationIterator(new DifferentialRevision());
  $iterator->setPageSize(3);

  foreach ($iterator as $key => $rev) {
    echo "{$key}: ".$rev->getID()."\n";
  }

Ran it and verified sensible iteration results.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Maniphest Tasks: T1562

Differential Revision: https://secure.phabricator.com/D3076
2012-07-26 12:01:57 -07:00
epriestley
5d4a6bcf95 Break AphrontDatabaseConnection dependencies on PhabricatorEnv
Summary: We pull "retries" and a doc link from PhabricatorEnv directly. Break these dependencies so the classes can move to libphutil.

Test Plan: Browsed site, triggered a schema exception and verified I still got the useful footer text.

Reviewers: btrahan, vrana

Reviewed By: vrana

CC: aran

Maniphest Tasks: T1283

Differential Revision: https://secure.phabricator.com/D3053
2012-07-24 10:47:27 -07:00
epriestley
a7bcc532da Add an iterator to make it easier to perform database migrations
Summary: See D3006, D3007. Make it easier to do migrations like that without holding all results in memory.

Test Plan:
Ran this code with an artificially small page size (2):

  foreach (new LiskMigrationIterator(new DifferentialRevision()) as $rev) {
    echo "Revision ".$rev->getID()."\n";
  }

Verified each revision as loaded and processed.

Reviewers: vrana, btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1162

Differential Revision: https://secure.phabricator.com/D3008
2012-07-18 20:01:23 -07:00
epriestley
cb8120551f Don't special-case LiskDAO->load(0)
Summary:
Lisk currently behaves in two different ways if you call it like `load("cow")` (throws) versus `load(99999999)` (returns null), where neither ID exists.

This was intended to catch programming errors as distinct from missing data, but in practice the former is very rare and you have to handle the latter in most cases anyway. The case where you pass "0" is particularly confusing. See D2971 for an example.

On the balance, I think this ends up being far more confusing than helpful. Instead, just return NULL if we're sure there's no such object.

Test Plan: Reasoned about program behavior.

Reviewers: alanh, btrahan, vrana

Reviewed By: vrana

CC: aran

Differential Revision: https://secure.phabricator.com/D2977
2012-07-16 09:50:23 -07:00
epriestley
d86c4e0366 Store forced connections in the Lisk connection cache
Summary:
In unit tests which use fixtures, we open transactions on every connection we establish. However, since we don't track connections that are established with "$force_new" (currently, only GlobalLock connections) we never close these transactions normally.

Instead of not tracking these connections, track them using unique keys so we'll never get a cache hit on them.

Test Plan: Built unit tests on top of this, had them stop dying from unclosed transactions.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1162

Differential Revision: https://secure.phabricator.com/D2938
2012-07-09 10:39:21 -07:00
epriestley
534c0aa326 Minor, move all storage/query/db code to src/infrastructure/storage 2012-07-02 10:49:00 -07:00