Add basic per-object privacy policies
Summary:
Provides a basic start for access policies. Objects expose various capabilities, like CAN_VIEW, CAN_EDIT, etc., and set a policy for each capability. We currently implement three policies, PUBLIC (anyone, including logged-out), USERS (any logged-in) and NOONE (nobody). There's also a way to provide automatic capability grants (e.g., the owner of an object can always see it, even if some capability is set to "NOONE"), but I'm not sure how great the implementation feels and it might change.
Most of the code here is providing a primitive for efficient policy-aware list queries. The problem with doing queries naively is that you have to do crazy amounts of filtering, e.g. to show the user page 6, you need to filter at least 600 objects (and likely more) before you can figure out which ones are 500-600 for them. You can't just do "LIMIT 500, 100" because that might have only 50 results, or no results. Instead, the query looks like "WHERE id > last_visible_id", and then we fetch additional pages as necessary to satisfy the request.
The general idea is that we move all data access to Query classes and have them do object filtering. The ID paging primitive allows efficient paging in most cases, and the executeOne() method provides a concise way to do policy checks for edit/view screens.
We'll probably end up with mostly broader policy UIs or configuration-based policies, but there are at least a few cases for per-object privacy (e.g., marking tasks as "Security", and restricting things to the members of projects) so I figured we'd start with a flexible primitive and the simplify it in the UI where we can.
Test Plan: Unit tests, played around in the UI with various policy settings.
Reviewers: btrahan, vrana, jungejason
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D2210
2012-04-14 19:13:29 +02:00
|
|
|
<?php
|
|
|
|
|
2012-09-13 19:15:08 +02:00
|
|
|
final class PhabricatorPasteQuery
|
|
|
|
extends PhabricatorCursorPagedPolicyAwareQuery {
|
Add basic per-object privacy policies
Summary:
Provides a basic start for access policies. Objects expose various capabilities, like CAN_VIEW, CAN_EDIT, etc., and set a policy for each capability. We currently implement three policies, PUBLIC (anyone, including logged-out), USERS (any logged-in) and NOONE (nobody). There's also a way to provide automatic capability grants (e.g., the owner of an object can always see it, even if some capability is set to "NOONE"), but I'm not sure how great the implementation feels and it might change.
Most of the code here is providing a primitive for efficient policy-aware list queries. The problem with doing queries naively is that you have to do crazy amounts of filtering, e.g. to show the user page 6, you need to filter at least 600 objects (and likely more) before you can figure out which ones are 500-600 for them. You can't just do "LIMIT 500, 100" because that might have only 50 results, or no results. Instead, the query looks like "WHERE id > last_visible_id", and then we fetch additional pages as necessary to satisfy the request.
The general idea is that we move all data access to Query classes and have them do object filtering. The ID paging primitive allows efficient paging in most cases, and the executeOne() method provides a concise way to do policy checks for edit/view screens.
We'll probably end up with mostly broader policy UIs or configuration-based policies, but there are at least a few cases for per-object privacy (e.g., marking tasks as "Security", and restricting things to the members of projects) so I figured we'd start with a flexible primitive and the simplify it in the UI where we can.
Test Plan: Unit tests, played around in the UI with various policy settings.
Reviewers: btrahan, vrana, jungejason
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D2210
2012-04-14 19:13:29 +02:00
|
|
|
|
2012-08-15 19:45:06 +02:00
|
|
|
private $ids;
|
|
|
|
private $phids;
|
Add basic per-object privacy policies
Summary:
Provides a basic start for access policies. Objects expose various capabilities, like CAN_VIEW, CAN_EDIT, etc., and set a policy for each capability. We currently implement three policies, PUBLIC (anyone, including logged-out), USERS (any logged-in) and NOONE (nobody). There's also a way to provide automatic capability grants (e.g., the owner of an object can always see it, even if some capability is set to "NOONE"), but I'm not sure how great the implementation feels and it might change.
Most of the code here is providing a primitive for efficient policy-aware list queries. The problem with doing queries naively is that you have to do crazy amounts of filtering, e.g. to show the user page 6, you need to filter at least 600 objects (and likely more) before you can figure out which ones are 500-600 for them. You can't just do "LIMIT 500, 100" because that might have only 50 results, or no results. Instead, the query looks like "WHERE id > last_visible_id", and then we fetch additional pages as necessary to satisfy the request.
The general idea is that we move all data access to Query classes and have them do object filtering. The ID paging primitive allows efficient paging in most cases, and the executeOne() method provides a concise way to do policy checks for edit/view screens.
We'll probably end up with mostly broader policy UIs or configuration-based policies, but there are at least a few cases for per-object privacy (e.g., marking tasks as "Security", and restricting things to the members of projects) so I figured we'd start with a flexible primitive and the simplify it in the UI where we can.
Test Plan: Unit tests, played around in the UI with various policy settings.
Reviewers: btrahan, vrana, jungejason
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D2210
2012-04-14 19:13:29 +02:00
|
|
|
private $authorPHIDs;
|
2012-08-15 22:45:53 +02:00
|
|
|
private $parentPHIDs;
|
Add basic per-object privacy policies
Summary:
Provides a basic start for access policies. Objects expose various capabilities, like CAN_VIEW, CAN_EDIT, etc., and set a policy for each capability. We currently implement three policies, PUBLIC (anyone, including logged-out), USERS (any logged-in) and NOONE (nobody). There's also a way to provide automatic capability grants (e.g., the owner of an object can always see it, even if some capability is set to "NOONE"), but I'm not sure how great the implementation feels and it might change.
Most of the code here is providing a primitive for efficient policy-aware list queries. The problem with doing queries naively is that you have to do crazy amounts of filtering, e.g. to show the user page 6, you need to filter at least 600 objects (and likely more) before you can figure out which ones are 500-600 for them. You can't just do "LIMIT 500, 100" because that might have only 50 results, or no results. Instead, the query looks like "WHERE id > last_visible_id", and then we fetch additional pages as necessary to satisfy the request.
The general idea is that we move all data access to Query classes and have them do object filtering. The ID paging primitive allows efficient paging in most cases, and the executeOne() method provides a concise way to do policy checks for edit/view screens.
We'll probably end up with mostly broader policy UIs or configuration-based policies, but there are at least a few cases for per-object privacy (e.g., marking tasks as "Security", and restricting things to the members of projects) so I figured we'd start with a flexible primitive and the simplify it in the UI where we can.
Test Plan: Unit tests, played around in the UI with various policy settings.
Reviewers: btrahan, vrana, jungejason
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D2210
2012-04-14 19:13:29 +02:00
|
|
|
|
2012-08-24 22:20:20 +02:00
|
|
|
private $needContent;
|
2012-12-17 01:33:42 +01:00
|
|
|
private $needRawContent;
|
2012-08-24 22:20:20 +02:00
|
|
|
|
2012-08-15 19:45:06 +02:00
|
|
|
public function withIDs(array $ids) {
|
|
|
|
$this->ids = $ids;
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function withPHIDs(array $phids) {
|
|
|
|
$this->phids = $phids;
|
Add basic per-object privacy policies
Summary:
Provides a basic start for access policies. Objects expose various capabilities, like CAN_VIEW, CAN_EDIT, etc., and set a policy for each capability. We currently implement three policies, PUBLIC (anyone, including logged-out), USERS (any logged-in) and NOONE (nobody). There's also a way to provide automatic capability grants (e.g., the owner of an object can always see it, even if some capability is set to "NOONE"), but I'm not sure how great the implementation feels and it might change.
Most of the code here is providing a primitive for efficient policy-aware list queries. The problem with doing queries naively is that you have to do crazy amounts of filtering, e.g. to show the user page 6, you need to filter at least 600 objects (and likely more) before you can figure out which ones are 500-600 for them. You can't just do "LIMIT 500, 100" because that might have only 50 results, or no results. Instead, the query looks like "WHERE id > last_visible_id", and then we fetch additional pages as necessary to satisfy the request.
The general idea is that we move all data access to Query classes and have them do object filtering. The ID paging primitive allows efficient paging in most cases, and the executeOne() method provides a concise way to do policy checks for edit/view screens.
We'll probably end up with mostly broader policy UIs or configuration-based policies, but there are at least a few cases for per-object privacy (e.g., marking tasks as "Security", and restricting things to the members of projects) so I figured we'd start with a flexible primitive and the simplify it in the UI where we can.
Test Plan: Unit tests, played around in the UI with various policy settings.
Reviewers: btrahan, vrana, jungejason
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D2210
2012-04-14 19:13:29 +02:00
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
|
|
|
public function withAuthorPHIDs(array $phids) {
|
|
|
|
$this->authorPHIDs = $phids;
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
2012-08-15 22:45:53 +02:00
|
|
|
public function withParentPHIDs(array $phids) {
|
|
|
|
$this->parentPHIDs = $phids;
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
2012-08-24 22:20:20 +02:00
|
|
|
public function needContent($need_content) {
|
|
|
|
$this->needContent = $need_content;
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
2012-12-17 01:33:42 +01:00
|
|
|
public function needRawContent($need_raw_content) {
|
|
|
|
$this->needRawContent = $need_raw_content;
|
|
|
|
return $this;
|
|
|
|
}
|
|
|
|
|
2013-03-01 20:28:02 +01:00
|
|
|
protected function loadPage() {
|
Add basic per-object privacy policies
Summary:
Provides a basic start for access policies. Objects expose various capabilities, like CAN_VIEW, CAN_EDIT, etc., and set a policy for each capability. We currently implement three policies, PUBLIC (anyone, including logged-out), USERS (any logged-in) and NOONE (nobody). There's also a way to provide automatic capability grants (e.g., the owner of an object can always see it, even if some capability is set to "NOONE"), but I'm not sure how great the implementation feels and it might change.
Most of the code here is providing a primitive for efficient policy-aware list queries. The problem with doing queries naively is that you have to do crazy amounts of filtering, e.g. to show the user page 6, you need to filter at least 600 objects (and likely more) before you can figure out which ones are 500-600 for them. You can't just do "LIMIT 500, 100" because that might have only 50 results, or no results. Instead, the query looks like "WHERE id > last_visible_id", and then we fetch additional pages as necessary to satisfy the request.
The general idea is that we move all data access to Query classes and have them do object filtering. The ID paging primitive allows efficient paging in most cases, and the executeOne() method provides a concise way to do policy checks for edit/view screens.
We'll probably end up with mostly broader policy UIs or configuration-based policies, but there are at least a few cases for per-object privacy (e.g., marking tasks as "Security", and restricting things to the members of projects) so I figured we'd start with a flexible primitive and the simplify it in the UI where we can.
Test Plan: Unit tests, played around in the UI with various policy settings.
Reviewers: btrahan, vrana, jungejason
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D2210
2012-04-14 19:13:29 +02:00
|
|
|
$table = new PhabricatorPaste();
|
|
|
|
$conn_r = $table->establishConnection('r');
|
|
|
|
|
|
|
|
$data = queryfx_all(
|
|
|
|
$conn_r,
|
|
|
|
'SELECT paste.* FROM %T paste %Q %Q %Q',
|
|
|
|
$table->getTableName(),
|
|
|
|
$this->buildWhereClause($conn_r),
|
|
|
|
$this->buildOrderClause($conn_r),
|
|
|
|
$this->buildLimitClause($conn_r));
|
|
|
|
|
2012-08-24 22:20:20 +02:00
|
|
|
$pastes = $table->loadAllFromArray($data);
|
|
|
|
|
2013-05-29 15:28:47 +02:00
|
|
|
return $pastes;
|
|
|
|
}
|
|
|
|
|
|
|
|
protected function willFilterPage(array $pastes) {
|
|
|
|
if (!$pastes) {
|
|
|
|
return $pastes;
|
2012-12-17 01:33:42 +01:00
|
|
|
}
|
|
|
|
|
2013-05-29 15:28:47 +02:00
|
|
|
if ($this->needRawContent) {
|
|
|
|
$pastes = $this->loadRawContent($pastes);
|
|
|
|
}
|
|
|
|
|
|
|
|
if ($this->needContent) {
|
|
|
|
$pastes = $this->loadContent($pastes);
|
2012-08-24 22:20:20 +02:00
|
|
|
}
|
Add basic per-object privacy policies
Summary:
Provides a basic start for access policies. Objects expose various capabilities, like CAN_VIEW, CAN_EDIT, etc., and set a policy for each capability. We currently implement three policies, PUBLIC (anyone, including logged-out), USERS (any logged-in) and NOONE (nobody). There's also a way to provide automatic capability grants (e.g., the owner of an object can always see it, even if some capability is set to "NOONE"), but I'm not sure how great the implementation feels and it might change.
Most of the code here is providing a primitive for efficient policy-aware list queries. The problem with doing queries naively is that you have to do crazy amounts of filtering, e.g. to show the user page 6, you need to filter at least 600 objects (and likely more) before you can figure out which ones are 500-600 for them. You can't just do "LIMIT 500, 100" because that might have only 50 results, or no results. Instead, the query looks like "WHERE id > last_visible_id", and then we fetch additional pages as necessary to satisfy the request.
The general idea is that we move all data access to Query classes and have them do object filtering. The ID paging primitive allows efficient paging in most cases, and the executeOne() method provides a concise way to do policy checks for edit/view screens.
We'll probably end up with mostly broader policy UIs or configuration-based policies, but there are at least a few cases for per-object privacy (e.g., marking tasks as "Security", and restricting things to the members of projects) so I figured we'd start with a flexible primitive and the simplify it in the UI where we can.
Test Plan: Unit tests, played around in the UI with various policy settings.
Reviewers: btrahan, vrana, jungejason
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D2210
2012-04-14 19:13:29 +02:00
|
|
|
|
Allow policy-aware queries to prefilter results
Summary:
Provides a simple way for policy-aware queries to pre-filter results without needing to maintain separate cursors, and fixes a bunch of filter-related edge cases.
- For reverse-paged cursor queries, we previously reversed each individual set of results. If the final result set is built out of multiple pages, it's in the wrong order overall, with each page in the correct order in sequence. Instead, reverse everything at the end. This also simplifies construction of queries.
- `AphrontCursorPagerView` would always render a "<< First" link when paging backward, even if we were on the first page of results.
- Add a filtering hook to let queries perform in-application pre-policy filtering as simply as possible (i.e., without maintaing their own cursors over the result sets).
Test Plan: Made feed randomly prefilter half the results, and paged forward and backward. Observed correct result ordering, pagination, and next/previous links.
Reviewers: btrahan, vrana
Reviewed By: btrahan
CC: aran
Differential Revision: https://secure.phabricator.com/D3787
2012-10-23 21:01:11 +02:00
|
|
|
return $pastes;
|
Add basic per-object privacy policies
Summary:
Provides a basic start for access policies. Objects expose various capabilities, like CAN_VIEW, CAN_EDIT, etc., and set a policy for each capability. We currently implement three policies, PUBLIC (anyone, including logged-out), USERS (any logged-in) and NOONE (nobody). There's also a way to provide automatic capability grants (e.g., the owner of an object can always see it, even if some capability is set to "NOONE"), but I'm not sure how great the implementation feels and it might change.
Most of the code here is providing a primitive for efficient policy-aware list queries. The problem with doing queries naively is that you have to do crazy amounts of filtering, e.g. to show the user page 6, you need to filter at least 600 objects (and likely more) before you can figure out which ones are 500-600 for them. You can't just do "LIMIT 500, 100" because that might have only 50 results, or no results. Instead, the query looks like "WHERE id > last_visible_id", and then we fetch additional pages as necessary to satisfy the request.
The general idea is that we move all data access to Query classes and have them do object filtering. The ID paging primitive allows efficient paging in most cases, and the executeOne() method provides a concise way to do policy checks for edit/view screens.
We'll probably end up with mostly broader policy UIs or configuration-based policies, but there are at least a few cases for per-object privacy (e.g., marking tasks as "Security", and restricting things to the members of projects) so I figured we'd start with a flexible primitive and the simplify it in the UI where we can.
Test Plan: Unit tests, played around in the UI with various policy settings.
Reviewers: btrahan, vrana, jungejason
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D2210
2012-04-14 19:13:29 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
protected function buildWhereClause($conn_r) {
|
|
|
|
$where = array();
|
|
|
|
|
|
|
|
$where[] = $this->buildPagingClause($conn_r);
|
|
|
|
|
2012-08-15 19:45:06 +02:00
|
|
|
if ($this->ids) {
|
|
|
|
$where[] = qsprintf(
|
|
|
|
$conn_r,
|
|
|
|
'id IN (%Ld)',
|
|
|
|
$this->ids);
|
|
|
|
}
|
|
|
|
|
|
|
|
if ($this->phids) {
|
Add basic per-object privacy policies
Summary:
Provides a basic start for access policies. Objects expose various capabilities, like CAN_VIEW, CAN_EDIT, etc., and set a policy for each capability. We currently implement three policies, PUBLIC (anyone, including logged-out), USERS (any logged-in) and NOONE (nobody). There's also a way to provide automatic capability grants (e.g., the owner of an object can always see it, even if some capability is set to "NOONE"), but I'm not sure how great the implementation feels and it might change.
Most of the code here is providing a primitive for efficient policy-aware list queries. The problem with doing queries naively is that you have to do crazy amounts of filtering, e.g. to show the user page 6, you need to filter at least 600 objects (and likely more) before you can figure out which ones are 500-600 for them. You can't just do "LIMIT 500, 100" because that might have only 50 results, or no results. Instead, the query looks like "WHERE id > last_visible_id", and then we fetch additional pages as necessary to satisfy the request.
The general idea is that we move all data access to Query classes and have them do object filtering. The ID paging primitive allows efficient paging in most cases, and the executeOne() method provides a concise way to do policy checks for edit/view screens.
We'll probably end up with mostly broader policy UIs or configuration-based policies, but there are at least a few cases for per-object privacy (e.g., marking tasks as "Security", and restricting things to the members of projects) so I figured we'd start with a flexible primitive and the simplify it in the UI where we can.
Test Plan: Unit tests, played around in the UI with various policy settings.
Reviewers: btrahan, vrana, jungejason
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D2210
2012-04-14 19:13:29 +02:00
|
|
|
$where[] = qsprintf(
|
|
|
|
$conn_r,
|
2012-08-15 19:45:06 +02:00
|
|
|
'phid IN (%Ls)',
|
|
|
|
$this->phids);
|
Add basic per-object privacy policies
Summary:
Provides a basic start for access policies. Objects expose various capabilities, like CAN_VIEW, CAN_EDIT, etc., and set a policy for each capability. We currently implement three policies, PUBLIC (anyone, including logged-out), USERS (any logged-in) and NOONE (nobody). There's also a way to provide automatic capability grants (e.g., the owner of an object can always see it, even if some capability is set to "NOONE"), but I'm not sure how great the implementation feels and it might change.
Most of the code here is providing a primitive for efficient policy-aware list queries. The problem with doing queries naively is that you have to do crazy amounts of filtering, e.g. to show the user page 6, you need to filter at least 600 objects (and likely more) before you can figure out which ones are 500-600 for them. You can't just do "LIMIT 500, 100" because that might have only 50 results, or no results. Instead, the query looks like "WHERE id > last_visible_id", and then we fetch additional pages as necessary to satisfy the request.
The general idea is that we move all data access to Query classes and have them do object filtering. The ID paging primitive allows efficient paging in most cases, and the executeOne() method provides a concise way to do policy checks for edit/view screens.
We'll probably end up with mostly broader policy UIs or configuration-based policies, but there are at least a few cases for per-object privacy (e.g., marking tasks as "Security", and restricting things to the members of projects) so I figured we'd start with a flexible primitive and the simplify it in the UI where we can.
Test Plan: Unit tests, played around in the UI with various policy settings.
Reviewers: btrahan, vrana, jungejason
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D2210
2012-04-14 19:13:29 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
if ($this->authorPHIDs) {
|
|
|
|
$where[] = qsprintf(
|
|
|
|
$conn_r,
|
|
|
|
'authorPHID IN (%Ls)',
|
|
|
|
$this->authorPHIDs);
|
|
|
|
}
|
|
|
|
|
2012-08-15 22:45:53 +02:00
|
|
|
if ($this->parentPHIDs) {
|
|
|
|
$where[] = qsprintf(
|
|
|
|
$conn_r,
|
|
|
|
'parentPHID IN (%Ls)',
|
|
|
|
$this->parentPHIDs);
|
|
|
|
}
|
|
|
|
|
Add basic per-object privacy policies
Summary:
Provides a basic start for access policies. Objects expose various capabilities, like CAN_VIEW, CAN_EDIT, etc., and set a policy for each capability. We currently implement three policies, PUBLIC (anyone, including logged-out), USERS (any logged-in) and NOONE (nobody). There's also a way to provide automatic capability grants (e.g., the owner of an object can always see it, even if some capability is set to "NOONE"), but I'm not sure how great the implementation feels and it might change.
Most of the code here is providing a primitive for efficient policy-aware list queries. The problem with doing queries naively is that you have to do crazy amounts of filtering, e.g. to show the user page 6, you need to filter at least 600 objects (and likely more) before you can figure out which ones are 500-600 for them. You can't just do "LIMIT 500, 100" because that might have only 50 results, or no results. Instead, the query looks like "WHERE id > last_visible_id", and then we fetch additional pages as necessary to satisfy the request.
The general idea is that we move all data access to Query classes and have them do object filtering. The ID paging primitive allows efficient paging in most cases, and the executeOne() method provides a concise way to do policy checks for edit/view screens.
We'll probably end up with mostly broader policy UIs or configuration-based policies, but there are at least a few cases for per-object privacy (e.g., marking tasks as "Security", and restricting things to the members of projects) so I figured we'd start with a flexible primitive and the simplify it in the UI where we can.
Test Plan: Unit tests, played around in the UI with various policy settings.
Reviewers: btrahan, vrana, jungejason
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D2210
2012-04-14 19:13:29 +02:00
|
|
|
return $this->formatWhereClause($where);
|
|
|
|
}
|
|
|
|
|
2012-12-17 01:33:42 +01:00
|
|
|
private function getContentCacheKey(PhabricatorPaste $paste) {
|
|
|
|
return 'P'.$paste->getID().':content/'.$paste->getLanguage();
|
|
|
|
}
|
|
|
|
|
|
|
|
private function loadRawContent(array $pastes) {
|
|
|
|
$file_phids = mpull($pastes, 'getFilePHID');
|
|
|
|
$files = id(new PhabricatorFile())->loadAllWhere(
|
|
|
|
'phid IN (%Ls)',
|
|
|
|
$file_phids);
|
|
|
|
$files = mpull($files, null, 'getPHID');
|
|
|
|
|
2013-05-29 15:28:47 +02:00
|
|
|
foreach ($pastes as $key => $paste) {
|
2012-12-17 01:33:42 +01:00
|
|
|
$file = idx($files, $paste->getFilePHID());
|
2013-05-29 15:28:47 +02:00
|
|
|
if (!$file) {
|
|
|
|
unset($pastes[$key]);
|
|
|
|
continue;
|
2012-12-17 01:33:42 +01:00
|
|
|
}
|
2013-05-29 15:28:47 +02:00
|
|
|
$paste->attachRawContent($file->loadFileData());
|
2012-12-17 01:33:42 +01:00
|
|
|
}
|
2013-05-29 15:28:47 +02:00
|
|
|
|
|
|
|
return $pastes;
|
2012-12-17 01:33:42 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
private function loadContent(array $pastes) {
|
2012-12-31 02:04:38 +01:00
|
|
|
$cache = new PhabricatorKeyValueDatabaseCache();
|
|
|
|
|
|
|
|
$cache = new PhutilKeyValueCacheProfiler($cache);
|
|
|
|
$cache->setProfiler(PhutilServiceProfiler::getInstance());
|
Implement a more compact, general database-backed key-value cache
Summary:
See discussion in D4204. Facebook currently has a 314MB remarkup cache with a 55MB index, which is slow to access. Under the theory that this is an index size/quality problem (the current index is on a potentially-384-byte field, with many keys sharing prefixes), provide a more general index with fancy new features:
- It implements PhutilKeyValueCache, so it can be a component in cache stacks and supports TTL.
- It has a 12-byte hash-based key.
- It automatically compresses large blocks of data (most of what we store is highly-compressible HTML).
Test Plan:
- Basics:
- Loaded /paste/, saw caches generate and save.
- Reloaded /paste/, saw the page hit cache.
- GC:
- Ran GC daemon, saw nothing.
- Set maximum lifetime to 1 second, ran GC daemon, saw it collect the entire cache.
- Deflate:
- Selected row formats from the database, saw a mixture of 'raw' and 'deflate' storage.
- Used profiler to verify that 'deflate' is fast (12 calls @ 220us on my paste list).
- Ran unit tests
Reviewers: vrana, btrahan
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D4259
2012-12-21 23:17:56 +01:00
|
|
|
|
2012-12-17 01:33:42 +01:00
|
|
|
$keys = array();
|
|
|
|
foreach ($pastes as $paste) {
|
|
|
|
$keys[] = $this->getContentCacheKey($paste);
|
|
|
|
}
|
|
|
|
|
Implement a more compact, general database-backed key-value cache
Summary:
See discussion in D4204. Facebook currently has a 314MB remarkup cache with a 55MB index, which is slow to access. Under the theory that this is an index size/quality problem (the current index is on a potentially-384-byte field, with many keys sharing prefixes), provide a more general index with fancy new features:
- It implements PhutilKeyValueCache, so it can be a component in cache stacks and supports TTL.
- It has a 12-byte hash-based key.
- It automatically compresses large blocks of data (most of what we store is highly-compressible HTML).
Test Plan:
- Basics:
- Loaded /paste/, saw caches generate and save.
- Reloaded /paste/, saw the page hit cache.
- GC:
- Ran GC daemon, saw nothing.
- Set maximum lifetime to 1 second, ran GC daemon, saw it collect the entire cache.
- Deflate:
- Selected row formats from the database, saw a mixture of 'raw' and 'deflate' storage.
- Used profiler to verify that 'deflate' is fast (12 calls @ 220us on my paste list).
- Ran unit tests
Reviewers: vrana, btrahan
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D4259
2012-12-21 23:17:56 +01:00
|
|
|
$caches = $cache->getKeys($keys);
|
2013-05-29 15:28:47 +02:00
|
|
|
$results = array();
|
2012-12-17 01:33:42 +01:00
|
|
|
|
|
|
|
$need_raw = array();
|
2013-05-29 15:28:47 +02:00
|
|
|
foreach ($pastes as $key => $paste) {
|
2012-12-17 01:33:42 +01:00
|
|
|
$key = $this->getContentCacheKey($paste);
|
|
|
|
if (isset($caches[$key])) {
|
2013-02-16 01:38:46 +01:00
|
|
|
$paste->attachContent(phutil_safe_html($caches[$key]));
|
2013-05-29 15:28:47 +02:00
|
|
|
$results[$key] = $paste;
|
2012-12-17 01:33:42 +01:00
|
|
|
} else {
|
2013-05-29 15:28:47 +02:00
|
|
|
$need_raw[$key] = $paste;
|
2012-12-17 01:33:42 +01:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!$need_raw) {
|
2013-05-29 15:28:47 +02:00
|
|
|
return $results;
|
2012-12-17 01:33:42 +01:00
|
|
|
}
|
|
|
|
|
Implement a more compact, general database-backed key-value cache
Summary:
See discussion in D4204. Facebook currently has a 314MB remarkup cache with a 55MB index, which is slow to access. Under the theory that this is an index size/quality problem (the current index is on a potentially-384-byte field, with many keys sharing prefixes), provide a more general index with fancy new features:
- It implements PhutilKeyValueCache, so it can be a component in cache stacks and supports TTL.
- It has a 12-byte hash-based key.
- It automatically compresses large blocks of data (most of what we store is highly-compressible HTML).
Test Plan:
- Basics:
- Loaded /paste/, saw caches generate and save.
- Reloaded /paste/, saw the page hit cache.
- GC:
- Ran GC daemon, saw nothing.
- Set maximum lifetime to 1 second, ran GC daemon, saw it collect the entire cache.
- Deflate:
- Selected row formats from the database, saw a mixture of 'raw' and 'deflate' storage.
- Used profiler to verify that 'deflate' is fast (12 calls @ 220us on my paste list).
- Ran unit tests
Reviewers: vrana, btrahan
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D4259
2012-12-21 23:17:56 +01:00
|
|
|
$write_data = array();
|
|
|
|
|
2013-05-29 15:28:47 +02:00
|
|
|
$need_raw = $this->loadRawContent($need_raw);
|
|
|
|
foreach ($need_raw as $key => $paste) {
|
2012-12-17 01:33:42 +01:00
|
|
|
$content = $this->buildContent($paste);
|
|
|
|
$paste->attachContent($content);
|
|
|
|
|
Implement a more compact, general database-backed key-value cache
Summary:
See discussion in D4204. Facebook currently has a 314MB remarkup cache with a 55MB index, which is slow to access. Under the theory that this is an index size/quality problem (the current index is on a potentially-384-byte field, with many keys sharing prefixes), provide a more general index with fancy new features:
- It implements PhutilKeyValueCache, so it can be a component in cache stacks and supports TTL.
- It has a 12-byte hash-based key.
- It automatically compresses large blocks of data (most of what we store is highly-compressible HTML).
Test Plan:
- Basics:
- Loaded /paste/, saw caches generate and save.
- Reloaded /paste/, saw the page hit cache.
- GC:
- Ran GC daemon, saw nothing.
- Set maximum lifetime to 1 second, ran GC daemon, saw it collect the entire cache.
- Deflate:
- Selected row formats from the database, saw a mixture of 'raw' and 'deflate' storage.
- Used profiler to verify that 'deflate' is fast (12 calls @ 220us on my paste list).
- Ran unit tests
Reviewers: vrana, btrahan
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D4259
2012-12-21 23:17:56 +01:00
|
|
|
$write_data[$this->getContentCacheKey($paste)] = (string)$content;
|
2013-05-29 15:28:47 +02:00
|
|
|
$results[$key] = $paste;
|
2012-12-17 01:33:42 +01:00
|
|
|
}
|
Implement a more compact, general database-backed key-value cache
Summary:
See discussion in D4204. Facebook currently has a 314MB remarkup cache with a 55MB index, which is slow to access. Under the theory that this is an index size/quality problem (the current index is on a potentially-384-byte field, with many keys sharing prefixes), provide a more general index with fancy new features:
- It implements PhutilKeyValueCache, so it can be a component in cache stacks and supports TTL.
- It has a 12-byte hash-based key.
- It automatically compresses large blocks of data (most of what we store is highly-compressible HTML).
Test Plan:
- Basics:
- Loaded /paste/, saw caches generate and save.
- Reloaded /paste/, saw the page hit cache.
- GC:
- Ran GC daemon, saw nothing.
- Set maximum lifetime to 1 second, ran GC daemon, saw it collect the entire cache.
- Deflate:
- Selected row formats from the database, saw a mixture of 'raw' and 'deflate' storage.
- Used profiler to verify that 'deflate' is fast (12 calls @ 220us on my paste list).
- Ran unit tests
Reviewers: vrana, btrahan
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D4259
2012-12-21 23:17:56 +01:00
|
|
|
|
|
|
|
$cache->setKeys($write_data);
|
2013-05-29 15:28:47 +02:00
|
|
|
|
|
|
|
return $results;
|
2012-12-17 01:33:42 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
private function buildContent(PhabricatorPaste $paste) {
|
|
|
|
$language = $paste->getLanguage();
|
|
|
|
$source = $paste->getRawContent();
|
|
|
|
|
|
|
|
if (empty($language)) {
|
|
|
|
return PhabricatorSyntaxHighlighter::highlightWithFilename(
|
|
|
|
$paste->getTitle(),
|
|
|
|
$source);
|
|
|
|
} else {
|
|
|
|
return PhabricatorSyntaxHighlighter::highlightWithLanguage(
|
|
|
|
$language,
|
|
|
|
$source);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Add basic per-object privacy policies
Summary:
Provides a basic start for access policies. Objects expose various capabilities, like CAN_VIEW, CAN_EDIT, etc., and set a policy for each capability. We currently implement three policies, PUBLIC (anyone, including logged-out), USERS (any logged-in) and NOONE (nobody). There's also a way to provide automatic capability grants (e.g., the owner of an object can always see it, even if some capability is set to "NOONE"), but I'm not sure how great the implementation feels and it might change.
Most of the code here is providing a primitive for efficient policy-aware list queries. The problem with doing queries naively is that you have to do crazy amounts of filtering, e.g. to show the user page 6, you need to filter at least 600 objects (and likely more) before you can figure out which ones are 500-600 for them. You can't just do "LIMIT 500, 100" because that might have only 50 results, or no results. Instead, the query looks like "WHERE id > last_visible_id", and then we fetch additional pages as necessary to satisfy the request.
The general idea is that we move all data access to Query classes and have them do object filtering. The ID paging primitive allows efficient paging in most cases, and the executeOne() method provides a concise way to do policy checks for edit/view screens.
We'll probably end up with mostly broader policy UIs or configuration-based policies, but there are at least a few cases for per-object privacy (e.g., marking tasks as "Security", and restricting things to the members of projects) so I figured we'd start with a flexible primitive and the simplify it in the UI where we can.
Test Plan: Unit tests, played around in the UI with various policy settings.
Reviewers: btrahan, vrana, jungejason
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T603
Differential Revision: https://secure.phabricator.com/D2210
2012-04-14 19:13:29 +02:00
|
|
|
}
|