revi-archive/phorge-phorge
1
0
Fork 0
mirror of https://we.phorge.it/source/phorge.git synced 2024-11-18 21:02:41 +01:00
Code Issues Releases Wiki Activity
phorge-phorge/resources/sql/patches
History
epriestley 2b0b9a1573 Add a generic multistep Markup cache 
Summary:
The immediate issue this addresses is T1366, adding a rendering cache to Phriction. For wiki pages with code blocks especially, rerendering them each time is expensive.

The broader issue is that out markup caches aren't very good right now. They have three major problems:

**Problem 1: the data is stored in the wrong place.** We currently store remarkup caches on objects. This means we're always loading it and passing it around even when we don't need it, can't genericize cache management code (e.g., have one simple script to drop/GC caches), need to update authoritative rows to clear caches, and can't genericize rendering code since each object is different.

To solve this, I created a dedicated cache database that I plan to move all markup caches to use.

**Problem 2: time-variant rules break when cached.** Some rules like `**bold**` are time-invariant and always produce the same output, but some rules like `{Tnnn}` and `@username` are variant and may render differently (because a task was closed or a user is on vacation). Currently, we cache the raw output, so these time-variant rules get locked at whatever values they had when they were first rendered. This is the main reason Phriction doesn't have a cache right now -- I wanted `{Tnnn}` rules to reflect open/closed tasks.

To solve this, I split markup into a "preprocessing" phase (which does all the parsing and evaluates all time-invariant rules) and a "postprocessing" phase (which evaluates time-variant rules only). The preprocessing phase is most of the expense (and, notably, includes syntax highlighting) so this is nearly as good as caching the final output. I did most of the work here in D737 / D738, but we never moved to use it in Phabricator -- we currently just do the two operations serially in all cases.

This diff splits them apart and caches the output of preprocessing only, so we benefit from caching but also get accurate time-variant rendering.

**Problem 3: cache access isn't batched/pipelined optimally.** When we're rendering a list of markup blocks, we should be able to batch datafetching better than we do. D738 helped with this (fetching is batched within a single hunk of markup) and this improves batching on cache access. We could still do better here, but this is at least a step forward.

Also fixes a bug with generating a link in the Phriction history interface ($uri gets clobbered).

I'm using PHP serialization instead of JSON serialization because Remarkup does some stuff with non-ascii characters that might not survive JSON.

Test Plan:
  - Created a Phriction document and verified that previews don't go to cache (no rows appear in the cache table).
  - Verified that published documents come out of cache.
  - Verified that caches generate/regenerate correctly, time-variant rules render properly and old documents hit the right caches.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1366

Differential Revision: https://secure.phabricator.com/D2945
2012-07-09 15:20:56 -07:00
..
0000.legacy.sql Minor, completely remove references to PHID from schema patches so upgrade-from-scratch works. 2012-05-24 13:59:12 -07:00
000.project.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
001.maniphest_projects.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
002.oauth.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
003.more_oauth.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
004.daemonrepos.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
005.workers.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
006.repository.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
007.daemonlog.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
008.repoopt.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
009.repo_summary.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
010.herald.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
011.badcommit.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
012.dropphidtype.sql Minor, completely remove references to PHID from schema patches so upgrade-from-scratch works. 2012-05-24 13:59:12 -07:00
013.commitdetail.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
014.shortcuts.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
015.preferences.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
016.userrealnameindex.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
017.sessionkeys.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
018.owners.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
019.arcprojects.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
020.pathcapital.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
021.xhpastview.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
022.differentialcommit.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
023.dxkeys.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
024.mlistkeys.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
025.commentopt.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
026.diffpropkey.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
027.metamtakeys.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
028.systemagent.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
029.cursors.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
030.imagemacro.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
031.workerrace.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
032.viewtime.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
033.privtest.sql Improve schema upgrade workflow for unprivileged users 2011-04-30 00:50:48 -07:00
034.savedheader.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
035.proxyimage.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
036.mailkey.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
037.setuptest.sql Add a "setup" mode which guides new users through application configuration 2011-05-10 15:12:30 -07:00
038.admin.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
039.userlog.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
040.transform.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
041.heraldrepetition.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
042.commentmetadata.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
043.pastebin.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
044.countdown.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
045.timezone.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
046.conduittoken.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
047.projectstatus.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
048.relationshipkeys.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
049.projectowner.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
050.taskdenormal.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
051.projectfilter.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
052.pastelanguage.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
053.feed.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
054.subscribers.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
055.add_author_to_files.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
056.slowvote.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
057.parsecache.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
058.missingkeys.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
059.engines.php SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
060.phriction.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
061.phrictioncontent.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
062.phrictionmenu.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
063.pasteforks.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
064.subprojects.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
065.sshkeys.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
066.phrictioncontent.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
067.preferences.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
068.maniphestauxiliarystorage.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
069.heraldxscript.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
070.differentialaux.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
071.contentsource.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
072.blamerevert.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
073.reposymbols.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
074.affectedpath.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
075.revisionhash.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
076.indexedlanguages.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
077.originalemail.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
078.nametoken.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
079.nametokenindex.php Add a name token table so on-demand typeaheads can match last names 2011-10-23 14:25:26 -07:00
080.filekeys.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
081.filekeys.php Use a proper entropy source to generate file keys 2011-10-23 14:42:23 -07:00
082.xactionkey.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
083.dxviewtime.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
084.pasteauthorkey.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
085.packagecommitrelationship.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
086.formeraffil.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
087.phrictiondelete.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
088.audit.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
089.projectwiki.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
090.forceuniqueprojectnames.php Provide wiki pages for projects 2011-12-20 14:03:12 -08:00
091.uniqueslugkey.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
092.dropgithubnotification.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
093.gitremotes.php Make tracked git repositories use an implicit 'origin' remote 2011-12-29 08:35:32 -08:00
094.phrictioncolumn.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
095.directory.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
096.filename.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
097.heraldruletypes.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
098.heraldruletypemigration.php Created personal vs. global herald rule distingtion 2012-01-19 11:21:49 -08:00
099.drydock.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
100.projectxaction.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
101.heraldruleapplied.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
102.heraldcleanup.php Write fewer "applied" rows and clean up excess historical rows 2012-01-25 11:53:39 -08:00
103.heraldedithistory.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
104.searchkey.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
105.mimetype.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
106.chatlog.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
107.oauthserver.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
108.oauthscope.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
109.oauthclientphidkey.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
110.commitaudit.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
111.commitauditmigration.php Make PhabricatorRepositoryCommmit schema changes for audit 2012-02-28 21:06:34 -08:00
112.oauthaccesscoderedirecturi.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
113.lastreviewer.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
114.auditrequest.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
115.prepareutf8.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
116.utf8-backup-first-expect-wait.sql Minor, completely remove references to PHID from schema patches so upgrade-from-scratch works. 2012-05-24 13:59:12 -07:00
117.repositorydescription.php Drop unused column 2012-03-12 12:11:36 -07:00
118.auditinline.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
119.filehash.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
120.noop.sql Add "Flags" to allow users to collect the things they love 2012-03-27 16:22:40 -07:00
121.drydocklog.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
122.flag.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
123.heraldrulelog.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
124.subpriority.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
125.ipv6.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
126.edges.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
127.userkeybody.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
128.phabricatorcom.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
129.savedquery.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
130.denormalrevisionquery.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
131.migraterevisionquery.php Fix patch 131 for db with lots of revisions 2012-04-16 12:35:48 -07:00
132.phame.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
133.imagemacro.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
134.emptysearch.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
135.datecommitted.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
136.sex.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
137.auditmetadata.sql SQL Patch Management: SQL Changes 2012-04-30 07:53:53 -07:00
138.notification.sql SQL patch for notification 2012-06-08 12:42:59 -07:00
differentialbookmarks.sql [Phabricator] track Mercurial bookmarks for differential diffs 2012-06-30 15:41:58 -07:00
emailtable.sql Allow users to have multiple email addresses, and verify emails 2012-05-07 10:29:33 -07:00
emailtableport.php Minor, derrrp. 2012-05-07 10:34:07 -07:00
emailtableremove.sql Allow users to have multiple email addresses, and verify emails 2012-05-07 10:29:33 -07:00
harbormasterobject.sql Add a generic multistep Markup cache 2012-07-09 15:20:56 -07:00
holidays.sql Display holidays 2012-05-03 09:22:52 -07:00
ldapinfo.sql Made it possible to login using LDAP 2012-06-13 08:58:46 -07:00
markupcache.sql Add a generic multistep Markup cache 2012-07-09 15:20:56 -07:00
phiddrop.sql Remove PHID database, add Harbormaster database 2012-05-20 14:46:01 -07:00
testdatabase.sql Remove PHID database, add Harbormaster database 2012-05-20 14:46:01 -07:00
threadtopic.sql Make Thread-Topic human readable 2012-06-14 11:36:34 -07:00
userstatus.sql Allowing setting user status 2012-05-03 18:24:30 -07:00
usertranslation.sql Allow user override translation and implement PhutilPerson 2012-06-14 18:33:00 -07:00