Summary:
Ref T9890. Ref T9979. Several adjacent goals:
- The `SearchEngine` vs `ApplicationSearchEngine` thing is really confusing. There are also a bunch of confusing class names and class relationships within the fulltext indexing. I want to rename these classes to be more standard (`IndexEngine`, `IndexEngineExtension`, etc). Rename `SearchIndexer` to `IndexEngine`. A future change will rename `SearchEngine`.
- Add the index locks described in T9890.
- Structure things a little more normally so future diffs can do the "EngineExtension" thing more cleanly.
Test Plan:
Indexing:
- Renamed a task to have a unique word in the title.
- Ran `bin/search index Txxx`.
- Searched for unique word.
- Found task.
Locking:
- Added a `sleep(10)` after the `lock()` call.
- Ran `bin/search index Txxx` in two windows.
- Saw first one lock, sleep 10 seconds, index.
- Saw second one give up temporarily after failing to grab the lock.
Reviewers: chad
Reviewed By: chad
Maniphest Tasks: T9890, T9979
Differential Revision: https://secure.phabricator.com/D14834
Ref T8618. I think this was just typoed in D13070.
Immediate issue is that we're looking for symbols with the book name as a "phid".
Auditors: joshuaspence
Summary: Fixes T8352. Associate Diviner books and atoms with a repository. This relationship is not really surfaced anywhere in the UI but provides metadata that contextualises search results. Depends on D13091.
Test Plan: Ran `diviner generate --repository ARC` and then went to `/diviner/book/arcanist/`.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T7703, T8352
Differential Revision: https://secure.phabricator.com/D13070
Summary:
Ref T4558. This diff modernizes the #diviner application. Basically:
- Add an edit controller, accessible at `/book/$BOOK/edit/`.
- Add edit/view policies.
- Added an action menu to the `DivinerBookController` to expose the edit interface.
- Allows projects to be associated with books.
- Implement edges and transactions.
- Implemented `PhabricatorApplicationTransactionInterface` in `DivinerLiveBook`.
Test Plan:
- Generated a Diviner book with `./bin/diviner generate`.
- Added projects to a book and ensured that they persisted.
- Changed the view policy on a book and made sure it was effective.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T4558
Differential Revision: https://secure.phabricator.com/D13091
Summary: All classes should extend from some other class. See D13275 for some explanation.
Test Plan: `arc unit`
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin
Differential Revision: https://secure.phabricator.com/D13283
Summary: Ref T4558. When publishing a new Diviner book, use the most-open policy instead of `PhabricatorPolicies::POLICY_USER`.
Test Plan: Ran `diviner generate` in a directory which didn't have published Diviner documentation.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Maniphest Tasks: T4558
Differential Revision: https://secure.phabricator.com/D13254
Summary: Fixes T7458. Integrates #diviner into #applicationsearch by indexing `DivinerLiveBook` and `DivinerLiveSymbol` search documents. Depends on D13157.
Test Plan: Ran `./bin/search index --all --type BOOK` and `./bin/search index --all --type ATOM` and then searched for various symbols via global search.
Reviewers: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Maniphest Tasks: T7458
Differential Revision: https://secure.phabricator.com/D13090
Summary: I missed these in D13157.
Test Plan: Browsed around #diviner.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D13158
Summary: Fixes T8401. Change `withIncludeGhosts()` to `withExcludeGhosts()` and `withIncludeUndocumentable()` to `withExcludeDocumentable()`. In particular, this allows querying for atoms by PHID to work as expected.
Test Plan: I got confused with double negatives so I might have gotten some of these wrong... I poked around Diviner and re-generated documentation to verify that this is working as expected.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Maniphest Tasks: T8401
Differential Revision: https://secure.phabricator.com/D13157
Summary: Self-explanatory. Also made a few methods `final`.
Test Plan: Eyeball it.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D11598
Summary: The method is actually named `DivinerAtomRef::newFromDictionary`.
Test Plan: `./bin/diviner generate --publisher DivinerStaticPublisher` worked a bit better.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D11590
Summary: Allow the `DivinerPublisher` subclass to be specified via `./bin/divner generate --publisher ...`. In particular, this allows use of the (mostly broken) `DivinerStaticPublisher`.
Test Plan: Ran `./bin/diviner generate --publisher DivinerStaticPublisher`
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D11588
Summary: Ran `arc lint --apply-patches --everything` over rP, mainly to change double quotes to single quotes where appropriate. These changes also validate that the `ArcanistXHPASTLinter::LINT_DOUBLE_QUOTE` rule is working as expected.
Test Plan: Eyeballed it.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin, hach-que
Differential Revision: https://secure.phabricator.com/D9431
Summary: Ref T988. Make this more useful, and link it to the methods it describes.
Test Plan:
Before:
{F57553}
After:
{F57554}
Reviewers: chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6909
Summary:
Ref T988. This is //extremely// rough looking in the UI, but gets most of the information we need into the right places.
The controller rendering code is super rough too, I'm going to break that up shortly.
- Add `needChildren()` to `DivinerAtomQuery`.
- Compose and organize class methods when rendering classes. The old Diviner was not smart enough to do this, so a class would only document methods which the class itself implemented, not methods inherited from parents. I'd like to show those too to provide a more complete understanding of how to use a class (but they'll be marked with "inherited" or somesuch). This code walks the "extends" list and builds all of the class methods, annotating them with where they are defined and where they are implemented.
- Coompose and organize "tasks". The old Diviner was not smart enough to do this, but I want to reduce the amount of duplicate/busy work involved in documenting subclasses. In particular, I want them to inherit "@task" declarations from parents so that class trees are more cohesive. They now do so.
Test Plan: See screenshots.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6823
Summary:
Ref T988. This brings the class/interface atomizer over. A lot of parts of this are still varying degrees of very-rough, but most of the data ends up in approximatley the right place.
ALSO: PROGRESS BARS
Test Plan: See screenshots.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6817
Summary:
Ref T988. Various improvements:
- Generate function documentation, mostly correctly.
- Raise some warnings about bad documentation.
- Allow `.book` files to exclude paths from generation.
- Add a book for technical docs.
- Exclude "ghosts" from common queries (atoms which used to exist, but no longer do, but which we want to keep the PHIDs around for in case they come back later).
This is a bit rough still, but puts us much closer to being able to get rid of the old Diviner.
Test Plan: See screenshots.
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6812
Summary: We need to make sure the symbol is always saved before the atom. I mucked this up recently and didn't catch it locally since I'd already generated the atoms.
Test Plan: Ran `bin/diviner generate` after truncating all diviner tables.
Reviewers: chad
Reviewed By: chad
CC: aran
Differential Revision: https://secure.phabricator.com/D6146
Summary: Gets TOC populated for articles, at least, and fixes a few other things.
Test Plan: {F45474}
Reviewers: chad
Reviewed By: chad
CC: aran
Differential Revision: https://secure.phabricator.com/D6144
Summary: Ref T988. Mostly backend changes, with a very rough frontend on top of them. See Conpherence discussion.
Test Plan: {F45010}
Reviewers: btrahan, chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6113
Summary: Ref T988. Ref T2625. Rough cut of ApplicationSearch in Diviner, for detailed Atom queries. This isn't useful yet, and isn't linked in the UI.
Test Plan: {F44836}
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988, T2625
Differential Revision: https://secure.phabricator.com/D6094
Summary:
Ref T988. This adds basics for the non-static publishing target:
- Storage (called "Live", e.g. `DivinerLiveAtom` to distinguish it from shared classes like `DivinerAtom`).
- Mostly populate the storage.
- Some minor fixes and improvements.
Test Plan: Generated docs, looked at DB, saw mostly-sensible output.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D5973
Summary:
Do this somewhat reasonably:
- For links to the same documentation book (the common case), go look up that the thing you're linking to actualy exists. If it doesn't, render a <span> which we can make have a red background and warn about later.
- For links to some other book, just generate a link and hope it hits something. We can improve and augment this later.
- For non-documentation links (links in comments, e.g.) just generate a query link into the Diviner app. We'll do a query and figure out where to send the user after they click the link. We could pre-resolve these later.
Test Plan: Generated documentation, saw it build mostly-correct links when objects were referenced correctly. Used preview to generate various `@{x:y|z}` things and made sure they ended up reasonable-looking.
Reviewers: chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D5001
Summary: Cache refs in a single large index; rebuild the main index from them.
Test Plan: {F32334}
Reviewers: btrahan, vrana, chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D4900
Summary: Keep track of what we've written to disk, and regenerate only new documents.
Test Plan: Changed a small number of files, saw that number of files get regenerated. Ran with "--clean" and saw everything regenerate.
Reviewers: btrahan, vrana, chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D4897
Summary:
- Complete the "project" -> "book" stuff. This is cleaner conceptually and keeps us from having yet another meaning for the word "project".
- Normalize symbols during atomization. This simplifies publishing a great deal, and allows static documentation to link to dynamic documentation and vice versa, because the canonical names of symbols are agreed upon (we can tweak the actual algorithm).
- Give articles a specifiable name distinct from the title, and default to something like "support" instead of "Get Help! Get Support!" so URIs end up more readable (not "Get_Help!_Get_Support!").
- Have the atomizers set book information on atoms.
- Implement very basic publishers. Publishers are basically glue code between the atomization process and the rendering process -- the two we'll have initially are "static" (publish to files on disk) and "phabricator" (or similar -- publish into the database).
- Handle duplicate symbol definitions in the atomize and publish pipelines. This fixes the issue where a project defines two functions named "idx()" and we currently tell them not to do that and break. Realistically, this is common in the real world and we should just roll our eyes and do the legwork to generate documentation as best we can.
- Particularly, dirty all atoms with the same name as a dirty atom (e.g., if 'function f()' is updated, regnerate the documentation for all functions named f() in the book).
- When publishing, we publish these at "function/f/@1", "function/f/@2". The base page will offer to disambiguate ("There are 8 functions named 'f' in this codebase, which one do you want?").
- Implement a very very basic renderer. This generates the actual HTML (or text, or XML, or whatever else) for the documentation, which the publisher dumps onto disk or into a database or whatever.
- The atomize workflow actually needs to depend on books, at least sort of, so make it load config and use it properly.
- Propagate multilevel dirties through the graph. If "C extends B" and "B extends A", we should regenerate C when A changes. Prior to this diff, we would regnerate B only.
Test Plan: Generated some documentation. Named two articles "feedback", generated docs, saw "article/feedback/@1/" and "article/feedback/@2/" created.
Reviewers: btrahan, vrana, chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D4896
