Summary: Allow the `DivinerPublisher` subclass to be specified via `./bin/divner generate --publisher ...`. In particular, this allows use of the (mostly broken) `DivinerStaticPublisher`.
Test Plan: Ran `./bin/diviner generate --publisher DivinerStaticPublisher`
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D11588
Summary: Minor tidying and modernizing a few things.
Test Plan: Ran `./bin/diviner atomize` and `./bin/diviner generate`.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D11587
Summary: Ref T6822.
Test Plan: `grep`. This method is only called from within `PhutilArgumentWorkflow::__construct`.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Maniphest Tasks: T6822
Differential Revision: https://secure.phabricator.com/D11415
Summary: Allow `./bin/diviner generate` to continue even if there is an exception throw processing an atom. This allows Diviner documentation to be generated for PHP source code that cannot be parsed with XHPAST.
Test Plan: Ran `./bin/diviner generate` on a PHP repository which previously throw an `XHPASTSyntaxErrorException`.
Reviewers: btrahan, epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D10803
Summary: Ran `arc lint --apply-patches --everything` over rP, mainly to change double quotes to single quotes where appropriate. These changes also validate that the `ArcanistXHPASTLinter::LINT_DOUBLE_QUOTE` rule is working as expected.
Test Plan: Eyeballed it.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin, hach-que
Differential Revision: https://secure.phabricator.com/D9431
Summary:
Via HackerOne. In regular expressions, "$" matches "end of input, or before terminating newline". This means that the expression `/^A$/` matches two strings: `"A"`, and `"A\n"`.
When we care about this, use `\z` instead, which matches "end of input" only.
This allowed registration of `"username\n"` and similar.
Test Plan:
- Grepped codebase for all calls to `preg_match()` / `preg_match_all()`.
- Fixed the ones where this seemed like it could have an impact.
- Added and executed unit tests.
Reviewers: btrahan
Reviewed By: btrahan
Subscribers: aran, epriestley
Differential Revision: https://secure.phabricator.com/D8516
Summary: Ref T988. This makes it easier to generate documentation.
Test Plan: Ran with and without `--book`. Examined CLI output.
Reviewers: chad, btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D8415
Summary: Ref T988. This is primarily intended to let us add the "HEY! THIS ISN'T USER DOCUMENTATION" notices to the arcanist and libphutil technical docs.
Test Plan: Added some prefaces, generated docs, looked at them.
Reviewers: chad, btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D8410
Summary:
Ref T2015. Not directly related to Drydock, but I've wanted to do this for a bit.
Introduce a common base class for all the workflows in the scripts in `bin/*`. This slightly reduces code duplication by moving `isExecutable()` to the base, but also provides `getViewer()`. This is a little nicer than `PhabricatorUser::getOmnipotentUser()` and gives us a layer of indirection if we ever want to introduce more general viewer mechanisms in scripts.
Test Plan: Lint; ran some of the scripts.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2015
Differential Revision: https://secure.phabricator.com/D7838
Summary:
Ref T988.
- Render "Implements:" as tags, too.
- Minor CSS tweak to tags in property lists.
- Add a bunch of group patterns to the Phabricator book.
- Fix some stuff with how hashes are computed and cached.
- Minor tweak to reuse the Diviner engine for slightly improved performance.
Test Plan: Regenerated and looked at documentation.
Reviewers: chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T3811, T988
Differential Revision: https://secure.phabricator.com/D6912
Summary:
Ref T988. Currently, every class/function needs to be annotated with `@group`, but 99% of this data can be inferred from file structure, at least in this project. Allow group specifications like:
"paste" : {
"name" : "Paste",
"include" : "(^src/applications/paste/)"
}
..to automatically put everything defined there in the "paste" group. A list of regexps is also supported. Depends on D6855.
Test Plan: Regenerated documentation with `bin/diviner generate --book src/docs/book/phabricator.book --clean`, observed all Paste stuff go in the paste group.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6856
Summary:
Ref T988. This brings the class/interface atomizer over. A lot of parts of this are still varying degrees of very-rough, but most of the data ends up in approximatley the right place.
ALSO: PROGRESS BARS
Test Plan: See screenshots.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6817
Summary:
Ref T988. Various improvements:
- Generate function documentation, mostly correctly.
- Raise some warnings about bad documentation.
- Allow `.book` files to exclude paths from generation.
- Add a book for technical docs.
- Exclude "ghosts" from common queries (atoms which used to exist, but no longer do, but which we want to keep the PHIDs around for in case they come back later).
This is a bit rough still, but puts us much closer to being able to get rid of the old Diviner.
Test Plan: See screenshots.
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6812
Summary: Ref T988. Mostly backend changes, with a very rough frontend on top of them. See Conpherence discussion.
Test Plan: {F45010}
Reviewers: btrahan, chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6113
Summary:
Ref T988. This adds basics for the non-static publishing target:
- Storage (called "Live", e.g. `DivinerLiveAtom` to distinguish it from shared classes like `DivinerAtom`).
- Mostly populate the storage.
- Some minor fixes and improvements.
Test Plan: Generated docs, looked at DB, saw mostly-sensible output.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D5973
Summary:
- Currently, the atomizers don't emit atoms with the right file in all cases. Make them always emit it correctly.
- Currently, we use absolute paths in some cases and relative paths in other cases. Use them consistently: relative when storing/presenting, absolute when accessing data.
- Don't preserve linebreaks when marking up documentation (documentation is generally wrapped at 80col, but should not be wrapped in this way when displayed).
- Markup Diviner link rules (albeit uselesly).
Test Plan:
Before:
{F33044}
After:
{F33045}
Reviewers: chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D4992
Summary: Take a few more steps forward toward usability.
Test Plan: {F33040}
Reviewers: chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D4991
Summary:
- Complete the "project" -> "book" stuff. This is cleaner conceptually and keeps us from having yet another meaning for the word "project".
- Normalize symbols during atomization. This simplifies publishing a great deal, and allows static documentation to link to dynamic documentation and vice versa, because the canonical names of symbols are agreed upon (we can tweak the actual algorithm).
- Give articles a specifiable name distinct from the title, and default to something like "support" instead of "Get Help! Get Support!" so URIs end up more readable (not "Get_Help!_Get_Support!").
- Have the atomizers set book information on atoms.
- Implement very basic publishers. Publishers are basically glue code between the atomization process and the rendering process -- the two we'll have initially are "static" (publish to files on disk) and "phabricator" (or similar -- publish into the database).
- Handle duplicate symbol definitions in the atomize and publish pipelines. This fixes the issue where a project defines two functions named "idx()" and we currently tell them not to do that and break. Realistically, this is common in the real world and we should just roll our eyes and do the legwork to generate documentation as best we can.
- Particularly, dirty all atoms with the same name as a dirty atom (e.g., if 'function f()' is updated, regnerate the documentation for all functions named f() in the book).
- When publishing, we publish these at "function/f/@1", "function/f/@2". The base page will offer to disambiguate ("There are 8 functions named 'f' in this codebase, which one do you want?").
- Implement a very very basic renderer. This generates the actual HTML (or text, or XML, or whatever else) for the documentation, which the publisher dumps onto disk or into a database or whatever.
- The atomize workflow actually needs to depend on books, at least sort of, so make it load config and use it properly.
- Propagate multilevel dirties through the graph. If "C extends B" and "B extends A", we should regenerate C when A changes. Prior to this diff, we would regnerate B only.
Test Plan: Generated some documentation. Named two articles "feedback", generated docs, saw "article/feedback/@1/" and "article/feedback/@2/" created.
Reviewers: btrahan, vrana, chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D4896
Summary:
I want to allow a single project to generate multiple "books" of documentation, so we can separate user-facing documentation from technical documentation and such.
Generalize the ".divinerconfig" file into a "diviner book" configuration file.
Since only the "generate" workflow actually reads any of this stuff, move it all down into the generate workflow.
Also, namespace the cache.
Test Plan: Ran `bin/diviner generate --book src/docs/user.book`, saw appropriate output. Verified cache generated in a namespace in `.divinercache/`.
Reviewers: btrahan, indiefan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D4857
Summary:
This implements most/all of the difficult parts of Diviner on top of Phabricator instead of as standalone components. See T988. In particular, here are the things I want to fix:
**Performance** The Diviner parser works in two stages. The first stage breaks source files into "Atoms". The second stage renders atoms into a display format (e.g., HTML). Diviner currently has a good caching story on the first step of the pipeline, but zero caching in the second step. This means it's very slow, even for a fairly small project like Phabricator. We must re-render every piece of documentation every time, instead of only changed documentation. Most of this diff concerns itself with addressing this problem. There's a fairly large explanatory comment about it, but the trickiest part is that when an atom changes, other atoms (defined in other places) may also change -- for example, if `class B extends A`, editing A should dirty B, even if B is in an entirely different file. We perform analysis in two stages to propagate these changes: first detecting direct changes, then detecting indirect changes. This isn't completely implemented -- we need to propagate 'extends' through more levels -- but I believe it's structurally correct and good enough until we actually document classes.
**Inheritance** Diviner currently has a very weak story on inheritance. I want to inherit a lot more metas/docs. If an interface documents a method, we should just pull that documentation in to every implementation by default (implementations can still override it if they want). It can be shown in grey or something, but it should be desirable and correct to omit documentation of a method implementation when you are implementing a parent. Similarly, I want to pull in inherited methods and @tasks and such. This diff sets up for that, by formalizing "extends" relationships between atoms.
**Overspecialization** Diviner currently specializes atoms (FileAtom, FunctionAtom, ClassAtom, etc.). This is pretty much not useful, because Atomizers (which produce the atoms) need to be highly specialized, and Renderers/Publishers (which consume the atoms) also need to be highly specialized. Nothing interesting actually lives in the atom specializations, and we don't benefit from having them -- it just costs us generality in storage/caches for them. In the new code, I've used a single Atom class to represent any type of atom.
**URIs** We have fairly hideous URIs right now, which are very cumbersome For in-app doc links, I want to provide nice URIs ("/h/notfications" or similar) which are stable redirects, and probably add remarkup for it: !{notifications} or similar. This diff isn't related to that since it's too premature.
**Search** Once we have a database generation target, we can index the documentation.
**Design** Chad has some nice mocks.
Test Plan: Ran `bin/diviner generate`, `bin/diviner generate --clean`. Saw appropriate graph propagation after edits. This diff doesn't do anything very useful yet.
Reviewers: btrahan, vrana
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D4340
