Summary:
I am hitting this error when generating Diviner documentation:
```
COMMAND
'/usr/src/phabricator/bin/diviner' atomize --ugly --book $SOME_BOOK --atomizer 'DivinerPHPAtomizer' -- $SOME_PATHS
STDOUT
(empty)
STDERR
[2015-02-18 23:05:01] EXCEPTION: (RuntimeException) Undefined variable: type at [<phutil>/src/error/PhutilErrorHandler.php:210]
#0 PhutilErrorHandler::handleError(integer, string, string, integer, array) called at [<phabricator>/src/applications/diviner/atomizer/DivinerPHPAtomizer.php:315]
#1 DivinerPHPAtomizer::parseReturnType(DivinerAtom, XHPASTNode) called at [<phabricator>/src/applications/diviner/atomizer/DivinerPHPAtomizer.php:116]
#2 DivinerPHPAtomizer::executeAtomize(string, string) called at [<phabricator>/src/applications/diviner/atomizer/DivinerAtomizer.php:23]
#3 DivinerAtomizer::atomize(string, string, array) called at [<phabricator>/src/applications/diviner/workflow/DivinerAtomizeWorkflow.php:109]
#4 DivinerAtomizeWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:396]
#5 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:292]
#6 PhutilArgument... (87 more bytes) ... at [<phutil>/src/future/exec/ExecFuture.php:416]
#0 ExecFuture::resolvex(NULL) called at [<phutil>/src/future/exec/ExecFuture.php:438]
#1 ExecFuture::resolveJSON() called at [<phabricator>/src/applications/diviner/workflow/DivinerGenerateWorkflow.php:349]
#2 DivinerGenerateWorkflow::resolveAtomizerFutures(array, array) called at [<phabricator>/src/applications/diviner/workflow/DivinerGenerateWorkflow.php:209]
#3 DivinerGenerateWorkflow::buildAtomCache() called at [<phabricator>/src/applications/diviner/workflow/DivinerGenerateWorkflow.php:170]
#4 DivinerGenerateWorkflow::generateBook(string, PhutilArgumentParser) called at [<phabricator>/src/applications/diviner/workflow/DivinerGenerateWorkflow.php:74]
#5 DivinerGenerateWorkflow::execute(PhutilArgumentParser) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:396]
#6 PhutilArgumentParser::parseWorkflowsFull(array) called at [<phutil>/src/parser/argument/PhutilArgumentParser.php:292]
#7 PhutilArgumentParser::parseWorkflows(array) called at [<phabricator>/scripts/diviner/diviner.php:21]
```
Test Plan: N/A
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D11807
Summary: Use `PhutilXHPASTBinary` methods instead of `xhpast_parse` functions. Depends on D11517.
Test Plan: N/A, this is a direct swap.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Differential Revision: https://secure.phabricator.com/D11612
Summary: Fixes T6890. This doesn't feel like a perfect solution, but I can't think of any cases in which this will produce the wrong result either.
Test Plan: Ran `./bin/diviner generate` and checked the generated documentation for `PhabricatorCommonPasswords::loadWordlist()`. The return type was corrected shown as `map<string, bool>`.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: Korvin, epriestley
Maniphest Tasks: T6890
Differential Revision: https://secure.phabricator.com/D11469
Summary: In a PHP5.3+ codebase with closures, Diviner would pick up anonymous functions and add them into the generated documentation. This causes them to be skipped.
Test Plan: Ran `bin/diviner generate --clean` before and after change, no longer got a bunch of unnamed functions dumped into the documentation.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley, #blessed_reviewers
Subscribers: epriestley, Korvin
Differential Revision: https://secure.phabricator.com/D9786
Summary: Ran `arc lint --apply-patches --everything` over rP, mainly to change double quotes to single quotes where appropriate. These changes also validate that the `ArcanistXHPASTLinter::LINT_DOUBLE_QUOTE` rule is working as expected.
Test Plan: Eyeballed it.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin, hach-que
Differential Revision: https://secure.phabricator.com/D9431
Summary:
Ref T988. Currently, every class/function needs to be annotated with `@group`, but 99% of this data can be inferred from file structure, at least in this project. Allow group specifications like:
"paste" : {
"name" : "Paste",
"include" : "(^src/applications/paste/)"
}
..to automatically put everything defined there in the "paste" group. A list of regexps is also supported. Depends on D6855.
Test Plan: Regenerated documentation with `bin/diviner generate --book src/docs/book/phabricator.book --clean`, observed all Paste stuff go in the paste group.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6856
Summary: Ref T988. Adds support for the "abstract" and "final" keywords in the atomizer.
Test Plan: Looked at abstract/final stuff.
Reviewers: btrahan, chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6850
Summary:
Ref T988. As mentioned elsewhere, one broad goal of this iteration is to reduce the amount of boilerplate documentation we need to write. For example:
- I want to set `@group` by default in most cases.
- I want to inherit things like `@param` and `@return` by default.
- I want to inherit method documentation by default.
- I want to inherit `@task` information.
This implements most of the method inheritance stuff.
This //looks// super gross, but I believe we now compose all of the information of interest at display time and can work on rendering it sensibly in the near future.
Test Plan: {F56790}
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran, sascha-egerer
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6849
Summary:
Ref T988. This brings the class/interface atomizer over. A lot of parts of this are still varying degrees of very-rough, but most of the data ends up in approximatley the right place.
ALSO: PROGRESS BARS
Test Plan: See screenshots.
Reviewers: btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6817
Summary:
Ref T988. Various improvements:
- Generate function documentation, mostly correctly.
- Raise some warnings about bad documentation.
- Allow `.book` files to exclude paths from generation.
- Add a book for technical docs.
- Exclude "ghosts" from common queries (atoms which used to exist, but no longer do, but which we want to keep the PHIDs around for in case they come back later).
This is a bit rough still, but puts us much closer to being able to get rid of the old Diviner.
Test Plan: See screenshots.
Reviewers: btrahan, chad
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D6812
Summary:
- Currently, the atomizers don't emit atoms with the right file in all cases. Make them always emit it correctly.
- Currently, we use absolute paths in some cases and relative paths in other cases. Use them consistently: relative when storing/presenting, absolute when accessing data.
- Don't preserve linebreaks when marking up documentation (documentation is generally wrapped at 80col, but should not be wrapped in this way when displayed).
- Markup Diviner link rules (albeit uselesly).
Test Plan:
Before:
{F33044}
After:
{F33045}
Reviewers: chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D4992
Summary:
- Complete the "project" -> "book" stuff. This is cleaner conceptually and keeps us from having yet another meaning for the word "project".
- Normalize symbols during atomization. This simplifies publishing a great deal, and allows static documentation to link to dynamic documentation and vice versa, because the canonical names of symbols are agreed upon (we can tweak the actual algorithm).
- Give articles a specifiable name distinct from the title, and default to something like "support" instead of "Get Help! Get Support!" so URIs end up more readable (not "Get_Help!_Get_Support!").
- Have the atomizers set book information on atoms.
- Implement very basic publishers. Publishers are basically glue code between the atomization process and the rendering process -- the two we'll have initially are "static" (publish to files on disk) and "phabricator" (or similar -- publish into the database).
- Handle duplicate symbol definitions in the atomize and publish pipelines. This fixes the issue where a project defines two functions named "idx()" and we currently tell them not to do that and break. Realistically, this is common in the real world and we should just roll our eyes and do the legwork to generate documentation as best we can.
- Particularly, dirty all atoms with the same name as a dirty atom (e.g., if 'function f()' is updated, regnerate the documentation for all functions named f() in the book).
- When publishing, we publish these at "function/f/@1", "function/f/@2". The base page will offer to disambiguate ("There are 8 functions named 'f' in this codebase, which one do you want?").
- Implement a very very basic renderer. This generates the actual HTML (or text, or XML, or whatever else) for the documentation, which the publisher dumps onto disk or into a database or whatever.
- The atomize workflow actually needs to depend on books, at least sort of, so make it load config and use it properly.
- Propagate multilevel dirties through the graph. If "C extends B" and "B extends A", we should regenerate C when A changes. Prior to this diff, we would regnerate B only.
Test Plan: Generated some documentation. Named two articles "feedback", generated docs, saw "article/feedback/@1/" and "article/feedback/@2/" created.
Reviewers: btrahan, vrana, chad
Reviewed By: chad
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D4896
Summary:
This implements most/all of the difficult parts of Diviner on top of Phabricator instead of as standalone components. See T988. In particular, here are the things I want to fix:
**Performance** The Diviner parser works in two stages. The first stage breaks source files into "Atoms". The second stage renders atoms into a display format (e.g., HTML). Diviner currently has a good caching story on the first step of the pipeline, but zero caching in the second step. This means it's very slow, even for a fairly small project like Phabricator. We must re-render every piece of documentation every time, instead of only changed documentation. Most of this diff concerns itself with addressing this problem. There's a fairly large explanatory comment about it, but the trickiest part is that when an atom changes, other atoms (defined in other places) may also change -- for example, if `class B extends A`, editing A should dirty B, even if B is in an entirely different file. We perform analysis in two stages to propagate these changes: first detecting direct changes, then detecting indirect changes. This isn't completely implemented -- we need to propagate 'extends' through more levels -- but I believe it's structurally correct and good enough until we actually document classes.
**Inheritance** Diviner currently has a very weak story on inheritance. I want to inherit a lot more metas/docs. If an interface documents a method, we should just pull that documentation in to every implementation by default (implementations can still override it if they want). It can be shown in grey or something, but it should be desirable and correct to omit documentation of a method implementation when you are implementing a parent. Similarly, I want to pull in inherited methods and @tasks and such. This diff sets up for that, by formalizing "extends" relationships between atoms.
**Overspecialization** Diviner currently specializes atoms (FileAtom, FunctionAtom, ClassAtom, etc.). This is pretty much not useful, because Atomizers (which produce the atoms) need to be highly specialized, and Renderers/Publishers (which consume the atoms) also need to be highly specialized. Nothing interesting actually lives in the atom specializations, and we don't benefit from having them -- it just costs us generality in storage/caches for them. In the new code, I've used a single Atom class to represent any type of atom.
**URIs** We have fairly hideous URIs right now, which are very cumbersome For in-app doc links, I want to provide nice URIs ("/h/notfications" or similar) which are stable redirects, and probably add remarkup for it: !{notifications} or similar. This diff isn't related to that since it's too premature.
**Search** Once we have a database generation target, we can index the documentation.
**Design** Chad has some nice mocks.
Test Plan: Ran `bin/diviner generate`, `bin/diviner generate --clean`. Saw appropriate graph propagation after edits. This diff doesn't do anything very useful yet.
Reviewers: btrahan, vrana
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T988
Differential Revision: https://secure.phabricator.com/D4340
