Summary: Of note this now forces the pytest-cov plugin to be installed unless they turn it off.
Test Plan:
```
../arcanist/bin/arc unit
COVERAGE REPORT
97% changes/models/test.py
100% tests/changes/api/serializer/models/test_testcase.py
100% changes/api/serializer/models/testcase.py
UNIT OKAY No unit test failures.
Updated an existing Differential revision:
Revision URI: https://tails.corp.dropbox.com/D43387
Included changes:
M changes/api/serializer/models/testcase.py
M changes/models/test.py
M tests/changes/api/serializer/models/test_testcase.py
```
Reviewers: #blessed_reviewers, epriestley
Reviewed By: #blessed_reviewers, epriestley
Subscribers: epriestley, Korvin
Differential Revision: https://secure.phabricator.com/D8667
Summary: See discussion in D8460. Primarily, this prints out the failing value when a true/false assertion fails, so if it was something useful (like a function result) it's visible.
Test Plan:
Added `assertTrue("quack")`:
FAIL ArcanistDiffParserTestCase::testParser
Assertion failed, expected 'true' (at ArcanistDiffParserTestCase.php:16).
ACTUAL VALUE
quack
Added `assertFalse("quack")`:
FAIL ArcanistDiffParserTestCase::testParser
Assertion failed, expected 'false' (at ArcanistDiffParserTestCase.php:16).
ACTUAL VALUE
quack
Added `assertEqual("quack", "moo")`:
FAIL ArcanistDiffParserTestCase::testParser
Assertion failed, expected values to be equal (at ArcanistDiffParserTestCase.php:16).
Expected: quack
Actual: moo
Reviewers: joshuaspence
Reviewed By: joshuaspence
CC: aran
Differential Revision: https://secure.phabricator.com/D8465
Summary:
There are quite a few tests in Arcanist, libphutil and Phabricator that do something similar to `$this->assertEqual(false, ...)` or `$this->assertEqual(true, ...)`.
This is unnecessarily verbose and it would be cleaner if we had `assertFalse` and `assertTrue` methods.
Test Plan: I contemplated adding a unit test for the `getCallerInfo` method but wasn't sure if it was required / where it should live.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D8460
Summary:
I noticed that code coverage wasn't showing in Differential for
some repositories that we are using with Phabricator.
`arc unit` would should unit test coverage, but the paths were messy
(for example, `.//foo/bar.py` instead of `foo/bar.py`). As a result,
the code coverage info wasn't recognised as being for the correct
module.
I'm not sure why this logic is the way that it is... perhaps this is to
do with an older version of `nose` (I am using v1.3.0).
Test Plan:
I created a diff for an internal repository that we have, and
observed that code coverage was displayed in Differential.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: Korvin, epriestley, aran, seporaitis, avive, dctrwatson, zeeg
Differential Revision: https://secure.phabricator.com/D8433
Summary:
Fixes T4570. When a test case doesn't make any assertions, fail it:
- A tiny fraction of tests pass by not throwing. These tests can easily make a trivial assertion. It took about 5 minutes to fix them all (D8435, D8436).
- In other cases, no assertions means a test construction problem, as with T4570. In these cases, failing loudly catches a severe error.
- Fixes the no-assertion test cases in `arcanist/`
- Makes the PHP 5.4 test pass for the moment, see discussion in T4334.
Test Plan: Ran `arc unit --everything`.
Reviewers: leebyron, btrahan
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T4570
Differential Revision: https://secure.phabricator.com/D8437
Summary: This might not be universally desireable, but I found myself writing an additional linter (which I had called `WhitespaceTextLinter`) for the sake of these two linter tests. I figured it may be of use upstream, and so I decided to submit it as a diff. I won't be offended if it is rejected however.
Test Plan: `arc lint` and `arc unit` are both okay with it.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7957
Summary: Currently, running `arc unit -- src/` returns with `No tests to run`, even if there are test classes in `src/__tests__/`. This diff changes this behaviour so that `arc unit -- src/` executes unit tests in `src/__tests__/`.
Test Plan: N/A. I suppose you could create a file `src/__tests__/SomeTestCase.php` and see for yourself.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7915
Summary:
Before, if PHPUnit crashed (syntax error, undefined constant, etc), you would get a relatively unhelpful error message:
firehed@Eric-Sterns-Mac-Pro ~/dev/php-lcd: arc unit
Exception
Clover coverage XML report file is empty, it probably means that phpunit failed to run tests. Try running arc unit with --trace option and then run generated phpunit command yourself, you might get the answer.
(Run with --trace for a full exception trace.)
This now checks the json and code coverage reports and tries to pull a useful error message:
firehed@Eric-Sterns-Mac-Pro ~/dev/php-lcd: arc unit
Exception
The test '/Users/firehed/dev/php-lcd/tests/PlateButtonsTest.php' crashed with the following output:
Fatal error: Undefined class constant 'EFT' in /Users/firehed/dev/php-lcd/tests/PlateButtonsTest.php on line 25
Call Stack:
0.0002 233104 1. {main}() /Users/firehed/dev/php-lcd/vendor/phpunit/phpunit/composer/bin/phpunit:0
0.0039 564872 2. PHPUnit_TextUI_Command::main() /Users/firehed/dev/php-lcd/vendor/phpunit/phpunit/composer/bin/phpunit:63
0.0039 565496 3. PHPUnit_TextUI_Command->run() /Users/firehed/dev/php-lcd/vendor/phpunit/phpunit/PHPUnit/TextUI/Command.php:129
0.0247 2280168 4. PHPUnit_TextUI_TestRunner->doRun() /Users/firehed/dev/php-lcd/vendor/phpunit/phpunit/PHPUnit/TextUI/Command.php:176
0.0293 2730760 5. PHPUnit_Framework_TestSuite->run() /Users/firehed/dev/php-lcd/vendor/phpunit/phpunit/PHPUnit/TextUI/TestRunner.php:349
0.1211 3996832 6. PHPUnit_Framework_TestSuite->runTest() /Users/firehed/dev/php-lcd/vendor/phpunit/phpunit/PHPUnit/Framework/TestSuite.php:745
0.1211 3996832 7. PHPUnit_Framework_TestCase->run() /Users/firehed/dev/php-lcd/vendor/phpunit/phpunit/PHPUnit/Framework/TestSuite.php:775
0.1211 3996832 8. PHPUnit_Framework_TestResult->run() /Users/firehed/dev/php-lcd/vendor/phpunit/phpunit/PHPUnit/Framework/TestCase.php:783
0.1233 3999752 9. PHPUnit_Framework_TestCase->runBare() /Users/firehed/dev/php-lcd/vendor/phpunit/phpunit/PHPUnit/Framework/TestResult.php:648
0.1236 4016432 10. PHPUnit_Framework_TestCase->runTest() /Users/firehed/dev/php-lcd/vendor/phpunit/phpunit/PHPUnit/Framework/TestCase.php:838
0.1237 4017240 11. ReflectionMethod->invokeArgs() /Users/firehed/dev/php-lcd/vendor/phpunit/phpunit/PHPUnit/Framework/TestCase.php:983
0.1237 4017520 12. Firehed\PlateButtonsTest->testSingleButtonOnButtonDown() /Users/firehed/dev/php-lcd/vendor/phpunit/phpunit/PHPUnit/Framework/TestCase.php:983
(Run with --trace for a full exception trace.)
Or if nothing was written to `stderr`, you may get something like this:
firehed@Eric-Sterns-Mac-Pro ~/dev/php-lcd: arc unit --no-coverage
Exception
Test Firehed\PlateButtonsTest::testSingleButtonOnButtonDown did not finish
(Run with --trace for a full exception trace.)
Test Plan:
`arc unit` for arcanist itself
`arc unit` before and after the change on a project where:
* all tests pass
* some tests are failing
* a test causes a fatal error (undefined constant, bad method call, etc)
* a test file is invalid (syntax error)
No regressions that I could find, and all crashes now display a more useful error.
Reviewers: #blessed_reviewers, epriestley
Reviewed By: epriestley
CC: aurelijus, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7848
Summary: This updates the C# tools unit test engine (the derived version of xUnit.NET test engine that supports code coverage) to work after the updates in D7594.
Test Plan:
Switched the test engine from 'XUnitTestEngine' to 'CSharpToolsTestEngine' and ran `arc unit --everything`. The results were identical except that code coverage was provided.
Tested on Linux; will update or comment based on whether it works on Windows as well.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7640
Summary: This just removes the `print "Discovered test .."` code in the xUnit.NET unit test engine. If users need to diagnose what tests are being debugged, it can be found out with `--trace` and observing what build commands are invoked.
Test Plan: Ran `arc unit --everything` and didn't see the message any more.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7639
Summary: This fixes a few issues in the C# unit test engine. It fixes tests sitting in subdirectories not being tested correctly (the location of both the test assembly and the results file would be wrong). It also fixes a very strange issue where xUnit.NET seems to not output the resulting XML file when it executes; in this case we just retry running the test until the XML file appears after completion (and eventually it works).
Test Plan: Ran `arc unit --everything` and `arc unit --everything --no-coverage` and verified that it's all reliably working.
Reviewers: epriestley, #blessed_reviewers, hach-que
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7594
Summary:
somewhat related to D7271 and D7377.
Not actually broken right now, but might be worth it for completeness.
Test Plan: arc unit invokes PHPUnit; Can't test Csharp/XUnit on my end.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: aurelijus, Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7381
Summary:
In case that the $source_path is `.` or empty, it produces filenames that
look like `./foo.py`, which differential doesn't like.
Test Plan: arc diff, see coverage data.
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, aran
Differential Revision: https://secure.phabricator.com/D7181
Summary:
Many test frameworks can format their output in xUnit-like format.
Test Plan: Tested the Nose engine with a Nose one, and the pytest with a demo project.
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, aran
Differential Revision: https://secure.phabricator.com/D7011
Conflicts:
src/__phutil_library_map__.php
Summary: PHP whines about this:
> [2013-09-23 08:09:22] ERROR 2048: Declaration of CSharpToolsTestEngine::loadEnvironment() should be compatible with XUnitTestEngine::loadEnvironment($config_item = 'unit.xunit...') at [/INSECURE/devtools/arcanist/src/unit/engine/CSharpToolsTestEngine.php:13]
Auditors: jamesr, btrahan
Summary:
This implements Arcanist support for the xUnit testing framework. It also supports code coverage by way of `cstools` (which is something I've written).
The unit test support works under both Linux and Windows, while the code coverage support has only been tested under Linux (one would assume it would also work under Windows given that Windows has a super-set of functionality in the C# world).
The Arcanist support assumes that the directory layout will be something like:
* MyProject
* MyProject.Tests
When files are changed in either MyProject or MyProject.Tests, it causes MyProject.Tests to be built and the xUnit runner to be executed on the resulting binary.
Test Plan: I guess if really wanted to, you could create a C# project in MonoDevelop, set up `.arcconfig` to point to this unit test engine, and download and build xUnit and cstools. Run `arc unit --coverage` to see the results of your unit test coverage.
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, aran
Maniphest Tasks: T3859
Differential Revision: https://secure.phabricator.com/D7058
Summary: PhpunitTestEngine incorrectly included some source files as tests when run on a case-insensive fiesystem. This fixes that behavior. It could introduce problems for case-sensive users, but it's extremely unlikely to be an issue in practice (who would put a file in both Tests/ and tests/ and expect different results for the two?)
Test Plan: arc unit before and after change on OS X. Stubs and autoloaders in .../tests/... directories are no longer picked up.
Reviewers: epriestley
Reviewed By: epriestley
CC: aurelijus, Korvin, aran
Differential Revision: https://secure.phabricator.com/D7048
Summary:
If present, this will override the default phpunit path. Allows easy
integration of Composer-provided installs of phpunit (ex. set phpunit_binary to
vendor/bin/phpunit). If the path provided doesn't resolve to an executable it
will assume the path is relative to the project root.
fix line length issue
Test Plan:
Added phpunit_binary to .arcconfig in a simple project using Composer with
phpunit/phpunit package as part of require-dev (installed to
$ROOT/vendor/bin/phpunit). Phpunit not otherwise installed on the system. Set
unit.engine to PhpunitTestEngine. Confirmed that 'arc unit' used the specified
binary, both at project root and from subdirectories.
Reviewers: epriestley
CC: aurelijus, Korvin, aran
Differential Revision: https://secure.phabricator.com/D6791
Summary: Found an issue where if arc unit runs phpunit and the datasets included a dataprovider which spanned multiple lines, it wasn't filtered out correctly in the result parser, this fixes it
Test Plan: accidentally ran tests against phpunit itself which exhibits this problem. no longer a problem after this fix, nothing else breaks.
Reviewers: epriestley
Reviewed By: epriestley
CC: aurelijus, epriestley, aran
Differential Revision: https://secure.phabricator.com/D6773
Summary: The test result parser in PhpunitTestEngine was receiving $test_path from the previous loop instead of $path from the current one. The variable isn't actually used in the PhpunitResultParser object (it exists for strict compatibility with the parent class) so it didn't cause any problems, but who knows if that could change in the future
Test Plan: Review diff. No changes to the output of running 'arc unit' when using the Phpunit engine, as expected
Reviewers: epriestley
CC: aran, epriestley, aurelijus, chad
Differential Revision: https://secure.phabricator.com/D6587
Summary: We use custom `assert*` functions here and there. Remove them from backtrace.
Test Plan: Ran `XHPASTTreeTestCase`.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D6411
Summary:
When `arc diff` runs unit tests it uses all of the affected files as the base array of paths. These files may have been deleted. If the deleted file fits the test criteria `unit` will try and run the test and in some cases fail.
An example of this fataling is with PHPUnit;
> Fatal error: Uncaught exception 'Exception' with message 'JSON report file is em
pty, it probably means that phpunit failed to run tests. Try running arc unit wi
th --trace option and then run generated phpunit command yourself, you might get
the answer.' in C:\Websites\facebook\arcanist\src\unit\engine\PhpunitResultPars
er.php on line 156
Test Plan: Re-ran the tests that were causing issues with `arc unit --rev HEAD^ --trace`
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin, aurelijus
Differential Revision: https://secure.phabricator.com/D6246
Summary: On Windows consoles, the star on the unit tests (I'm assuming it's a Unicode character of some kind), doesn't render correctly and you just get garbage. This fixes it so that on Windows, it falls back to just using an ASCII asterisk.
Test Plan: On Windows run some unit tests, the star should now be a plain asterisk.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D6087
Summary: Replace this old hard-coded implementation with the new vector-based, unicode-capable one.
Test Plan: Ran unit tests. Looked at revisions in Differential, using whitespace modes to bypass cache.
Reviewers: btrahan, Afaque_Hussain
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2379
Differential Revision: https://secure.phabricator.com/D6016
Summary:
The NoseTestEngine class has some usefull code for parsing nosetest, but it assumes a very specific code/test
layout. I've pulled this logic abit apart, which lets me reuse the nose-related parts with my existing
project layout, by having my custom engine just call runTests() with the relevant paths.
Test Plan: Run on a customized project with coverage, see coverage results.
Reviewers: epriestley, roman.barzyczak
Reviewed By: epriestley
CC: seporaitis, aran, Korvin, zeeg
Differential Revision: https://secure.phabricator.com/D5921
Summary: unlike the hooks I copy/pasted, these hooks have parameters
Test Plan: read it
Reviewers: vrana
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D5264
Summary: These hooks allow test cases to build shared resources -- notably, database fixtures.
Test Plan: See next diff.
Reviewers: vrana
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D5258
Summary: We don't set $paths when running --everything.
Test Plan: Ran `arc unit --everything` with coverage.
Reviewers: vrana
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D5260
Summary:
Some tests take longer (fixtures usually around 1 second for me) and also FB runs all tests on deploy.
I want to see all results immediately.
Test Plan:
Added `usleep(200000)` to `resultTest()`, then:
$ arc unit
Saw results printed one by one.
Also didn't pass `$renderer` to `ArcanistPhutilTestCase` and saw empty output.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D5141
Test Plan: Ran PhpunitTestEngine unit test and used both test result parsers to generate test results.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin, aurelijus
Differential Revision: https://secure.phabricator.com/D4676
Summary: PHPUnit 3.7 now includes user message as well.
Test Plan: Ran phpunit tests
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4662
Test Plan: Ran PhpunitTestEngine unit test. Also used refactored PhpunitTestEngine to run phpunit tests.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin, aurelijus
Differential Revision: https://secure.phabricator.com/D4651
Summary: Adds "arc unit --everything", which runs every available test, provided the test engine supports it. Also add JSON output.
Test Plan: Ran `arc unit --everything` in arcanist/, libphutil/ and phabricator/. Saw all tests run.
Reviewers: btrahan, vrana
Reviewed By: vrana
CC: aran
Maniphest Tasks: T2065
Differential Revision: https://secure.phabricator.com/D4214
Summary:
This commit doesn't change license of any file. It just makes the license implicit (inherited from LICENSE file in the root directory).
We are removing the headers for these reasons:
- It wastes space in editors, less code is visible in editor upon opening a file.
- It brings noise to diff of the first change of any file every year.
- It confuses Git file copy detection when creating small files.
- We don't have an explicit license header in other files (JS, CSS, images, documentation).
- Using license header in every file is not obligatory: http://www.apache.org/dev/apply-license.html#new.
This change is approved by Alma Chao (Lead Open Source and IP Counsel at Facebook).
Test Plan: Verified that the license survived only in unit tests and LICENSE file.
Reviewers: epriestley, btrahan, edward
Reviewed By: epriestley
CC: aran, Korvin, davidrecordon
Maniphest Tasks: T2035
Differential Revision: https://secure.phabricator.com/D3881
Summary: Currently, a shutdown exception ("script exited with open transactions!") overwhelms the actual test failure exception, which is the one that needs to be fixed.
Test Plan: Ran a fixture test which opens a transaction and then throws. Got diagnostically useful output after this patch.
Reviewers: vrana, btrahan
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D3780
Summary:
See https://github.com/facebook/arcanist/pull/53
- Work correctly for directories with `%` in their name; this breaks under sprintf().
- Search for `src/` -> `tests/` style directories.
- Add coverage for search paths.
Test Plan:
Ran unit tests.
I don't have a working test case for PHPUnit tests, can one of you guys apply this and verify I didn't break your setups?
Reviewers: quard, aurelijus
Reviewed By: aurelijus
CC: aran
Differential Revision: https://secure.phabricator.com/D3558
Summary: I want to use it from outside.
Test Plan:
$ arc unit src/lint/linter/__tests__/
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D3493
