Summary: This updates the C# tools unit test engine (the derived version of xUnit.NET test engine that supports code coverage) to work after the updates in D7594.
Test Plan:
Switched the test engine from 'XUnitTestEngine' to 'CSharpToolsTestEngine' and ran `arc unit --everything`. The results were identical except that code coverage was provided.
Tested on Linux; will update or comment based on whether it works on Windows as well.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7640
Summary: This just removes the `print "Discovered test .."` code in the xUnit.NET unit test engine. If users need to diagnose what tests are being debugged, it can be found out with `--trace` and observing what build commands are invoked.
Test Plan: Ran `arc unit --everything` and didn't see the message any more.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7639
Summary: This fixes a few issues in the C# unit test engine. It fixes tests sitting in subdirectories not being tested correctly (the location of both the test assembly and the results file would be wrong). It also fixes a very strange issue where xUnit.NET seems to not output the resulting XML file when it executes; in this case we just retry running the test until the XML file appears after completion (and eventually it works).
Test Plan: Ran `arc unit --everything` and `arc unit --everything --no-coverage` and verified that it's all reliably working.
Reviewers: epriestley, #blessed_reviewers, hach-que
Reviewed By: epriestley
CC: Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7594
Summary:
somewhat related to D7271 and D7377.
Not actually broken right now, but might be worth it for completeness.
Test Plan: arc unit invokes PHPUnit; Can't test Csharp/XUnit on my end.
Reviewers: epriestley, #blessed_reviewers
Reviewed By: epriestley
CC: aurelijus, Korvin, epriestley, aran
Differential Revision: https://secure.phabricator.com/D7381
Summary:
In case that the $source_path is `.` or empty, it produces filenames that
look like `./foo.py`, which differential doesn't like.
Test Plan: arc diff, see coverage data.
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, aran
Differential Revision: https://secure.phabricator.com/D7181
Summary:
Many test frameworks can format their output in xUnit-like format.
Test Plan: Tested the Nose engine with a Nose one, and the pytest with a demo project.
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, aran
Differential Revision: https://secure.phabricator.com/D7011
Conflicts:
src/__phutil_library_map__.php
Summary: PHP whines about this:
> [2013-09-23 08:09:22] ERROR 2048: Declaration of CSharpToolsTestEngine::loadEnvironment() should be compatible with XUnitTestEngine::loadEnvironment($config_item = 'unit.xunit...') at [/INSECURE/devtools/arcanist/src/unit/engine/CSharpToolsTestEngine.php:13]
Auditors: jamesr, btrahan
Summary:
This implements Arcanist support for the xUnit testing framework. It also supports code coverage by way of `cstools` (which is something I've written).
The unit test support works under both Linux and Windows, while the code coverage support has only been tested under Linux (one would assume it would also work under Windows given that Windows has a super-set of functionality in the C# world).
The Arcanist support assumes that the directory layout will be something like:
* MyProject
* MyProject.Tests
When files are changed in either MyProject or MyProject.Tests, it causes MyProject.Tests to be built and the xUnit runner to be executed on the resulting binary.
Test Plan: I guess if really wanted to, you could create a C# project in MonoDevelop, set up `.arcconfig` to point to this unit test engine, and download and build xUnit and cstools. Run `arc unit --coverage` to see the results of your unit test coverage.
Reviewers: epriestley
Reviewed By: epriestley
CC: Korvin, aran
Maniphest Tasks: T3859
Differential Revision: https://secure.phabricator.com/D7058
Summary: PhpunitTestEngine incorrectly included some source files as tests when run on a case-insensive fiesystem. This fixes that behavior. It could introduce problems for case-sensive users, but it's extremely unlikely to be an issue in practice (who would put a file in both Tests/ and tests/ and expect different results for the two?)
Test Plan: arc unit before and after change on OS X. Stubs and autoloaders in .../tests/... directories are no longer picked up.
Reviewers: epriestley
Reviewed By: epriestley
CC: aurelijus, Korvin, aran
Differential Revision: https://secure.phabricator.com/D7048
Summary:
If present, this will override the default phpunit path. Allows easy
integration of Composer-provided installs of phpunit (ex. set phpunit_binary to
vendor/bin/phpunit). If the path provided doesn't resolve to an executable it
will assume the path is relative to the project root.
fix line length issue
Test Plan:
Added phpunit_binary to .arcconfig in a simple project using Composer with
phpunit/phpunit package as part of require-dev (installed to
$ROOT/vendor/bin/phpunit). Phpunit not otherwise installed on the system. Set
unit.engine to PhpunitTestEngine. Confirmed that 'arc unit' used the specified
binary, both at project root and from subdirectories.
Reviewers: epriestley
CC: aurelijus, Korvin, aran
Differential Revision: https://secure.phabricator.com/D6791
Summary: Found an issue where if arc unit runs phpunit and the datasets included a dataprovider which spanned multiple lines, it wasn't filtered out correctly in the result parser, this fixes it
Test Plan: accidentally ran tests against phpunit itself which exhibits this problem. no longer a problem after this fix, nothing else breaks.
Reviewers: epriestley
Reviewed By: epriestley
CC: aurelijus, epriestley, aran
Differential Revision: https://secure.phabricator.com/D6773
Summary: The test result parser in PhpunitTestEngine was receiving $test_path from the previous loop instead of $path from the current one. The variable isn't actually used in the PhpunitResultParser object (it exists for strict compatibility with the parent class) so it didn't cause any problems, but who knows if that could change in the future
Test Plan: Review diff. No changes to the output of running 'arc unit' when using the Phpunit engine, as expected
Reviewers: epriestley
CC: aran, epriestley, aurelijus, chad
Differential Revision: https://secure.phabricator.com/D6587
Summary: We use custom `assert*` functions here and there. Remove them from backtrace.
Test Plan: Ran `XHPASTTreeTestCase`.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D6411
Summary:
When `arc diff` runs unit tests it uses all of the affected files as the base array of paths. These files may have been deleted. If the deleted file fits the test criteria `unit` will try and run the test and in some cases fail.
An example of this fataling is with PHPUnit;
> Fatal error: Uncaught exception 'Exception' with message 'JSON report file is em
pty, it probably means that phpunit failed to run tests. Try running arc unit wi
th --trace option and then run generated phpunit command yourself, you might get
the answer.' in C:\Websites\facebook\arcanist\src\unit\engine\PhpunitResultPars
er.php on line 156
Test Plan: Re-ran the tests that were causing issues with `arc unit --rev HEAD^ --trace`
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin, aurelijus
Differential Revision: https://secure.phabricator.com/D6246
Summary: On Windows consoles, the star on the unit tests (I'm assuming it's a Unicode character of some kind), doesn't render correctly and you just get garbage. This fixes it so that on Windows, it falls back to just using an ASCII asterisk.
Test Plan: On Windows run some unit tests, the star should now be a plain asterisk.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D6087
Summary: Replace this old hard-coded implementation with the new vector-based, unicode-capable one.
Test Plan: Ran unit tests. Looked at revisions in Differential, using whitespace modes to bypass cache.
Reviewers: btrahan, Afaque_Hussain
Reviewed By: btrahan
CC: aran
Maniphest Tasks: T2379
Differential Revision: https://secure.phabricator.com/D6016
Summary:
The NoseTestEngine class has some usefull code for parsing nosetest, but it assumes a very specific code/test
layout. I've pulled this logic abit apart, which lets me reuse the nose-related parts with my existing
project layout, by having my custom engine just call runTests() with the relevant paths.
Test Plan: Run on a customized project with coverage, see coverage results.
Reviewers: epriestley, roman.barzyczak
Reviewed By: epriestley
CC: seporaitis, aran, Korvin, zeeg
Differential Revision: https://secure.phabricator.com/D5921
Summary: unlike the hooks I copy/pasted, these hooks have parameters
Test Plan: read it
Reviewers: vrana
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D5264
Summary: These hooks allow test cases to build shared resources -- notably, database fixtures.
Test Plan: See next diff.
Reviewers: vrana
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D5258
Summary: We don't set $paths when running --everything.
Test Plan: Ran `arc unit --everything` with coverage.
Reviewers: vrana
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D5260
Summary:
Some tests take longer (fixtures usually around 1 second for me) and also FB runs all tests on deploy.
I want to see all results immediately.
Test Plan:
Added `usleep(200000)` to `resultTest()`, then:
$ arc unit
Saw results printed one by one.
Also didn't pass `$renderer` to `ArcanistPhutilTestCase` and saw empty output.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D5141
Test Plan: Ran PhpunitTestEngine unit test and used both test result parsers to generate test results.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin, aurelijus
Differential Revision: https://secure.phabricator.com/D4676
Summary: PHPUnit 3.7 now includes user message as well.
Test Plan: Ran phpunit tests
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D4662
Test Plan: Ran PhpunitTestEngine unit test. Also used refactored PhpunitTestEngine to run phpunit tests.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin, aurelijus
Differential Revision: https://secure.phabricator.com/D4651
Summary: Adds "arc unit --everything", which runs every available test, provided the test engine supports it. Also add JSON output.
Test Plan: Ran `arc unit --everything` in arcanist/, libphutil/ and phabricator/. Saw all tests run.
Reviewers: btrahan, vrana
Reviewed By: vrana
CC: aran
Maniphest Tasks: T2065
Differential Revision: https://secure.phabricator.com/D4214
Summary:
This commit doesn't change license of any file. It just makes the license implicit (inherited from LICENSE file in the root directory).
We are removing the headers for these reasons:
- It wastes space in editors, less code is visible in editor upon opening a file.
- It brings noise to diff of the first change of any file every year.
- It confuses Git file copy detection when creating small files.
- We don't have an explicit license header in other files (JS, CSS, images, documentation).
- Using license header in every file is not obligatory: http://www.apache.org/dev/apply-license.html#new.
This change is approved by Alma Chao (Lead Open Source and IP Counsel at Facebook).
Test Plan: Verified that the license survived only in unit tests and LICENSE file.
Reviewers: epriestley, btrahan, edward
Reviewed By: epriestley
CC: aran, Korvin, davidrecordon
Maniphest Tasks: T2035
Differential Revision: https://secure.phabricator.com/D3881
Summary: Currently, a shutdown exception ("script exited with open transactions!") overwhelms the actual test failure exception, which is the one that needs to be fixed.
Test Plan: Ran a fixture test which opens a transaction and then throws. Got diagnostically useful output after this patch.
Reviewers: vrana, btrahan
Reviewed By: vrana
CC: aran
Differential Revision: https://secure.phabricator.com/D3780
Summary:
See https://github.com/facebook/arcanist/pull/53
- Work correctly for directories with `%` in their name; this breaks under sprintf().
- Search for `src/` -> `tests/` style directories.
- Add coverage for search paths.
Test Plan:
Ran unit tests.
I don't have a working test case for PHPUnit tests, can one of you guys apply this and verify I didn't break your setups?
Reviewers: quard, aurelijus
Reviewed By: aurelijus
CC: aran
Differential Revision: https://secure.phabricator.com/D3558
Summary: I want to use it from outside.
Test Plan:
$ arc unit src/lint/linter/__tests__/
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D3493
Summary: This is Arcanist part of D3434.
Test Plan: None.
Reviewers: epriestley, wez
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D3450
Summary:
See discussion on T1630.
extraData provides more scope for extensions to piggy-back
more data on the test results and have that pulled up to the UI.
We're using keys like "facebook:complexity" to store additional
data as part of the test results.
Test Plan:
Nothing in the codebase touches extraData at the moment, so
you'll just have to have faith/prove by inspection.
Reviewers: vrana, nh, tuomaspelkonen, epriestley
Reviewed By: epriestley
CC: aran, epriestley
Maniphest Tasks: T1630
Differential Revision: https://secure.phabricator.com/D3276
Summary:
According to @epriestley, it's nasty and kind of crazy: D2933#1.
It also stands in my way for D2614.
Test Plan: Rewrote our callsite to event listener and verified that it still works.
Reviewers: tuomaspelkonen, epriestley
Reviewed By: epriestley
CC: aran, epriestley
Differential Revision: https://secure.phabricator.com/D3171
Test Plan: Show Full Unit Results on this diff.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D2777
Summary:
I just hope that we will not create `ArcanistPhutilTestCaseTestCaseTestCase`.
Also fix a copy/paste error (@edward's this time).
Test Plan: `arc unit`
Reviewers: epriestley
Reviewed By: epriestley
CC: edward, aran, Korvin
Differential Revision: https://secure.phabricator.com/D2692
Summary: Almost revert D2673 but leave the support for 'repeat'.
Test Plan: `arc help unit`.
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D2689
Summary:
We run all tests before deploy but some of them are expected to fail.
We need to skip them.
It can be useful also for someone else.
I don't propagate this to `arc diff` as that would be much more complicated - we would need a new state 'partially skipped'.
The 'path' argument needs to be a file, skipping whole dirs is not supported.
The 'path' argument is in fact engine specific and engines can support format like 'Class::testMethod' but I didn't bother to document it to not confuse people.
Test Plan:
`arc unit --skip test.php`
`arc unit --skip ../src/test.php`
Reviewers: epriestley
Reviewed By: epriestley
CC: aran, Korvin
Differential Revision: https://secure.phabricator.com/D2673
