- Read TEST_TAG/TEST_FILTER env vars from cmake/RunTests.cmake. Setting these
environment variables will pass --tags/--filter to busted, which can used to
filter which tests are executed.
- Remove calls to nvim msgpack-rpc API outside tests. This removes the
requirement of having a static `clear` call in test/functional/helpers.lua
- Use the new busted command-line option "--lazy" to ensure the setup/teardown
hooks are only executed when a suite runs at least one test.
Now its possible to run/debug a single test like this:
```sh
TEST_FILTER='some test string' make test
```
Which will only run tests containing "some test string" in the title.
Another option is:
```sh
TEST_TAG=some-tag make test
```
After putting #some-tag into the test title. This also improves debugging
experience because there will be no unnecessary gdbserver instances whe GDB=1 is
passed.
This is can be used for spawning nvim outside a test context. Also refactor
screen.lua to use this function when loading the color map(It is better because
the GDB/VALGRIND environment variables are ignored)
This is necessary for newer versions of Busted, otherwise assert will be
nil and the tests will die.
Note: this does not mean the tests now work with the latest Busted.
There are still several issues preventing that from happening.
When a test that fails leaves nvim in a 'Press Enter...' state, the whole suite
will hang because the `qa!` command executed before the next test won't be
processed until '<enter>' is sent.
Now the lua client can send a signal with when `Session:exit()` is called, so
the `qa!` request is no longer necessary.
Also:
- Set noswapfile at startup to prevent tests from leaving .s* swap files(should
also improve test environment determinism)
- Use `assert(false, msg) instead of `error(msg)` to report screen assertion
failures.
The $GDB env var can be set to run tests under gdbserver. If $VALGRIND is also
set, it will add the --vgdb=yes command-line option to valgrind instead of
starting gdbserver.
The vim_input function accepts raw terminal input and so is better to emulate
real user, especially because it is not deferred as vim_feedkeys.
Using this function required a number of changes:
- expect() was refactored to use curbuf_contents()
- The vim_eval function in request() was moved to curbuf_contents(). For most
cases this is enough(we only care for synchronizing api calls with user input
when verifying buffer contents).
- <C-@>(NUL) is preprocessed before being passed to replace_termcodes.
- Legacy test 4 had a bug that only became visible when using vim_input, it is
fixed now.
- An extra blank line deletion was required for test 101
The last two items show that vim_feedkeys because it is not 100% equivalent to
receiving terminal input.
Tests which spin the event loop and stop it in a notification handler have a
chance of re-entering the event loop due to the `vim_eval` call in the
`request()` helper(assuming the request call is what triggered the
notification). Since this will cause an error to be thrown by the lua client,
don't send the extra `vim_eval` request when the loop has been stopped.
During test setup, we used to call a vimscript function(BeforeEachTest) that
attempted to restore Nvim to it's initial state as much as possible in order to
provide a clean environment for running new tests. This approach has proven to
be unreliable, as some tests leave state that can affect other tests, eventually
causing failures that are difficult to debug.
This commit changes the 'clear' function so it will restart Nvim every time it
is called, which is a slower, but more reliable solution that will simplify
spotting bugs in the future.
Some other improvements/fixes were also performed:
- Whenever an error is detected in a handler passed to "run()", the event loop
will be stopped and the error will be propagated to the main thread.
- Errors and the "cleanup()" function will always send a quit command to the
current Nvim instance. This should prevent memory starvation when running
tests under valgrind(where each Nvim instance can consume a lot of memory).
- Fixed a wrong assertion in server_requests_spec.lua. Previously the failure
was undetected in a notification handler.
- Fixed some tests to expect fully clean registers. The deleted cleanup function
used to put an empty string in every register, but that resulted in a extra
line being added.
- Move the cleanup function definition into `restart()` so restart can be
selectively used as a hook
- Improve error handling: Before this, errors while running the event loop would
cause busted to get stuck. Now the error is properly raised by stopping the
event loop first.
Sanity API checks made by the python-client in the api-python travis target were
converted to lua and will now live in this repository. This will simplify
performing breaking changes to the API as it won't be necessary to send parallel
PRs the python-client.
Now that the lua client is available, python/lupa are no longer necessary to run
the functional tests. The helper functions previously defined in
run-functional-tests.py were adapted to test/functional/helpers.lua.
The 'lupa' python package provides a simple way to seamless integrate lua and
python code.
This commit replaces vroom by a python script that exposes the 'neovim' package
to a lua state, and invokes busted to run functional tests. This is a temporary
solution that will enable writing functional tests using lua/bused while a lua
client library is not available.
The reason for dropping vroom is flexibility: Lua/busted has a nice DSL-style
syntax while also providing the customization power of a full programming
language. Another reason is to use a single framework for unit/functional tests.
Two other changes were performed in this commit:
- Instead of "gcc-unittest/gcc-ia32", the travis builds for gcc are now
identified by "gcc/gcc-32". They will run unit/functional tests for both 64
and 32 bits.
- Old integration tests(in src/nvim/testdir) are now ran by the 'oldtest' target
