Eliminates lua-client and non-static libluv as test time dependencies
Note: the API for a public lua-client is not yet finished.
The interface needs to be adjusted to work in the embedded loop
of a nvim instance (to use it to talk between instances)
This partially reverts commit 42aeb5c5b1.
Setting cmake policies is normally not required as
cmake_minimum_required automatically sets these. One exception is cmake
script mode (-P) since it automatically resets all policy changes.
Closes: https://github.com/neovim/neovim/issues/20286
Problem:
RunTests.cmake adds $TEST_PATH to $TMPDIR with the implication that it
gives more isolation. But this is misleading because $TEST_PATH is only
defined once. Full test runs use the same $TMPDIR for all tests.
This was likely added with the intention of invoking RunTests.cmake
once-per-testfile from a wrapper than does the isolation/orchestration.
But even so, Nvim's vim_maketempdir() / vim_mktempdir() _already_
creates a unique tempdir per session.
Solution:
Don't append $TEST_PATH to $TMPDIR. Avoids confusion and makes the path
shorter.
Problem:
Since 22b52dd462#11501, log_path_init is called in log_init, so it is
now called at a deterministic time. So the "just in time" complexity of
log_path_init is no longer needed.
Solution:
Remove logic intended to try to "heal" partial initialization.
Problem:
Some tests were not passing on my machine, specifically in
`test/functional/api/vim_spec.lua` the two tests under
`describe('nvim_get_runtime_file...`
Solution:
Unset `XDG_DATA_DIRS` in the test runner.
Note: Window CI failed if we set it to the same value as `XDG_DATA_HOME`.
Set a maximum test run-time of 20 minutes to:
- fail faster
- avoid wasting CI resources
- set a bound on the test time (if tests take longer than 20 minutes we
need to invest in parallelizing them...)
Timeout looks like:
-- Tests exited non-zero: Process terminated due to timeout
-- No output to stderr.
CMake Error at /…/neovim/cmake/RunTests.cmake:86
(message):
functional tests failed with error: Process terminated due to timeout
* tests: move "busted" dir to "test"
It is used for outputHandlers only, and clearly belongs to the tests.
Use the full module name with the `-o` option to `busted` then for
clarity.
* luacheck
* test/busted/outputHandlers/TAP.lua: use/extend upstream
Just set it from `${CMAKE_HOST_SYSTEM_NAME}` directly, instead of passing it
from the main CMake file (CMAKE_SYSTEM_NAME defaults to it, but is empty
in script mode).
Initially added in 9ce81f7b2, but then even used with unrelated commands
(that do not use RunTests.cmake, e.g. 221f6fffa).
The feature is very rudimentary [1], and causes problems when not using
bundled deps, where it might then fail to find `busted.runner` due to
this.
E.g. with "-DUSE_BUNDLED=OFF -DUSE_BUNDLED_LUAROCKS=ON":
`.deps/usr/bin/busted` contains `exec '/usr/bin/luajit' -e …`, i.e. it
run luajit with adjusted lpath etc.
But using `--lua /usr/bin/luajit` then causes this `busted` wrapper to
be replaced with just `/usr/bin/luajit`, i.e. it is missing the adjusted
lpath then.
1: 9eb6c9cf93
- allow for passing in BUSTED_ARGS via env
- quote values of TEST_TAG/TEST_FILTER
Previously TEST_FILTER="'foo bar'" was required.
This allows for:
make functionaltest TEST_FILE=test/functional/terminal/tui_spec.lua \
BUSTED_ARGS="--no-keep-going --shuffle" \
TEST_FILTER="TUI background color handles"
* RunTests.cmake: BUILD_DIR for Xtest files, isolated TMPDIR
Assume relative path for given TEST_PATH.
ref #9280
Introduce the `vim.compat` module, to help environments with system Lua
5.2+ run the build/tests. Include the module implicitly in all tests.
ref #8677
legacy `vim` module:
beep
buffer
command
dict
eval
firstline
lastline
line
list
open
type
window
Tests that check localized error messages need a stable locale, else
errors like this occur:
[ FAILED ] 2 tests, listed below:
[ FAILED ] ...npack/file/vim/neovim/test/functional/eval/null_spec.lua @ 29: NULL list is accepted as an empty list by writefile()
...npack/file/vim/neovim/test/functional/eval/null_spec.lua:30: Expected objects to be the same.
Passed in:
(string) '
E484: Cannot open file Xtest-functional-viml-null'
Expected:
(string) '
E484: Can't open file Xtest-functional-viml-null'
stack traceback:
...npack/file/vim/neovim/test/functional/eval/null_spec.lua:30: in function <...npack/file/vim/neovim/test/functional/eval/null_spec.lua:29>
[ FAILED ] ...k/file/vim/neovim/test/functional/ex_cmds/write_spec.lua @ 81: :write errors out correctly
...k/file/vim/neovim/test/functional/ex_cmds/write_spec.lua:97: Expected objects to be the same.
Passed in:
(string) 'Vim(write):E510: Cannot make backup file (add ! to override)'
Expected:
(string) 'Vim(write):E510: Can't make backup file (add ! to override)'
stack traceback:
...k/file/vim/neovim/test/functional/ex_cmds/write_spec.lua:97: in function <...k/file/vim/neovim/test/functional/ex_cmds/write_spec.lua:81>
10 SKIPPED TESTS
2 FAILED TESTS
-- Output to stderr:
2017/07/29 00:41:32 ERROR 31133/open_log_file:170: Logging to stderr, failed to open $NVIM_LOG_FILE: Xtest-startup-xdg-logpath/nvim/log
2017/07/29 00:41:32 WARN 31133/call_set_error:815: RPC: ch 1 was closed by the client
CMake Error at /home/shlomif/Download/unpack/file/vim/neovim/cmake/RunTests.cmake:50 (message):
Running functional tests failed with error: 1.
FAILED: CMakeFiles/functionaltest
cd /home/shlomif/Download/unpack/file/vim/neovim/build && /usr/bin/cmake -DBUSTED_PRG=/home/shlomif/Download/unpack/file/vim/neovim/.deps/usr/bin/busted -DLUA_PRG=/home/shlomif/Download/unpack/file/vim/neovim/.deps/usr/bin/luajit -DNVIM_PRG=/home/shlomif/Download/unpack/file/vim/neovim/build/bin/nvim -DWORKING_DIR=/home/shlomif/Download/unpack/file/vim/neovim -DBUSTED_OUTPUT_TYPE=nvim -DTEST_DIR=/home/shlomif/Download/unpack/file/vim/neovim/test -DBUILD_DIR=/home/shlomif/Download/unpack/file/vim/neovim/build -DTEST_TYPE=functional -DSYSTEM_NAME=Linux -P /home/shlomif/Download/unpack/file/vim/neovim/cmake/RunTests.cmake
ninja: build stopped: subcommand failed.
Makefile:102: recipe for target 'functionaltest' failed
make: *** [functionaltest] Error 1
- Do not delete it: may need to inspect it after tests finished.
- Avoids writing to stderr in cases where the test-local $XDG_DATA_HOME
was not created yet.
- Add support for TEST_FILE to the `oldtest` target, for consistency
with the busted/lua tests.
Caveat: with the busted/lua tests TEST_FILE takes a full path, whereas
for `oldtest` it must be "test_foo.res".
- Add support for NVIM_PRG, again so that all test-related targets are
consistent.
- Use consistent name for NVIM_PRG. But still need to support NVIM_PROG
for QuickBuild CI.
Note: The `oldtest` target is driven by the top-level Makefile, because
it requires a TTY. CMake 3.2 added a USES_TERMINAL flag to
add_custom_target(). But we support CMake 2.8...
add_custom_target(oldtest
COMMAND make clean
COMMAND make NVIM_PRG=$<TARGET_FILE:nvim> $ENV{MAKEOVERRIDES}
DEPENDS nvim
WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/src/nvim/testdir"
USES_TERMINAL true
)
This was a workaround from long ago, but it doesn't seem to be needed
anymore. And it breaks the $PATH on the Windows build (AppVeyor CI).
After this change python3 (and 2) is correctly detected on AppVeyor CI.
References #5946
In Windows Lua's os.tmpname() returns relative paths starting with \s,
prepend them with $TEMP to generate a valid path.
In OS X os.tmpname() returns paths in '/tmp' but they should be in
'/private/tmp'. We cannot use os_name() for platform detection because
some tests use tempname() before nvim is spawned, instead use one of the
following:
1. Set SYSTEM_NAME environment variable before calling the tests, it
is set from CMAKE_SYSTEM_NAME(i.e. uname -s or 'Windows')
2. Call uname -s
3. Assume windows
Otherwise, busted may run a different interpreter than the one we
detected, without the capabilities we expect. (For instance, we might
have detected the luajit interpreter, but busted might run the lua
interpreter, without the ffi module.)
- Read TEST_TAG/TEST_FILTER env vars from cmake/RunTests.cmake. Setting these
environment variables will pass --tags/--filter to busted, which can used to
filter which tests are executed.
- Remove calls to nvim msgpack-rpc API outside tests. This removes the
requirement of having a static `clear` call in test/functional/helpers.lua
- Use the new busted command-line option "--lazy" to ensure the setup/teardown
hooks are only executed when a suite runs at least one test.
Now its possible to run/debug a single test like this:
```sh
TEST_FILTER='some test string' make test
```
Which will only run tests containing "some test string" in the title.
Another option is:
```sh
TEST_TAG=some-tag make test
```
After putting #some-tag into the test title. This also improves debugging
experience because there will be no unnecessary gdbserver instances whe GDB=1 is
passed.
This requires a couple of extra modules that are not installed by
default, and it requires capturing stdout of the tests--otherwise CMake
output is intermixed with the XML output of busted.
It turns out that Busted started cleaning the environment in 2.0rc5 as a
result of Olivine-Labs/busted#62. This, in turn, caused the ffi module
to be reloaded for each spec file, and LuaJIT doesn't appreciate it.
The net effect is an assertion error in LuaJIT.
By using the --helper feature of Busted, we can pre-load some modules
ahead of Busted and prevent it from reloading them--making LuaJIT happy
again.
If you aren't just building everything into build/, then the functional
tests fail because they can't find the nvim executable. Let's pass in
the location of the nvim executable, and set NVIM_PRG environment
variable accordingly.
Now that the lua client is available, python/lupa are no longer necessary to run
the functional tests. The helper functions previously defined in
run-functional-tests.py were adapted to test/functional/helpers.lua.
Busted can only discover tests from a single directory. In order to allow tests
under 'legacy' to run as a functional test, it needed to be moved to
'test/functional'.
The 'lupa' python package provides a simple way to seamless integrate lua and
python code.
This commit replaces vroom by a python script that exposes the 'neovim' package
to a lua state, and invokes busted to run functional tests. This is a temporary
solution that will enable writing functional tests using lua/bused while a lua
client library is not available.
The reason for dropping vroom is flexibility: Lua/busted has a nice DSL-style
syntax while also providing the customization power of a full programming
language. Another reason is to use a single framework for unit/functional tests.
Two other changes were performed in this commit:
- Instead of "gcc-unittest/gcc-ia32", the travis builds for gcc are now
identified by "gcc/gcc-32". They will run unit/functional tests for both 64
and 32 bits.
- Old integration tests(in src/nvim/testdir) are now ran by the 'oldtest' target
