Commit Graph

44 Commits

Author SHA1 Message Date
bfredl
13aa23b62a refactor(tests): run unittests using main nvim binary in interpreter mode
This allows us to get rid of the separate "nvim-test" target
2023-01-31 19:28:26 +01:00
dundargoc
5046b4b4ad
ci: add cirrus to isCI function to skip tests (#20526)
The environment variable CIRRUS_CI is manually passed to RunTests.cmake
as it doesn't get passed when using cmake script mode.
2022-10-17 17:16:31 +02:00
dundargoc
600136cfb6
revert: "build: remove unnecessary policy related code" #20289
This partially reverts commit 42aeb5c5b1.

Setting cmake policies is normally not required as
cmake_minimum_required automatically sets these. One exception is cmake
script mode (-P) since it automatically resets all policy changes.

Closes: https://github.com/neovim/neovim/issues/20286
2022-09-23 07:16:17 -07:00
dundargoc
42aeb5c5b1 build: remove unnecessary policy related code
Having cmake version 3.10 as the required minimum version ensures these
are set to new by default.
2022-09-08 14:15:13 +02:00
Justin M. Keyes
a9cb0b8a5a
fix(tests): remove misleading $TEST_PATH segment #19050
Problem:
RunTests.cmake adds $TEST_PATH to $TMPDIR with the implication that it
gives more isolation. But this is misleading because $TEST_PATH is only
defined once. Full test runs use the same $TMPDIR for all tests.

This was likely added with the intention of invoking RunTests.cmake
once-per-testfile from a wrapper than does the isolation/orchestration.
But even so, Nvim's vim_maketempdir() / vim_mktempdir() _already_
creates a unique tempdir per session.

Solution:
Don't append $TEST_PATH to $TMPDIR. Avoids confusion and makes the path
shorter.
2022-06-23 16:15:55 -07:00
Justin M. Keyes
4d9e2247c9
refactor(log): simplify log_path_init #18898
Problem:
Since 22b52dd462 #11501, log_path_init is called in log_init, so it is
now called at a deterministic time. So the "just in time" complexity of
log_path_init is no longer needed.

Solution:
Remove logic intended to try to "heal" partial initialization.
2022-06-08 08:47:34 -07:00
zeertzjq
1495c4d166 ci: show failed message on Windows 2022-03-03 08:16:31 +08:00
Dundar Göc
82c5a02050 ci: skip tests that fail on windows 2022-02-20 10:22:39 +01:00
James McCoy
267ec67fab
test: allow excluding functional/unit tests using TEST_FILTER_OUT
Although this can already be done using `BUSTED_ARGS`, it complements
our existing shortcut of `TEST_FILTER.`

[skip ci]
2021-12-18 12:55:09 -05:00
Javier Lopez
33e79237bc build(tests): isolate the user environment XDG_DATA_DIRS #16003
Problem:
Some tests were not passing on my machine, specifically in
`test/functional/api/vim_spec.lua` the two tests under
`describe('nvim_get_runtime_file...`

Solution:
Unset `XDG_DATA_DIRS` in the test runner.
Note: Window CI failed if we set it to the same value as `XDG_DATA_HOME`.
2021-10-12 20:12:11 -07:00
Justin M. Keyes
7b822d4b4b
Merge branch 'master' into histfile 2021-09-10 07:05:11 -07:00
Jun T
c7ccb26260 do not write into user's ~/.bash_history 2021-09-10 18:21:29 +09:00
James McCoy
8d93223f83
test: use $TEST_TIMEOUT to specify timeout 2021-09-09 20:49:26 -04:00
Justin M. Keyes
5e5a329ea2
test: timeout at 20 minutes #15597
Set a maximum test run-time of 20 minutes to:
- fail faster
- avoid wasting CI resources
- set a bound on the test time (if tests take longer than 20 minutes we
  need to invest in parallelizing them...)

Timeout looks like:

    -- Tests exited non-zero: Process terminated due to timeout
    -- No output to stderr.
    CMake Error at /…/neovim/cmake/RunTests.cmake:86
    (message):
      functional tests failed with error: Process terminated due to timeout
2021-09-08 07:21:28 -07:00
Daniel Hahler
431cf56e2f
ci: RunTests: ensure that the logfile gets displayed (#10597)
This is meant to help with test failures where e.g. `busted` itself might
crash or get killed.

Ref: https://github.com/neovim/neovim/issues/10182#issuecomment-514450069
2019-07-24 18:12:55 +02:00
erw7
6614543316 test: Force $TEST_FILE to relative path [ci skip] #10535
- Always force TEST_PATH to a relative path.
- Eliminate rel_test_path. (The code simplification is worth the extra
  subdirectory below Xtest_tmpdir/.)
2019-07-19 13:51:18 +02:00
Daniel Hahler
34c6e5beaf
tests: move "busted" dir to "test" (#10518)
* tests: move "busted" dir to "test"

It is used for outputHandlers only, and clearly belongs to the tests.
Use the full module name with the `-o` option to `busted` then for
clarity.

* luacheck

* test/busted/outputHandlers/TAP.lua: use/extend upstream
2019-07-16 22:02:50 +02:00
Daniel Hahler
39ba35b38d cmake/RunTests.cmake: fix TEST_TAG/TEST_FILTER [ci skip] #10371
Followup to f1f9a2f97.
The mentioned example there did not work after all, likely due to
changing quoting during PR review.
2019-06-29 19:15:48 +02:00
Daniel Hahler
35b959c600
CMakeLists: remove/cleanup passing of CMAKE_SYSTEM_NAME (#10351)
Just set it from `${CMAKE_HOST_SYSTEM_NAME}` directly, instead of passing it
from the main CMake file (CMAKE_SYSTEM_NAME defaults to it, but is empty
in script mode).

Initially added in 9ce81f7b2, but then even used with unrelated commands
(that do not use RunTests.cmake, e.g. 221f6fffa).
2019-06-27 20:14:00 +02:00
Daniel Hahler
93297cfc6f
tests: busted: do not use "--lua" (#10303)
The feature is very rudimentary [1], and causes problems when not using
bundled deps, where it might then fail to find `busted.runner` due to
this.

E.g. with "-DUSE_BUNDLED=OFF -DUSE_BUNDLED_LUAROCKS=ON":
`.deps/usr/bin/busted` contains `exec '/usr/bin/luajit' -e …`, i.e. it
run luajit with adjusted lpath etc.
But using `--lua /usr/bin/luajit` then causes this `busted` wrapper to
be replaced with just `/usr/bin/luajit`, i.e. it is missing the adjusted
lpath then.

1: 9eb6c9cf93
2019-06-24 08:47:02 +02:00
Daniel Hahler
f1f9a2f97b
tests: improve RunTests.cmake (#10239)
- allow for passing in BUSTED_ARGS via env
- quote values of TEST_TAG/TEST_FILTER

  Previously TEST_FILTER="'foo bar'" was required.

This allows for:

    make functionaltest TEST_FILE=test/functional/terminal/tui_spec.lua \
      BUSTED_ARGS="--no-keep-going --shuffle" \
      TEST_FILTER="TUI background color handles"

* RunTests.cmake: BUILD_DIR for Xtest files, isolated TMPDIR

Assume relative path for given TEST_PATH.
2019-06-22 06:44:52 +02:00
Justin M. Keyes
221f6fffad runtime/lua/vim/compat.lua
ref #9280
Introduce the `vim.compat` module, to help environments with system Lua
5.2+ run the build/tests. Include the module implicitly in all tests.

ref #8677
legacy `vim` module:
    beep
    buffer
    command
    dict
    eval
    firstline
    lastline
    line
    list
    open
    type
    window
2018-12-20 11:57:30 +01:00
Justin M. Keyes
f95e0ae926
build: adjust some messages (#9087) 2018-10-06 18:45:34 +02:00
Justin M. Keyes
1cc3b32f4c build/test: skip empty TEST_TAG, TEST_FILTER
- Empty "--tags=", "--filter=" prevents TEST_FILE from running.
- Fix another bug: TEST_FILTER overwrote TEST_TAG.
2018-06-16 02:12:35 +02:00
Matthieu Coudron
6a826fce91 busted: explicit LUA_PATH to test scripts (#7864)
in case LUA_PATH does not contain `./?.lua`, busted can not load lua test scripts.
(for instance on nixos). Thus we make it explicit.
2018-01-20 14:43:16 +01:00
Justin M. Keyes
5c08c8c009 test: force LC_ALL=en_US.UTF-8
Tests that check localized error messages need a stable locale, else
errors like this occur:

    [  FAILED  ] 2 tests, listed below:
    [  FAILED  ] ...npack/file/vim/neovim/test/functional/eval/null_spec.lua @ 29: NULL list is accepted as an empty list by writefile()
    ...npack/file/vim/neovim/test/functional/eval/null_spec.lua:30: Expected objects to be the same.
    Passed in:
    (string) '
    E484: Cannot open file Xtest-functional-viml-null'
    Expected:
    (string) '
    E484: Can't open file Xtest-functional-viml-null'

    stack traceback:
            ...npack/file/vim/neovim/test/functional/eval/null_spec.lua:30: in function <...npack/file/vim/neovim/test/functional/eval/null_spec.lua:29>

    [  FAILED  ] ...k/file/vim/neovim/test/functional/ex_cmds/write_spec.lua @ 81: :write errors out correctly
    ...k/file/vim/neovim/test/functional/ex_cmds/write_spec.lua:97: Expected objects to be the same.
    Passed in:
    (string) 'Vim(write):E510: Cannot make backup file (add ! to override)'
    Expected:
    (string) 'Vim(write):E510: Can't make backup file (add ! to override)'

    stack traceback:
            ...k/file/vim/neovim/test/functional/ex_cmds/write_spec.lua:97: in function <...k/file/vim/neovim/test/functional/ex_cmds/write_spec.lua:81>

     10 SKIPPED TESTS
     2 FAILED TESTS
    -- Output to stderr:
    2017/07/29 00:41:32 ERROR 31133/open_log_file:170: Logging to stderr, failed to open $NVIM_LOG_FILE: Xtest-startup-xdg-logpath/nvim/log
    2017/07/29 00:41:32 WARN  31133/call_set_error:815: RPC: ch 1 was closed by the client

    CMake Error at /home/shlomif/Download/unpack/file/vim/neovim/cmake/RunTests.cmake:50 (message):
      Running functional tests failed with error: 1.

    FAILED: CMakeFiles/functionaltest
    cd /home/shlomif/Download/unpack/file/vim/neovim/build && /usr/bin/cmake -DBUSTED_PRG=/home/shlomif/Download/unpack/file/vim/neovim/.deps/usr/bin/busted -DLUA_PRG=/home/shlomif/Download/unpack/file/vim/neovim/.deps/usr/bin/luajit -DNVIM_PRG=/home/shlomif/Download/unpack/file/vim/neovim/build/bin/nvim -DWORKING_DIR=/home/shlomif/Download/unpack/file/vim/neovim -DBUSTED_OUTPUT_TYPE=nvim -DTEST_DIR=/home/shlomif/Download/unpack/file/vim/neovim/test -DBUILD_DIR=/home/shlomif/Download/unpack/file/vim/neovim/build -DTEST_TYPE=functional -DSYSTEM_NAME=Linux -P /home/shlomif/Download/unpack/file/vim/neovim/cmake/RunTests.cmake
    ninja: build stopped: subcommand failed.
    Makefile:102: recipe for target 'functionaltest' failed
    make: *** [functionaltest] Error 1
2017-07-29 14:21:58 +02:00
Justin M. Keyes
a49c92fc5b test: Set $NVIM_LOG_FILE to test-local path
- Do not delete it: may need to inspect it after tests finished.
- Avoids writing to stderr in cases where the test-local $XDG_DATA_HOME
  was not created yet.
2017-06-07 00:27:26 +02:00
Justin M. Keyes
1ea9ebf112 test: Use workspace-local temp directory.
Closes #6291
2017-03-30 02:55:00 +02:00
Justin M. Keyes
30826cb2d6 build: oldtest target: TEST_FILE, NVIM_PRG (#6098)
- Add support for TEST_FILE to the `oldtest` target, for consistency
  with the busted/lua tests.
  Caveat: with the busted/lua tests TEST_FILE takes a full path, whereas
  for `oldtest` it must be "test_foo.res".
- Add support for NVIM_PRG, again so that all test-related targets are
  consistent.
- Use consistent name for NVIM_PRG. But still need to support NVIM_PROG
  for QuickBuild CI.

Note: The `oldtest` target is driven by the top-level Makefile, because
it requires a TTY. CMake 3.2 added a USES_TERMINAL flag to
add_custom_target(). But we support CMake 2.8...

    add_custom_target(oldtest
      COMMAND make clean
      COMMAND make NVIM_PRG=$<TARGET_FILE:nvim> $ENV{MAKEOVERRIDES}
      DEPENDS nvim
      WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/src/nvim/testdir"
      USES_TERMINAL true
    )
2017-02-12 01:02:54 +01:00
Justin M. Keyes
bbfdb84ae1 build: Do not force busted into front of $PATH
This was a workaround from long ago, but it doesn't seem to be needed
anymore. And it breaks the $PATH on the Windows build (AppVeyor CI).

After this change python3 (and 2) is correctly detected on AppVeyor CI.

References #5946
2017-02-04 11:07:49 +01:00
Rui Abreu Ferreira
9ce81f7b2b functionaltest: Create lua helper for os.tmpname()
In Windows Lua's os.tmpname() returns relative paths starting with \s,
prepend them with $TEMP to generate a valid path.

In OS X os.tmpname() returns paths in '/tmp' but they should be in
'/private/tmp'. We cannot use os_name() for platform detection because
some tests use tempname() before nvim is spawned, instead use one of the
following:

1. Set SYSTEM_NAME environment variable before calling the tests, it
   is set from CMAKE_SYSTEM_NAME(i.e. uname -s or 'Windows')
2. Call uname -s
3. Assume windows
2016-08-31 11:32:28 +01:00
Tommy Allen
966f1abd8b tests: Set env variables in RunTests.cmake
- NVIM_RPLUGIN_MANIFEST
- XDG_CONFIG_HOME
- XDG_DATA_HOME
2016-08-17 16:22:21 -04:00
Josh Triplett
fc98d2d96d Pass busted the path to the detected Lua interpreter
Otherwise, busted may run a different interpreter than the one we
detected, without the capabilities we expect.  (For instance, we might
have detected the luajit interpreter, but busted might run the lua
interpreter, without the ffi module.)
2016-07-13 00:37:50 -07:00
Thiago de Arruda
1ee7ca7bc0 test: Improve functional test debuggability and efficiency
- Read TEST_TAG/TEST_FILTER env vars from cmake/RunTests.cmake. Setting these
  environment variables will pass --tags/--filter to busted, which can used to
  filter which tests are executed.
- Remove calls to nvim msgpack-rpc API outside tests. This removes the
  requirement of having a static `clear` call in test/functional/helpers.lua
- Use the new busted command-line option "--lazy" to ensure the setup/teardown
  hooks are only executed when a suite runs at least one test.

Now its possible to run/debug a single test like this:

```sh
TEST_FILTER='some test string' make test
```

Which will only run tests containing "some test string" in the title.
Another option is:

```sh
TEST_TAG=some-tag make test
```

After putting #some-tag into the test title. This also improves debugging
experience because there will be no unnecessary gdbserver instances whe GDB=1 is
passed.
2015-03-24 12:46:15 -03:00
John Szakmeister
afae3e25d5 build: add support for running the tests in junit format
This requires a couple of extra modules that are not installed by
default, and it requires capturing stdout of the tests--otherwise CMake
output is intermixed with the XML output of busted.
2015-03-21 09:47:49 -04:00
John Szakmeister
d8ef23849a tests: prevent busted from reloading the ffi module and others
It turns out that Busted started cleaning the environment in 2.0rc5 as a
result of Olivine-Labs/busted#62.  This, in turn, caused the ffi module
to be reloaded for each spec file, and LuaJIT doesn't appreciate it.
The net effect is an assertion error in LuaJIT.

By using the --helper feature of Busted, we can pre-load some modules
ahead of Busted and prevent it from reloading them--making LuaJIT happy
again.
2015-03-01 15:25:39 -05:00
Florian Walch
281cd3856c CMake: Hide stderr output for successful test runs.
Fixes #1301.
2015-02-06 13:39:23 +01:00
John Szakmeister
346e07e8da build: print the error result when the tests fail
Any diagnostic information is useful when things fail.  In my case, it
printed out the fact that the tests were segfaulting.
2014-11-25 06:15:57 -05:00
Thiago de Arruda
5e5525a78b test: Set VIMRUNTIME before running tests
This is required for testing scripts under the 'runtime' directory
2014-11-05 14:02:04 -03:00
Florian Walch
afa5831d73 CMake: Fix running individual (functional/unit) test. 2014-11-05 13:57:29 +01:00
John Szakmeister
5ba1d980fb build: fix running of functional tests directly with CMake
If you aren't just building everything into build/, then the functional
tests fail because they can't find the nvim executable.  Let's pass in
the location of the nvim executable, and set NVIM_PRG environment
variable accordingly.
2014-11-05 07:26:35 -05:00
Thiago de Arruda
69561ea922 test: Remove run-functional-tests.py
Now that the lua client is available, python/lupa are no longer necessary to run
the functional tests. The helper functions previously defined in
run-functional-tests.py were adapted to test/functional/helpers.lua.
2014-10-16 09:21:37 -03:00
Thiago de Arruda
549fc9548d test: Move 'test/legacy' to 'test/functional'
Busted can only discover tests from a single directory. In order to allow tests
under 'legacy' to run as a functional test, it needed to be moved to
'test/functional'.
2014-10-01 09:05:28 -03:00
Thiago de Arruda
42d5b526b9 test: Replace vroom by lua/busted for functional tests
The 'lupa' python package provides a simple way to seamless integrate lua and
python code.

This commit replaces vroom by a python script that exposes the 'neovim' package
to a lua state, and invokes busted to run functional tests. This is a temporary
solution that will enable writing functional tests using lua/bused while a lua
client library is not available.

The reason for dropping vroom is flexibility: Lua/busted has a nice DSL-style
syntax while also providing the customization power of a full programming
language. Another reason is to use a single framework for unit/functional tests.

Two other changes were performed in this commit:

- Instead of "gcc-unittest/gcc-ia32", the travis builds for gcc are now
  identified by "gcc/gcc-32". They will run unit/functional tests for both 64
  and 32 bits.
- Old integration tests(in src/nvim/testdir) are now ran by the 'oldtest' target
2014-09-30 17:37:16 -03:00