dispatch.sr.ht is being deprecated, meaning that using sourcehut CI
won't be possible (see https://github.com/neovim/neovim/issues/19609).
Since Github Actions doesn't provide any BSD runners an external service
is required and Cirrus CI seems like a good replacement for sourcehut.
Initially experimented with using FreeBSD and OpenBSD virtual machines
in GitHub Actions, but Cirrus has been a much better fit with better
performance, logs and overall experience.
Failing tests are automatically skipped on FreeBSD regardless if it's on
CI or not. Ideally these tests should only be skipped in CI with the
help of `isCI` helper function. Unfortunately, the tests don't recognize
the environment variable CIRRUS_CI even if it's set manually. This
workaround is good enough for the time being, but we might want to only
skip tests when using the CI (or even better, fix the failing tests).
Closes: https://github.com/neovim/neovim/issues/19609
Problem:
- Since c57f6b28d7#8519, sockets are created in ~/.local/… but XDG
spec says: "XDG_RUNTIME_DIR: Must be on the local filesystem", which
implies that XDG_STATE_DIR is potentially non-local.
- Not easy to inspect Nvim-created temp files (for debugging etc).
Solution:
- Store sockets in stdpath('run') ($XDG_RUNTIME_DIR).
- Establish "/tmp/nvim.user/" as the tempdir root shared by all Nvims.
- Make ok() actually useful.
- Introduce assert_nolog().
closes#3517closes#17093
Problem:
Dirs "config", "packaging", and "third-party" are all closely related
but this is not obvious from the layout. This adds friction for new
contributors.
Solution:
- rename config/ to cmake.config/
- rename test/config/ to test/cmakeconfig/ because it is used in Lua
tests: require('test.cmakeconfig.paths').
- rename packaging/ to cmake.packaging/
- rename third-party/ to cmake.deps/ (parallel with .deps/)
Problem:
RunTests.cmake adds $TEST_PATH to $TMPDIR with the implication that it
gives more isolation. But this is misleading because $TEST_PATH is only
defined once. Full test runs use the same $TMPDIR for all tests.
This was likely added with the intention of invoking RunTests.cmake
once-per-testfile from a wrapper than does the isolation/orchestration.
But even so, Nvim's vim_maketempdir() / vim_mktempdir() _already_
creates a unique tempdir per session.
Solution:
Don't append $TEST_PATH to $TMPDIR. Avoids confusion and makes the path
shorter.
Problem:
1. Log messages (especially in CI) are hard to correlate with tests.
2. Since b353a5c05f#11886, dumplog() prints the logs next to test
failures. This is noisy and gets in the way of the test results.
Solution:
1. Associate an incrementing id with each test and include it in log
messages.
- FUTURE: add v:name so Nvim instances can be formally "named"?
2. Mention "child" in log messages if the current Nvim is a child (based
on the presence of $NVIM).
BEFORE:
DBG … 12345 UI: event
DBG … 12345 log_server_msg:722: RPC ->ch 1: …
DBG … 12345 UI: flush
DBG … 12345 inbuf_poll:444: blocking... events_enabled=1 events_pending=0
DBG … 23454 UI: stop
INF … 23454 os_exit:594: Nvim exit: 0
AFTER:
DBG … T57 UI: event
DBG … T57 log_server_msg:722: RPC ->ch 1: …
DBG … T57 UI: flush
DBG … T57 inbuf_poll:444: blocking... events_enabled=1 events_pending=0
DBG … T57/child UI: stop
INF … T57/child os_exit:594: Nvim exit: 0
Read_file_list is used to read back data from the nvim log file as the
testsuite executes. However, the nvim log file can get really big
(each full run of the testsuite appends roughly 150MB of data to this
file). Reading each line of this file can thus be extremely slow, and so
are the repeated table.insert/table.removes that are done for each line.
A solution to this issue is tto only read the end of the file.
This results in a sizeable improvement in testsuite run times in some
cases, e.g. on my computer:
Without this commit:
real 20m0.431s
user 17m11.163s
sys 1m59.422s
With this commit:
real 4m25.356s
user 1m41.673s
sys 1m31.253s
Problem
- `redir_exec` is obsolete, but it keeps getting used in new tests
because people copy existing tests.
- Disadvantages of `redir_exec`:
- Captures extra junk before the actual error/message that we _want_ to test.
- Does not fail on error, unlike e.g. `command()`.
Solution
- Use new functions like `nvim_exec` and `pcall_err`.
For example, implicitly converting a table to a string works in LuaJIT,
but needs to be done explicitly with tostring() in Lua 5.1.
This can cause issues when testing a non-JIT build if eq(), for example,
fails with a table argument. E.g: eq({}, {1}) will not print the details
of the assertion failure, but will instead print a less helpful "string
expected, got table" error.
The official developer documentation in in :h dev-lua-doc specifies to
use "--@" for special/magic tokens. However, this format is not
consistent with EmmyLua notation (used by some Lua language servers) nor
with the C version of the magic docstring tokens which use three comment
characters.
Further, the code base is currently split between usage of "--@",
"---@", and "--- @". In an effort to remain consistent, change all Lua
magic tokens to use "---@" and update the developer documentation
accordingly.
- The previous commit lost information in the tests. Instead, add some
more "normalization" substitutions in pcall_err(), so that the general
shape of the stacktrace is included in the asserted text.
- Eliminate contains(), it is redundant with matches()
- Add vim variable meta accessors: vim.env, vim.{g,v,w,bo,wo}
- Redo gen_char_blob to generate multiple blobs instead of just one
so that multiple Lua modules can be inlined.
- Reorder vim.lua inclusion so that it can use previously defined C
functions and utility functions like vim.shared and vim.inspect things.
- Inline shared.lua into nvim, but also keep it available in runtime.
- Rename `meth_pcall`.
- Make `pcall_err` raise an error if the function does not fail.
- Add `vim.pesc()` to treat a string as literal where a Lua pattern is
expected.
luassert uses 3 by default, which is often not enough.
Instead of documenting how to increase it, let's use a more fitting
(sane) default of 100 levels.
This is where "pure functions" can live, which can be shared by Nvim and
test logic which may not have a running Nvim instance available.
If in the future we use Nvim itself as the Lua engine for tests, then
these functions could be moved directly onto the `vim` Lua module.
closes#6580
Automatically include all "global helper" util functions in the
unit.helpers and functional.helpers and modules. So tests don't need to
expicitly do:
local global_helpers = require('test.helpers')
