neovim/test/README.md

Tests
=====

Tests are broadly divided into *unit tests* ([test/unit](https://github.com/neovim/neovim/tree/master/test/unit/)),
*functional tests* ([test/functional](https://github.com/neovim/neovim/tree/master/test/functional/)),
and *old tests* ([src/nvim/testdir/](https://github.com/neovim/neovim/tree/master/src/nvim/testdir/)).

- _Unit_ testing is achieved by compiling the tests as a shared library which is
  loaded and called by [LuaJit FFI](http://luajit.org/ext_ffi.html).
- _Functional_ tests are driven by RPC, so they do not require LuaJit (as
  opposed to Lua).

You can learn the [key concepts of Lua in 15 minutes](http://learnxinyminutes.com/docs/lua/).
Use any existing test as a template to start writing new tests.

Tests are run by `/cmake/RunTests.cmake` file, using `busted` (a Lua test-runner).
For some failures, `.nvimlog` (or `$NVIM_LOG_FILE`) may provide insight.

Depending on the presence of binaries (e.g., `xclip`) some tests will be
ignored. You must compile with libintl to prevent `E319: The command is not
available in this version` errors.


---

- [Running tests](#running-tests)
- [Writing tests](#writing-tests)
- [Lint](#lint)
- [Configuration](#configuration)

---


Layout
======

- `/test/benchmark` : benchmarks
- `/test/functional` : functional tests
- `/test/unit` : unit tests
- `/test/config` : contains `*.in` files which are transformed into `*.lua`
  files using `configure_file` CMake command: this is for acessing CMake
  variables in lua tests.
- `/test/includes` : include-files for use by luajit `ffi.cdef` C definitions
  parser: normally used to make macros not accessible via this mechanism
  accessible the other way.
- `/test/*/preload.lua` : modules preloaded by busted `--helper` option
- `/test/**/helpers.lua` : common utility functions for test code
- `/test/*/**/*_spec.lua` : actual tests. Files that do not end with
  `_spec.lua` are libraries like `/test/**/helpers.lua`, except that they have
  some common topic.
- `/src/nvim/testdir` : old tests (from Vim)


Running tests
=============

Executing Tests
---------------

To run all tests (except "old" tests):

    make test

To run only _unit_ tests:

    make unittest

To run only _functional_ tests:

    make functionaltest


Legacy tests
------------

To run all legacy Vim tests:

    make oldtest

To run a *single* legacy test set `TEST_FILE`, for example:

    TEST_FILE=test_syntax.res make oldtest

- The `.res` extension (instead of `.vim`) is required.
- Specify only the test file name, not the full path.


Debugging tests
---------------

- You can set `$GDB` to [run tests under gdbserver](https://github.com/neovim/neovim/pull/1527).
  And if `$VALGRIND` is set it will pass `--vgdb=yes` to valgrind instead of
  starting gdbserver directly.
- Hanging tests often happen due to unexpected `:h press-enter` prompts. The
  default screen width is 50 columns. Commands that try to print lines longer
  than 50 columns in the command-line, e.g. `:edit very...long...path`, will
  trigger the prompt. In this case, a shorter path or `:silent edit` should be
  used.
- If you can't figure out what is going on, try to visualize the screen. Put
  this at the beginning of your test:

    ```lua
    local Screen = require('test.functional.ui.screen')
    local screen = Screen.new()
    screen:attach()
    ```

  Afterwards, put `screen:snapshot_util()` at any position in your test. See the
  comment at the top of `test/functional/ui/screen.lua` for more.

Filtering Tests
---------------

### Filter by name

Another filter method is by setting a pattern of test name to `TEST_FILTER`.

``` lua
it('foo api',function()
  ...
end)
it('bar api',function()
  ...
end)
```

To run only test with filter name:

    TEST_TAG='foo.*api' make functionaltest

### Filter by file

To run a *specific* unit test:

    TEST_FILE=test/unit/foo.lua make unittest

To run a *specific* functional test:

    TEST_FILE=test/functional/foo.lua make functionaltest

To *repeat* a test:

    .deps/usr/bin/busted --lpath='build/?.lua' --filter 'foo' --repeat 1000 test/functional/ui/foo_spec.lua

### Filter by tag

Tests can be "tagged" by adding `#` before a token in the test description.

``` lua
it('#foo bar baz', function()
  ...
end)
it('#foo another test', function()
  ...
end)
```

To run only the tagged tests:

    TEST_TAG=foo make functionaltest

**NOTE:**

* `TEST_FILE` is not a pattern string like `TEST_TAG` or `TEST_FILTER`. The
  given value to `TEST_FILE` must be a path to an existing file.
* Both `TEST_TAG` and `TEST_FILTER` filter tests by the string descriptions
  found in `it()` and `describe()`.


Writing tests
=============

Guidelines
----------

- Consider [BDD](http://en.wikipedia.org/wiki/Behavior-driven_development)
  guidelines for organization and readability of tests. Describe what you're
  testing (and the environment if applicable) and create specs that assert its
  behavior.
- For testing static functions or functions that have side effects visible only
  in module-global variables, create accessors for the modified variables. For
  example, say you are testing a function in misc1.c that modifies a static
  variable, create a file `test/c-helpers/misc1.c` and add a function that
  retrieves the value after the function call. Files under `test/c-helpers` will
  only be compiled when building the test shared library.
- Luajit needs to know about type and constant declarations used in function
  prototypes. The
  [helpers.lua](https://github.com/neovim/neovim/blob/master/test/unit/helpers.lua)
  file automatically parses `types.h`, so types used in the tested functions
  must be moved to it to avoid having to rewrite the declarations in the test
  files (even though this is how it's currently done currently in the misc1/fs
  modules, but contributors are encouraged to refactor the declarations).
  - Macro constants must be rewritten as enums so they can be "visible" to the
    tests automatically.
- Busted supports various "output providers". The
  **[gtest](https://github.com/Olivine-Labs/busted/pull/394) output provider**
  shows verbose details that can be useful to diagnose hung tests. Either modify
  the Makefile or compile with `make
  CMAKE_EXTRA_FLAGS=-DBUSTED_OUTPUT_TYPE=gtest` to enable it.
- **Use busted's `pending()` feature** to skip tests
  ([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)).
  Do not silently skip the test with `if-else`. If a functional test depends on
  some external factor (e.g. the existence of `md5sum` on `$PATH`), *and* you
  can't mock or fake the dependency, then skip the test via `pending()` if the
  external factor is missing. This ensures that the *total* test-count
  (success + fail + error + pending) is the same in all environments.
    - *Note:* `pending()` is ignored if it is missing an argument _unless_ it is
      [contained in an `it()` block](https://github.com/neovim/neovim/blob/d21690a66e7eb5ebef18046c7a79ef898966d786/test/functional/ex_cmds/grep_spec.lua#L11).
      Provide empty function argument if the `pending()` call is outside of `it()`
      ([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)).
- Really long `source([=[...]=])` blocks may break syntax highlighting. Try
  `:syntax sync fromstart` to fix it.

Where tests go
--------------

Tests in `/test/unit` and `/test/functional` are divided into groups
by the semantic component they are testing.

- _Unit tests_
  ([test/unit](https://github.com/neovim/neovim/tree/master/test/unit)) should
  match 1-to-1 with the structure of `src/nvim/`, because they are testing
  functions directly. E.g. unit-tests for `src/nvim/undo.c` should live in
  `test/unit/undo_spec.lua`.
- _Functional tests_
  ([test/functional](https://github.com/neovim/neovim/tree/master/test/functional))
  are higher-level (plugins and user input) than unit tests; they are organized
  by concept. 
    - Try to find an existing `test/functional/*/*_spec.lua` group that makes
      sense, before creating a new one.


Lint
====

`make lint` (and `make lualint`) runs [luacheck](https://github.com/mpeterv/luacheck)
on the test code.

If a luacheck warning must be ignored, specify the warning code. Example:

    -- luacheck: ignore 621

http://luacheck.readthedocs.io/en/stable/warnings.html

Ignore the smallest applicable scope (e.g. inside a function, not at the top of
the file).


Configuration
=============

busted (luassert) by default does not print the full result of deeply-nested
tables. But sometimes it's useful while debugging a test.

    assert:set_parameter('TableFormatLevel', 1000000)

Test behaviour is affected by environment variables. Currently supported 
(Functional, Unit, Benchmarks) (when Defined; when set to _1_; when defined, 
treated as Integer; when defined, treated as String; when defined, treated as 
Number; !must be defined to function properly):

- `GDB` (F) (D): makes nvim instances to be run under `gdbserver`. It will be
  accessible on `localhost:7777`: use `gdb build/bin/nvim`, type `target remote
  :7777` inside.

- `GDBSERVER_PORT` (F) (I): overrides port used for `GDB`.

- `VALGRIND` (F) (D): makes nvim instances to be run under `valgrind`. Log
  files are named `valgrind-%p.log` in this case. Note that non-empty valgrind
  log may fail tests. Valgrind arguments may be seen in
  `/test/functional/helpers.lua`. May be used in conjunction with `GDB`.

- `VALGRIND_LOG` (F) (S): overrides valgrind log file name used for `VALGRIND`.

- `TEST_SKIP_FRAGILE` (F) (D): makes test suite skip some fragile tests.

- `NVIM_PROG`, `NVIM_PRG` (F) (S): override path to Neovim executable (default
  to `build/bin/nvim`).

- `CC` (U) (S): specifies which C compiler to use to preprocess files.
  Currently only compilers with gcc-compatible arguments are supported.

- `NVIM_TEST_MAIN_CDEFS` (U) (1): makes `ffi.cdef` run in main process. This
  raises a possibility of bugs due to conflicts in header definitions, despite
  the counters, but greatly speeds up unit tests by not requiring `ffi.cdef` to
  do parsing of big strings with C definitions.

- `NVIM_TEST_PRINT_I` (U) (1): makes `cimport` print preprocessed, but not yet
  filtered through `formatc` headers. Used to debug `formatc`. Printing is done
  with the line numbers.

- `NVIM_TEST_PRINT_CDEF` (U) (1): makes `cimport` print final lines which will
  be then passed to `ffi.cdef`. Used to debug errors `ffi.cdef` happens to
  throw sometimes.

- `NVIM_TEST_PRINT_SYSCALLS` (U) (1): makes it print to stderr when syscall
  wrappers are called and what they returned. Used to debug code which makes
  unit tests be executed in separate processes.

- `NVIM_TEST_RUN_FAILING_TESTS` (U) (1): makes `itp` run tests which are known
  to fail (marked by setting third argument to `true`).

- `LOG_DIR` (FU) (S!): specifies where to seek for valgrind and ASAN log files.

- `NVIM_TEST_CORE_*` (FU) (S): a set of environment variables which specify
  where to search for core files. Are supposed to be defined all at once.

- `NVIM_TEST_CORE_GLOB_DIRECTORY` (FU) (S): directory where core files are
  located. May be `.`. This directory is then recursively searched for core
  files. Note: this variable must be defined for any of the following to have
  any effect.

- `NVIM_TEST_CORE_GLOB_RE` (FU) (S): regular expression which must be matched
  by core files. E.g. `/core[^/]*$`. May be absent, in which case any file is
  considered to be matched.

- `NVIM_TEST_CORE_EXC_RE` (FU) (S): regular expression which excludes certain
  directories from searching for core files inside. E.g. use `^/%.deps$` to not
  search inside `/.deps`. If absent, nothing is excluded.

- `NVIM_TEST_CORE_DB_CMD` (FU) (S): command to get backtrace out of the
  debugger. E.g. `gdb -n -batch -ex "thread apply all bt full"
  "$_NVIM_TEST_APP" -c "$_NVIM_TEST_CORE"`. Defaults to the example command.
  This debug command may use environment variables `_NVIM_TEST_APP` (path to
  application which is being debugged: normally either nvim or luajit) and
  `_NVIM_TEST_CORE` (core file to get backtrace from).

- `NVIM_TEST_CORE_RANDOM_SKIP` (FU) (D): makes `check_cores` not check cores
  after approximately 90% of the tests. Should be used when finding cores is
  too hard for some reason. Normally (on OS X or when
  `NVIM_TEST_CORE_GLOB_DIRECTORY` is defined and this variable is not) cores
  are checked for after each test.

- `NVIM_TEST_RUN_TESTTEST` (U) (1): allows running
  `test/unit/testtest_spec.lua` used to check how testing infrastructure works.

- `NVIM_TEST_TRACE_LEVEL` (U) (N): specifies unit tests tracing level: `0`
  disables tracing (the fastest, but you get no data if tests crash and there
  was no core dump generated), `1` or empty/undefined leaves only C function
  cals and returns in the trace (faster then recording everything), `2` records
  all function calls, returns and lua source lines exuecuted.

- `NVIM_TEST_TRACE_ON_ERROR` (U) (1): makes unit tests yield trace on error in
  addition to regular error message.

- `NVIM_TEST_MAXTRACE` (U) (N): specifies maximum number of trace lines to
  keep. Default is 1024.