Reasoning:
1. gtest is better then something like utfTerminal, yet it is way too verbose.
2. gtest cannot be configured to show colors always.
3. Actually I am going to add a CMake target which will allow running tests
(especially, functional tests) in parallel, but this is not going to work
well with any of the default output handlers. Build in this case must be more
or less silent, yet debuggable. New handler does not support this in this
commit though.
1. CI_TARGET now determines which run_${CI_TARGET}.sh script to use. Defaults to
`tests`.
2. Build no longer halts on the first failing suit: e.g. if functional tests
failed it will continue with unit tests, etc.
3. All ${MAKE_CMD} occurrences moved to `top_make` function, added `build_make`
as an alias to `make -C build` (`"${MAKE_CMD}" -C "${BUILD_DIR}"`) which is
too verbose.
`suite.sh` was copied from powerline (tests/common.sh file), assumes running
with POSIX shells (and actually uses dash in powerline). Then some convenience
functions were added (run_test and below).
- Do not exclude any directories from `find` search, remove dumps before tests
instead.
- Install `apport` on travis so that linux tests should produce core dumps
(based on information from travis-ci/travis-ci#3754, not sure whether it still
applies).
- Check cores in lua so that one has an idea which test is failing exactly. Do
this only 10% of time on linux because traversing the file system is slow.
Unit tests are still not touched, though it is what `app` argument in
`check_cores` is for.
TODO? consider using `find`, it may be faster. Consider retiring `os.execute`,
dealing with escaping is bad.
This fixes failures with TSAN builds like
FATAL: ThreadSanitizer can not mmap the shadow memory (something is mapped at 0x55deea465000 < 0x7cf000000000)
FATAL: Make sure to compile with -fPIE and to link with -pie.
Update to a recent, but not bleeding-edge, version of macOS and xcode.
At present, travis defaults to OS X 10.9.5 / Xcode 6.1.
QuickBuild runs macOS 10.10.
- Check if MIN_LOG_LEVEL value is a number 0-3, default to
INFO (1) or ignore it in Release mode
- When TRAVIS_CI_BUILD is ON the default is DEBUG (0)
- Add local.mk.example
Install Python 3.3 from the Deadsnakes PPA. As this doesn't have pip,
install it manually into ~/.local.
~/.local/bin is apparently in Travis's default PATH, meaning "pip"
doesn't refer to Python 2's pip anymore, but to the manually
installed Python 3 version. Updated the scripts to use version-
suffixed executable names (e.g. pip2.7).
Set CC=cc to use system's default compiler when installing Python
modules, as gcc on OS X had a problem with compiling one of the
dependencies of the Neovim Python module.
Instead of just caching the third-party build output, cache the full
build directory. Always run make to ensure that updated dependencies
are downloaded.
This fixes gcov/coveralls warnings like the following:
Segmentation fault (core dumped)
charset.c.gcno:version '501*', prefer '406*'
Out of memory allocating 33061786568 bytes after a total of 2522648 bytes
http://stackoverflow.com/a/14676272/249642
* Split build steps to utilize the Travis build lifecycle.
* Move shell code from `.travis.yml` into Bash files in `.ci/`,
one file for each step of the Travis build lifecycle.
* Use configuration variables in `.travis.yml` to change
build behavior (e.g. build 32-bit with `BUILD_32BIT=ON`).
* Keep all configuration in environment variables in
`.travis.yml`. In scripts, concatenate environment variables
according to configuration to change to different behavior.
* Add GCC 5 builds for Linux.
* Use Travis's caching feature [1] for third-party dependencies
and pip packages.
* Allow failures MSan, as the errors it reports have to be
fixed first.
Valgrind is still disabled, but can be enabled by setting
`env: VALGRIND=ON` for a job in `.travis.yml`.
[1] http://docs.travis-ci.com/user/caching
* Functional tests fail with SIGPIPE: disable them until we
figure out the exact problem.
* MSan reports some warnings: allow failures for the Travis
build to allow fixing them in individual follow-up PRs.
Sanity API checks made by the python-client in the api-python travis target were
converted to lua and will now live in this repository. This will simplify
performing breaking changes to the API as it won't be necessary to send parallel
PRs the python-client.
The 'lupa' python package provides a simple way to seamless integrate lua and
python code.
This commit replaces vroom by a python script that exposes the 'neovim' package
to a lua state, and invokes busted to run functional tests. This is a temporary
solution that will enable writing functional tests using lua/bused while a lua
client library is not available.
The reason for dropping vroom is flexibility: Lua/busted has a nice DSL-style
syntax while also providing the customization power of a full programming
language. Another reason is to use a single framework for unit/functional tests.
Two other changes were performed in this commit:
- Instead of "gcc-unittest/gcc-ia32", the travis builds for gcc are now
identified by "gcc/gcc-32". They will run unit/functional tests for both 64
and 32 bits.
- Old integration tests(in src/nvim/testdir) are now ran by the 'oldtest' target
Add section `before_install` in `.travis.yml` to create test group and
add current user to this group.
It is needed because by default user on Travis-CI belongs only to one
primary group derived from that user. So we have no alternative to
change group of the file.
To simplify modification/inclusion of continuous integration targets, this
removes travis.sh which contains a big if statement in favor of multiple scripts
under the new '.ci' directory.
Run only on push to branch coverity-scan. We can use a cron script to do
this 4 times a week (that's our allowance).
NOTE: possible future improvements are:
1. Fold the build matrix item into another short one so we don't overburden
travis. It's a little less clear but it should be nicer on the
infrastructure.
2. Change the security token, one can do that from the coverity admin page.
3. Don't do the naive `make depend`, but use the prebuilt libraries.
- Add a 'expect' utility script that can run simple API tests using clients
developed for any platform.
- Extend travis build matrix to run API tests using the python client and
valgrind.
This script can be used to write API tests without having to manage nvim's
lifetime:
- It starts a single nvim instance listening on a known socket
- Invokes the test runner, which should connect to NEOVIM_LISTEN_ADDRESS
- The nvim instance started by the script provides a `BeforeEachTest` function,
which should be called before each test to reset nvim to a clean state.
- It takes care of shutting down nvim once the tests are finished.
As explained
[here](https://github.com/neovim/neovim/pull/737#issuecomment-43941520), it's
not possible to fully reset nvim to it's initial state, but the `BeforeEachTest`
function should be enough for most test cases. Tests requiring a fully clean
nvim instance should take care of starting/stopping nvim.
- Build targeting 32-bit with travis
- Code in `before_install`/`after_success` was moved to travis.sh since it
provides greater flexibility for detecting the build matrix environment. This
improves the build speed since we now install only what's necessary.
- Now clint has a dedicated travis worker
- Valgrind configuration removed
- Fix errors reported by the undefined behavior sanitizer
- Travis will now run two build steps:
- A normal build of a shared library for unit testing(in parallel with gcc)
- A clang build with some sanitizers enabled for integration testing.
After these changes travis will run much faster, while providing valgrind-like
error detection.
