The removes the previous restriction that nvim_buf_set_extmark()
could not be used to highlight arbitrary multi-line regions
The problem can be summarized as follows: let's assume an extmark with a
hl_group is placed covering the region (5,0) to (50,0) Now, consider
what happens if nvim needs to redraw a window covering the lines 20-30.
It needs to be able to ask the marktree what extmarks cover this region,
even if they don't begin or end here.
Therefore the marktree needs to be augmented with the information covers
a point, not just what marks begin or end there. To do this, we augment
each node with a field "intersect" which is a set the ids of the
marks which overlap this node, but only if it is not part of the set of
any parent. This ensures the number of nodes that need to be explicitly
marked grows only logarithmically with the total number of explicitly
nodes (and thus the number of of overlapping marks).
Thus we can quickly iterate all marks which overlaps any query position
by looking up what leaf node contains that position. Then we only need
to consider all "start" marks within that leaf node, and the "intersect"
set of that node and all its parents.
Now, and the major source of complexity is that the tree restructuring
operations (to ensure that each node has T-1 <= size <= 2*T-1) also need
to update these sets. If a full inner node is split in two, one of the
new parents might start to completely overlap some ranges and its ids
will need to be moved from its children's sets to its own set.
Similarly, if two undersized nodes gets joined into one, it might no
longer completely overlap some ranges, and now the children which do
needs to have the have the ids in its set instead. And then there are
the pivots! Yes the pivot operations when a child gets moved from one
parent to another.
This involves two redesigns of the map.c implementations:
1. Change of macro style and code organization
The old khash.h and map.c implementation used huge #define blocks with a
lot of backslash line continuations.
This instead uses the "implementation file" .c.h pattern. Such a file is
meant to be included multiple times, with different macros set prior to
inclusion as parameters. we already use this pattern e.g. for
eval/typval_encode.c.h to implement different typval encoders reusing a
similar structure.
We can structure this code into two parts. one that only depends on key
type and is enough to implement sets, and one which depends on both key
and value to implement maps (as a wrapper around sets, with an added
value[] array)
2. Separate the main hash buckets from the key / value arrays
Change the hack buckets to only contain an index into separate key /
value arrays
This is a common pattern in modern, state of the art hashmap
implementations. Even though this leads to one more allocated array, it
is this often is a net reduction of memory consumption. Consider
key+value consuming at least 12 bytes per pair. On average, we will have
twice as many buckets per item.
Thus old implementation:
2*12 = 24 bytes per item
New implementation
1*12 + 2*4 = 20 bytes per item
And the difference gets bigger with larger items.
One might think we have pulled a fast one here, as wouldn't the average size of
the new key/value arrays be 1.5 slots per items due to amortized grows?
But remember, these arrays are fully dense, and thus the accessed memory,
measured in _cache lines_, the unit which actually matters, will be the
fully used memory but just rounded up to the nearest cache line
boundary.
This has some other interesting properties, such as an insert-only
set/map will be fully ordered by insert only. Preserving this ordering
in face of deletions is more tricky tho. As we currently don't use
ordered maps, the "delete" operation maintains compactness of the item
arrays in the simplest way by breaking the ordering. It would be
possible to implement an order-preserving delete although at some cost,
like allowing the items array to become non-dense until the next rehash.
Finally, in face of these two major changes, all code used in khash.h
has been integrated into map.c and friends. Given the heavy edits it
makes no sense to "layer" the code into a vendored and a wrapper part.
Rather, the layered cake follows the specialization depth: code shared
for all maps, code specialized to a key type (and its equivalence
relation), and finally code specialized to value+key type.
Here's the headline: when run in sync mode (last argument cb=NULL),
these functions don't actually use the uv_loop_t.
An earlier version of this patch instead replaced fs_loop with
using main_loop.uv on the main thread and luv_loop() on luv worker
threads. However this made the code more complicated for no reason.
Also arbitrarily, half of these functions would attempt to handle
UV_ENOMEM by try_to_free_memory(). This would mostly happen
on windows because it needs to allocate a converted WCHAR buffer.
This should be a quite rare situation. Your system is pretty
much hosed already if you cannot allocate like 50 WCHAR:s.
Therefore, take the liberty of simply removing this fallback.
In addition, we tried to "recover" from ENOMEM in read()/readv()
this way which doesn't make any sense. The read buffer(s) are already
allocated at this point.
This would also be an issue when using these functions on a worker
thread, as try_to_free_memory() is not thread-safe. Currently
os_file_is_readable() and os_is_dir() is used by worker threads
(as part of nvim__get_runtime(), to implement require from 'rtp' in
threads).
In the end, these changes makes _all_ os/fs.c functions thread-safe,
and we thus don't need to document and maintain a thread-safe subset.
Previously, there were three low-level delay entry points
- os_delay(ms, ignoreinput=true): sleep for ms, only break on got_int
- os_delay(ms, ignoreinput=false): sleep for ms, break on any key input
os_microdelay(us, false): equivalent, but in μs (not directly called)
- os_microdelay(us, true): sleep for μs, never break.
The implementation of the latter two both used uv_cond_timedwait()
This could have been for two reasons:
1. allow another thread to "interrupt" the wait
2. uv_cond_timedwait() has higher resolution than uv_sleep()
However we (1) never used the first, even when TUI was a thread, and
(2) nowhere in the codebase are we using μs resolution, it is always a ms
multiplied with 1000.
In addition, os_delay(ms, false) would completely block the thread for
100ms intervals and in between check for input. This is not how event handling
is done alound here.
Therefore:
Replace the implementation of os_delay(ms, false) to use
LOOP_PROCESS_EVENTS_UNTIL which does a proper epoll wait with a timeout,
instead of the 100ms timer panic.
Replace os_microdelay(us, false) with a direct wrapper of uv_sleep.
System headers on macOS arm64 contain 128-bit numeric types. These types
are built into clang and GCC as extensions. Unfortunately, they break
the LuaJIT C importer. Define dummy typedefs for the missing numeric
types to satisfy the ffi C importer.
unittests relied on the exact setup of coredumps on CI to detect
process crashing, and otherwise completely discarded errors.
Dectect child process failure reliably using process status, so that
unittests actually work locally as well.
Problem:
Dirs "config", "packaging", and "third-party" are all closely related
but this is not obvious from the layout. This adds friction for new
contributors.
Solution:
- rename config/ to cmake.config/
- rename test/config/ to test/cmakeconfig/ because it is used in Lua
tests: require('test.cmakeconfig.paths').
- rename packaging/ to cmake.packaging/
- rename third-party/ to cmake.deps/ (parallel with .deps/)
fs_init() must be called before early_init() in init/helpers.lua
If I run 'make unittest' on my Mac (macOS 10.14/Mojave or 12/Big Sur, intel
CPU), every test produce a core dump.
Call sequence in the core is:
early_init() main.c:197
set_init_1() option.c:508
runtimepath_default() runtime.c:1205
get_lib_dir() runtime.c:1175
os_isdir() fs.c:137
os_getperm() fs.c:777
os_stat() fs.c:761
fs_loop_lock() fs.c:72
uv_mutex_lock(&fs_loop_mutex) thread.c:352
abort()
.deps/build/src/libuv/src/unix/thread.c:
void uv_mutex_lock(uv_mutex_t* mutex) {
if (pthread_mutex_lock(mutex))
abort(); // line 352
}
So pthread_mutex_lock(&fs_loop_mutex) failed. The reason seems to be simple.
fs_init() was not called and fs_loop_mutex has not been initialized. fs_init()
was moved out from early_init() in main.c by
b87867e69e, but unit/helpers.lua was not updated
accordingly.
This is where "pure functions" can live, which can be shared by Nvim and
test logic which may not have a running Nvim instance available.
If in the future we use Nvim itself as the Lua engine for tests, then
these functions could be moved directly onto the `vim` Lua module.
closes#6580
Automatically include all "global helper" util functions in the
unit.helpers and functional.helpers and modules. So tests don't need to
expicitly do:
local global_helpers = require('test.helpers')
- Minimum required libuv is now v1.12
- Because `uv_os_getenv` requires allocating, we must manage a map
(`envmap` in `env.c`) to maintain the old behavior of `os_getenv` .
- free() map-items after removal. khash.h does not make copies of
anything, so even its keys must be memory-managed by the caller.
closes#8398closes#9267
Reason: test may contain cleanup at the endwhich is needed for GC to work
properly, but is not done if test fails. With collectgarbage() in former
position it would crash when collecting garbage.
When building with certain GCC versions, a _Float128 type is present
when setting up the ffi for unit tests.
./test/unit/helpers.lua:256: declaration specifier expected near '_Float128' at line 396
/usr/bin/luajit: /usr/share/lua/5.1/busted/runner.lua:99: attempt to concatenate local 'message' (a table value)
stack traceback:
/usr/share/lua/5.1/busted/runner.lua:99: in function 'fn'
/usr/share/lua/5.1/mediator.lua:103: in function 'publish'
/usr/share/lua/5.1/busted/modules/helper_loader.lua:21: in function 'helperLoader'
/usr/share/lua/5.1/busted/runner.lua:147: in function </usr/share/lua/5.1/busted/runner.lua:11>
/usr/bin/busted:3: in main chunk
[C]: at 0x004044a0
CMake Error at /<<PKGBUILDDIR>>/cmake/RunTests.cmake:53 (message):
Running unit tests failed with error: 1.
Since this is being pulled in by a dependency, not directly used by
nvim, just ignore the type.
Closes#7423
Currently supported nodes:
- Register as it is one of the simplest value nodes (even numbers are
not that simple with that dot handling).
- Plus, both unary and binary.
- Parenthesis, both nesting and calling.
Note regarding unit tests: it stores data for AST in highlighting in
strings in place of tables because luassert fails to do a good job at
representing big tables. Squashing a bunch of data into a single string
simply yields more readable result.
Some benchmarks:
MAIN_CDEFS + NO_TRACE: 3.81s user 1.65s system 33% cpu 16.140 total
MAIN_CDEFS: 73.61s user 10.98s system 154% cpu 54.690 total
NO_TRACE: 18.49s user 4.30s system 73% cpu 30.804 total
(default): 77.11s user 14.74s system 126% cpu 1:12.79 total
