Use the grapheme break algorithm from utf8proc to support grapheme
clusters from recent unicode versions.
Handle variant selector VS16 turning some codepoints into double-width
emoji. This means we need to use ptr2cells rather than char2cells when
possible.
Problems:
- Illegal bytes after valid UTF-8 char cause utf_cp_*_off() to fail.
- When stream isn't NUL-terminated, utf_cp_*_off() may go over the end.
Solution: Don't go over end of the char of end of the string.
Problem:
Not all Lua code is checked by stylua. Automating code-style is an
important mechanism for reducing time spent on accidental
(non-essential) complexity.
Solution:
- Enable lintlua for `test/unit/` directory.
- TODO: only `test/functional/` remains unchecked.
previous: 45fe4d11ad
previous: 517f0cc634
Problem: buffer text with composing chars are converted from UTF-8
to an array of up to seven UTF-32 values and then converted back
to UTF-8 strings.
Solution: Convert buffer text directly to UTF-8 based schar_T values.
The limit of the text size is now in schar_T bytes, which is currently
31+1 but easily could be raised as it no longer multiplies the size
of the entire screen grid when not used, the full size is only required
for temporary scratch buffers.
Also does some general cleanup to win_line text handling, which was
unnecessarily complicated due to multibyte rendering being an "opt-in"
feature long ago. Nowadays, a char is just a char, regardless if it consists
of one ASCII byte or multiple bytes.
14:13:04,119 INFO - # ./test/unit/helpers.lua @ 760: mbyte utf_char2bytes for chars 0xa000 - 0xafff
14:13:06,307 WARN - E908: using an invalid value as a String
/usr/home/quickbuild/buildagent/workspace/root/neovim/pull-requests-automated/.deps/usr/bin/luajit:
./test/unit/helpers.lua:459: write() error: 32: Broken pipe
14:13:06,308 WARN - stack traceback:
14:13:06,308 WARN - [C]: in function 'throw'
14:13:06,308 WARN - ...quests-automated/.deps/usr/share/lua/5.1/busted/core.lua:149: in function 'error'
14:13:06,308 WARN - ...ts-automated/.deps/usr/share/lua/5.1/luassert/assert.lua:171: in function 'assert'
14:13:06,308 WARN - ./test/unit/helpers.lua:459: in function 'write'
14:13:06,308 WARN - ./test/unit/helpers.lua:626: in function 'hook'
14:13:06,308 WARN - ./test/unit/helpers.lua:574: in function <./test/unit/helpers.lua:557>
14:13:06,308 WARN - [C]: in function 'type'
14:13:06,308 WARN - ...d/.deps/usr/share/lua/5.1/busted/outputHandlers/base.lua:57: in function 'copyElement'
14:13:06,308 WARN - ...d/.deps/usr/share/lua/5.1/busted/outputHandlers/base.lua:66: in function 'format'
14:13:06,308 WARN - ...d/.deps/usr/share/lua/5.1/busted/outputHandlers/base.lua:172: in function 'fn'
14:13:06,308 WARN - ...-requests-automated/.deps/usr/share/lua/5.1/mediator.lua:103: in function 'publish'
14:13:06,308 WARN - ...quests-automated/.deps/usr/share/lua/5.1/busted/core.lua:201: in function 'safe'
14:13:06,308 WARN - ...quests-automated/.deps/usr/share/lua/5.1/busted/core.lua:312: in function 'execute'
14:13:06,308 WARN - ...sts-automated/.deps/usr/share/lua/5.1/busted/execute.lua:58: in function 'execute'
14:13:06,308 WARN - ...ests-automated/.deps/usr/share/lua/5.1/busted/runner.lua:197: in function <...ests-automated/.deps/usr/share/lua/5.1/busted/runner.lua:11>
14:13:06,308 WARN - ./.deps/usr/lib/luarocks/rocks/busted/2.0.0-1/bin/busted:3: in main chunk
14:13:06,308 WARN - [C]: at 0x004041a0
14:13:06,323 WARN - Terminated
14:13:06,325 INFO - Executing post-execute action...
14:13:06,526 INFO - Checking step execute condition...
14:13:06,526 INFO - Step execute condition satisfied, executing...
14:13:06,706 INFO - Executing pre-execute action...
14:13:06,706 INFO - Running step...
Use LuaJIT FFI to create char pointer.
Validate output with utf_ptr2char(), vim_iswordc() and vim_iswordp().
Use const for LuaJIT string-to-char conversion.
Used
sed -r -i -e '/ helpers =/ s/$/\nlocal itp = helpers.gen_itp(it)/; s/^(\s*)it\(/\1itp(/' test/unit/**/*_spec.lua
to alter all tests. Locally they all run fine now.
Reasoning:
1. General: state from one test should not affect other tests.
2. Local: travis build is failing with something which may be an output of
garbage collector. This should prevent state of the garbage collector from
interferring as well.
