Using -1 as the initial value can cause the pattern offset to become
negative, which in turn results in a negative subpriority, which fails
validation in nvim_buf_set_extmark.
Tree-sitter queries can add URLs to a capture using the `#set!`
directive, e.g.
(inline_link
(link_text) @text.reference
(link_destination) @text.uri
(#set! @text.reference "url" @text.uri))
The pattern above is included by default in the `markdown_inline`
highlight query so that users with supporting terminals will see
hyperlinks. For now, this creates a hyperlink for *all* Markdown URLs of
the pattern [link text](link url), even if `link url` does not contain
a valid protocol (e.g. if `link url` is a path to a file). We may wish to
change this in the future to only linkify when the URL has a valid
protocol scheme, but for now we delegate handling this to the terminal
emulator.
In order to support directives which reference other nodes, the
highlighter must be updated to use `iter_matches` rather than
`iter_captures`. The former provides the `match` table which maps
capture IDs to nodes. However, this has its own challenges:
- `iter_matches` does not guarantee the order in which patterns are
iterated matches the order in the query file. So we must enforce
ordering manually using "subpriorities" (#27131). The pattern index of
each match dictates the extmark's subpriority.
- When injections are used, the highlighter contains multiple trees. The
pattern indices of each tree must be offset relative to the maximum
pattern index from all previous trees to ensure that extmarks appear
in the correct order.
- The `iter_captures` implementation currently has a bug where the
"match" table is only returned for the first capture within a pattern
(see #27274). This bug means that `#set!` directives in a query
apply only to the first capture within a pattern. Unfortunately, many
queries in the wild have come to depend on this behavior.
`iter_matches` does not share this flaw, so switching to
`iter_matches` exposed bugs in existing highlight queries. These
queries have been updated in this repo, but may still need to be
updated by users. The `#set!` directive applies to the _entire_ query
pattern when used without a capture argument. To make `#set!`
apply only to a single capture, the capture must be given as an
argument.
- Added `@inlinedoc` so single use Lua types can be inlined into the
functions docs. E.g.
```lua
--- @class myopts
--- @inlinedoc
---
--- Documentation for some field
--- @field somefield integer
--- @param opts myOpts
function foo(opts)
end
```
Will be rendered as
```
foo(opts)
Parameters:
- {opts} (table) Object with the fields:
- somefield (integer) Documentation
for some field
```
- Marked many classes with with `@nodoc` or `(private)`.
We can eventually introduce these when we want to.
Problem:
The documentation flow (`gen_vimdoc.py`) has several issues:
- it's not very versatile
- depends on doxygen
- doesn't work well with Lua code as it requires an awkward filter script to convert it into pseudo-C.
- The intermediate XML files and filters makes it too much like a rube goldberg machine.
Solution:
Re-implement the flow using Lua, LPEG and treesitter.
- `gen_vimdoc.py` is now replaced with `gen_vimdoc.lua` and replicates a portion of the logic.
- `lua2dox.lua` is gone!
- No more XML files.
- Doxygen is now longer used and instead we now use:
- LPEG for comment parsing (see `scripts/luacats_grammar.lua` and `scripts/cdoc_grammar.lua`).
- LPEG for C parsing (see `scripts/cdoc_parser.lua`)
- Lua patterns for Lua parsing (see `scripts/luacats_parser.lua`).
- Treesitter for Markdown parsing (see `scripts/text_utils.lua`).
- The generated `runtime/doc/*.mpack` files have been removed.
- `scripts/gen_eval_files.lua` now instead uses `scripts/cdoc_parser.lua` directly.
- Text wrapping is implemented in `scripts/text_utils.lua` and appears to produce more consistent results (the main contributer to the diff of this change).
- `TSQuery`: userdata object for parsed query.
- `vim.treesitter.Query`: renamed from `Query`.
- Add a new field `lang`.
- `TSQueryInfo`:
- Move to `vim/treesitter/_meta.lua`, because C code owns it.
- Correct typing for `patterns`, should be a map from `integer`
(pattern_id) to `(integer|string)[][]` (list of predicates or
directives).
- `vim.treesitter.QueryInfo` is added.
- This currently has the same structure as `TSQueryInfo` (exported
from C code).
- Document the fields (see `TSQuery:inspect`).
- Add typing for `vim._ts_parse_query()`.
- Remove some unused fields
- Prefix classes with `vim.`
- Move around some functions so the query stuff is at the top.
- Improve type hints
- Rework how hl_cache is implemented
Problem:
Treesitter highlighter's on_line was iterating all the parsed trees,
which can be quite a lot when injection is used. This may slow down
scrolling and cursor movement in big files with many comment injections
(e.g., lsp/_meta/protocol.lua).
Solution:
In on_win, collect trees inside the visible range, and use them in
on_line.
NOTE:
This optimization depends on the correctness of on_win's botline_guess
parameter (i.e., it's always greater than or equal to the line numbers
passed to on_line). The documentation does not guarantee this, but I
have never noticed a problem so far.
Problem
---
If a highlighter query returns a significant number of predicate
non-matches, the highlighter will scan well past the end of the window.
Solution
---
In the iterator returned from `iter_captures`, accept an optional
parameter `end_line`. If no parameter provided, the behavior is
unchanged, hence this is a non-invasive tweak.
Fixes: #25113nvim-treesitter/nvim-treesitter#5057
Problem:
* The guessed botline might be smaller than the actual botline e.g. when
there are folds and the user is typing in insert mode. This may result
in incorrect treesitter highlights for injections.
* botline can be larger than the last line number of the buffer, which
results in errors when placing extmarks.
Solution:
* Take a more conservative approximation. I am not sure if it is
sufficient to guarantee correctness, but it seems to be good enough
for the case mentioned above.
* Clamp it to the last line number.
Co-authored-by: Lewis Russell <me@lewisr.dev>
Problem:
Treesitter highlighting is slow for large files with lots of injections.
Solution:
Only parse injections we are going to render during a redraw cycle.
---
- `LanguageTree:parse()` will no longer parse injections by default and
now requires an explicit range argument to be passed.
- `TSHighlighter` now parses injections incrementally during on_win
callbacks for the line range being rendered.
- Plugins which require certain injections to be parsed must run
`parser:parse({ start_row, end_row })` before using the tree.
* docs(lua): teach lua2dox how to table
* docs(lua): teach gen_vimdoc.py about local functions
No more need to mark local functions with @private
* docs(lua): mention @nodoc and @meta in dev-lua-doc
* fixup!
Co-authored-by: Justin M. Keyes <justinkz@gmail.com>
---------
Co-authored-by: Justin M. Keyes <justinkz@gmail.com>
When injections are added or removed make sure to:
- invoke 'changedtree' callbacks for when new trees are added.
- invoke 'changedtree' callbacks for when trees are invalidated
- redraw regions when languagetree children are removed
Problem:
Codebase inconsistently binds vim.api onto a or api.
Solution:
Use api everywhere. a as an identifier is too short to have at the
module level.
Problem:
Help tags like vim.treesitter.language.add() are confusing because
`vim.treesitter.language` is (thankfully) not a user-facing module.
Solution:
Ignore the "fstem" when generating "treesitter" tags.
- Added 'spell' option to extmarks:
Extmarks with this set will have the region spellchecked.
- Added 'noplainbuffer' option to 'spelloptions':
This is used to tell Neovim not to spellcheck the buffer. The old
behaviour was to spell check the whole buffer unless :syntax was set.
- Added spelling support to the treesitter highlighter:
@spell captures in highlights.scm are used to define regions which
should be spell checked.
- Added support for navigating spell errors for extmarks:
Works for both ephemeral and static extmarks
- Added '_on_spell_nav' callback for decoration providers:
Since ephemeral callbacks are only drawn for the visible screen,
providers must implement this callback to instruct Neovim which
regions in the buffer need can be spell checked.
The callback takes a start position and an end position.
Note: this callback is subject to change hence the _ prefix.
- Added spell captures for built-in support languages
Co-authored-by: Lewis Russell <lewis6991@gmail.com>
Co-authored-by: Björn Linse <bjorn.linse@gmail.com>
This removes the support for defining links via
vim.treesitter.highlighter.hl_map (never documented, but plugins did
anyway), or the uppercase-only `@FooGroup.Bar` to `FooGroup` rule.
The fallback is now strictly `@foo.bar.lang` to `@foo.bar` to `@foo`,
and casing is irrelevant (as it already was outside of treesitter)
For compatibility, define default links to builting syntax groups
as defined by pre-existing color schemes
The official developer documentation in in :h dev-lua-doc specifies to
use "--@" for special/magic tokens. However, this format is not
consistent with EmmyLua notation (used by some Lua language servers) nor
with the C version of the magic docstring tokens which use three comment
characters.
Further, the code base is currently split between usage of "--@",
"---@", and "--- @". In an effort to remain consistent, change all Lua
magic tokens to use "---@" and update the developer documentation
accordingly.
This changes the behavior of the hl_cache to the old one.
- when the capture exists as a hlgroup -> use it
- when hl_map contains a mapping -> use it
- else do nothing (before: map capture to non-existing capture)
Before also captures `@foo.bar` would intend to use the hlgroup `foo.bar`
which results in a confusing error since hlgroups can't contain dots.
