Problem: Injecting languages for file redirects (e.g., in bash) is not
possible.
Solution: Add `@injection.filename` capture that is piped through
`vim.filetype.match({ filename = node_text })`; the resulting filetype
(if not `nil`) is then resolved as a language (either directly or
through the list maintained via `vim.treesitter.language.register()`).
Note: `@injection.filename` is a non-standard capture introduced by
Helix; having two editors implement it makes it likely to be upstreamed.
Problem:
`TSNode:_rawquery()` is complicated, has known issues and the Lua and
C code is awkwardly coupled (see logic with `active`).
Solution:
- Add `TSQueryCursor` and `TSQueryMatch` bindings.
- Replace `TSNode:_rawquery()` with `TSQueryCursor:next_capture()` and `TSQueryCursor:next_match()`
- Do more stuff in Lua
- API for `Query:iter_captures()` and `Query:iter_matches()` remains the same.
- `treesitter.c` no longer contains any logic related to predicates.
- Add `match_limit` option to `iter_matches()`. Default is still 256.
Problem: Not all standard treesitter groups are documented.
Solution: Document them all (without relying on fallback); add default
link for new `*.builtin` groups to `Special` and `@keyword.type` to
`Structure`. Remove `@markup.environment.*` which only made sense for
LaTeX.
- Added `@inlinedoc` so single use Lua types can be inlined into the
functions docs. E.g.
```lua
--- @class myopts
--- @inlinedoc
---
--- Documentation for some field
--- @field somefield integer
--- @param opts myOpts
function foo(opts)
end
```
Will be rendered as
```
foo(opts)
Parameters:
- {opts} (table) Object with the fields:
- somefield (integer) Documentation
for some field
```
- Marked many classes with with `@nodoc` or `(private)`.
We can eventually introduce these when we want to.
Problem:
The documentation flow (`gen_vimdoc.py`) has several issues:
- it's not very versatile
- depends on doxygen
- doesn't work well with Lua code as it requires an awkward filter script to convert it into pseudo-C.
- The intermediate XML files and filters makes it too much like a rube goldberg machine.
Solution:
Re-implement the flow using Lua, LPEG and treesitter.
- `gen_vimdoc.py` is now replaced with `gen_vimdoc.lua` and replicates a portion of the logic.
- `lua2dox.lua` is gone!
- No more XML files.
- Doxygen is now longer used and instead we now use:
- LPEG for comment parsing (see `scripts/luacats_grammar.lua` and `scripts/cdoc_grammar.lua`).
- LPEG for C parsing (see `scripts/cdoc_parser.lua`)
- Lua patterns for Lua parsing (see `scripts/luacats_parser.lua`).
- Treesitter for Markdown parsing (see `scripts/text_utils.lua`).
- The generated `runtime/doc/*.mpack` files have been removed.
- `scripts/gen_eval_files.lua` now instead uses `scripts/cdoc_parser.lua` directly.
- Text wrapping is implemented in `scripts/text_utils.lua` and appears to produce more consistent results (the main contributer to the diff of this change).
Query patterns can contain quantifiers (e.g. (foo)+ @bar), so a single
capture can map to multiple nodes. The iter_matches API can not handle
this situation because the match table incorrectly maps capture indices
to a single node instead of to an array of nodes.
The match table should be updated to map capture indices to an array of
nodes. However, this is a massively breaking change, so must be done
with a proper deprecation period.
`iter_matches`, `add_predicate` and `add_directive` must opt-in to the
correct behavior for backward compatibility. This is done with a new
"all" option. This option will become the default and removed after the
0.10 release.
Co-authored-by: Christian Clason <c.clason@uni-graz.at>
Co-authored-by: MDeiml <matthias@deiml.net>
Co-authored-by: Gregory Anders <greg@gpanders.com>
Document that the `start` and `stop` parameters in
`Query:iter_captures()` and `Query:iter_matches()` are optional.
The tree-sitter lib has been bumped up to 0.20.9, so we also no longer
need "Requires treesitter >= 0.20.9".
- `TSQuery`: userdata object for parsed query.
- `vim.treesitter.Query`: renamed from `Query`.
- Add a new field `lang`.
- `TSQueryInfo`:
- Move to `vim/treesitter/_meta.lua`, because C code owns it.
- Correct typing for `patterns`, should be a map from `integer`
(pattern_id) to `(integer|string)[][]` (list of predicates or
directives).
- `vim.treesitter.QueryInfo` is added.
- This currently has the same structure as `TSQueryInfo` (exported
from C code).
- Document the fields (see `TSQuery:inspect`).
- Add typing for `vim._ts_parse_query()`.
Problem: Currently default color scheme defines most of treesitter
highlight groups. This might be an issue for users defining their own
color scheme as it breaks the "fallback property" for some of groups.
Solution: Define less default treesitter groups; just enough for default
color scheme to be useful. That is:
- All first level groups (`@character`, `@string`, etc.).
- All `@xxx.builtin` groups as a most common subgroup.
- Some special cases (links/URLs, `@diff.xxx`, etc.).
Docs for treesitter would benefit from including more real-world and
practical examples of queries and usages, rather than hypothetical ones
(e.g. names such as "foo", "bar"). Improved examples should be more
user-friendly and clear to understand.
In addition, align the capture names in some examples with the actual
ones being used in the built-in query files or in the nvim-treesitter
plugin, e.g.:
- `@parameter` -> `@variable.parameter`
- `@comment.doc.java` -> `@comment.documentation.java`
- etc.
Problem: Sharing queries with upstream and Helix is difficult due to
different capture names.
Solution: Define and document a new set of standard captures that
matches tree-sitter "standard captures" (where defined) and is closer to
Helix' Atom-style nested groups.
This is a breaking change for colorschemes that defined highlights based
on the old captures. On the other hand, the default colorscheme now
defines links for all standard captures (not just those used in bundled
queries), improving the out-of-the-box experience.
Problem: Some lines in the generated vim doc are overflowing, not
correctly wrapped at 78 characters. This happens when docs body contains
several consecutive 'inline' elements generated by doxygen.
Solution: Take into account the current column offset of the last line,
and prepend some padding before doc_wrap().
Improve error messages for `:InspectTree`, when no parsers are available
for the current buffer and filetype. We can show more informative and
helpful error message for users (e.g., which lang was searched for):
```
... No parser available for the given buffer:
+... no parser for 'custom_ft' language, see :help treesitter-parsers
```
Also improve the relevant docs for *treesitter-parsers*.
Problem: Not all default highlight groups show their actual colors.
Solution: Refactor `vimhelp.lua` and apply it to all relevant lists (UI
groups, syntax groups, treesitter groups, LSP groups, diagnostic groups).
PROBLEM: `vim.treesitter.get_node()` does not recognize the `lang` in
the option table. This option was used in somewhere else, for instance,
`vim.treesitter.dev` (for `inspect_tree`) but was never implemented.
SOLUTION: Make `get_node()` correctly use `opts.lang` when getting a
treesitter parser.
Problem:
With incremental injection parsing, injected languages' parsers parse
only the relevant regions and stores the result in _trees with the index
of the corresponding region. Therefore, there can be holes in _trees.
Solution:
* Use generic table functions where appropriate.
* Fix type annotations and docs.
Problem
---
If a highlighter query returns a significant number of predicate
non-matches, the highlighter will scan well past the end of the window.
Solution
---
In the iterator returned from `iter_captures`, accept an optional
parameter `end_line`. If no parameter provided, the behavior is
unchanged, hence this is a non-invasive tweak.
Fixes: #25113nvim-treesitter/nvim-treesitter#5057
The name for_each_child is misleading and caused bugs.
After #25111, #25115, there are no more usages of `for_each_child` in Nvim.
In the future if we want to restore this functionality we can consider a
generalized vim.traverse(node, key, visitor) function.
Problem:
Treesitter highlighting is slow for large files with lots of injections.
Solution:
Only parse injections we are going to render during a redraw cycle.
---
- `LanguageTree:parse()` will no longer parse injections by default and
now requires an explicit range argument to be passed.
- `TSHighlighter` now parses injections incrementally during on_win
callbacks for the line range being rendered.
- Plugins which require certain injections to be parsed must run
`parser:parse({ start_row, end_row })` before using the tree.
* feat(treesitter): add injection language fallback
Problem: injection languages are often specified via aliases (e.g.,
filetype or in upper case), requiring custom directives.
Solution: include lookup logic (try as parser name, then filetype, then
lowercase) in LanguageTree itself and remove `#inject-language`
directive.
Co-authored-by: Lewis Russell <me@lewisr.dev>
Problem:
Lua functions that return multiple results are declared by using
multiple `@return` docstring directives. But the generated docs don't
make it obvious what this represents.
Solution:
- Generate a "Return (multiple)" heading for multiple-value functions.
- Fix `@note` directives randomly placed after `@return`.
Problem:
"playground" is new jargon that overlaps with existing concepts:
"dev" (`:help dev`) and "view" (also "scratch" `:help scratch-buffer`) .
Solution:
We should consistently use "dev" as the namespace for where "developer
tools" live. For purposes of a "throwaway sandbox object", we can use
the name "view".
- Rename `TSPlayground` => `TSView`
- Rename `playground.lua` => `dev.lua`
