runtime(doc): Add Makefile for the Vim documentation on Windows (vim/vim#13467)
* Makefile for the Vim documentation on Windows
* Corrected comments
b23c1fc596
Co-authored-by: Restorer <69863286+RestorerZ@users.noreply.github.com>
Problem:
While LuaCATS's generics system are still considered WIP by luals, they
currently support type captured generics.
See "Capture with Backtick" example:
https://luals.github.io/wiki/annotations/#generic
Solution:
Add support for it in the LuaCATS grammar
- Sort sections according to custom preference instead of
alphabetically. It is ordered according to "most relevant" to "least
relevant" to users.
- Sort commits alphabetically
- Don't uppercase the first letter of the commit message
Specifically, functions that are run in the context of the test runner
are put in module `test/testutil.lua` while the functions that are run
in the context of the test session are put in
`test/functional/testnvim.lua`.
Closes https://github.com/neovim/neovim/issues/27004.
Problem:
Besides being redundant with vim.iter():flatten(), `tbl_flatten` has
these problems:
- Has `tbl_` prefix but only accepts lists.
- Discards some results! Compare the following:
- iter.flatten():
```
vim.iter({1, { { a = 2 } }, { 3 } }):flatten():totable()
```
- tbl_flatten:
```
vim.tbl_flatten({1, { { a = 2 } }, { 3 } })
```
Solution:
Deprecate tbl_flatten.
Note:
iter:flatten() currently fails ("flatten() requires a list-like table")
on this code from gen_lsp.lua:
local anonym = vim.iter({ -- remove nil
anonymous_num > 1 and '' or nil,
'---@class ' .. anonymous_classname,
}):flatten():totable()
Should we enhance :flatten() to work for arrays?
- Added `@inlinedoc` so single use Lua types can be inlined into the
functions docs. E.g.
```lua
--- @class myopts
--- @inlinedoc
---
--- Documentation for some field
--- @field somefield integer
--- @param opts myOpts
function foo(opts)
end
```
Will be rendered as
```
foo(opts)
Parameters:
- {opts} (table) Object with the fields:
- somefield (integer) Documentation
for some field
```
- Marked many classes with with `@nodoc` or `(private)`.
We can eventually introduce these when we want to.
Then we can just load metadata in C as a single msgpack blob. Which also
can be used directly as binarly data, instead of first unpacking all the
functions and ui_events metadata to immediately pack it again, which was
a bit of a silly walk (and one extra usecase of `msgpack_rpc_from_object`
which will get yak shaved in the next PR)
runtime(sh): Update ftplugin, fixvim/vim#14101 (vim/vim#14102)
Add the 'b' flag to 'comments', so that the shebang line is not detected as comment.
Fixesvim/vim#14101.
e84d2d4432
Co-authored-by: dkearns <dougkearns@gmail.com>
Problem:
The documentation flow (`gen_vimdoc.py`) has several issues:
- it's not very versatile
- depends on doxygen
- doesn't work well with Lua code as it requires an awkward filter script to convert it into pseudo-C.
- The intermediate XML files and filters makes it too much like a rube goldberg machine.
Solution:
Re-implement the flow using Lua, LPEG and treesitter.
- `gen_vimdoc.py` is now replaced with `gen_vimdoc.lua` and replicates a portion of the logic.
- `lua2dox.lua` is gone!
- No more XML files.
- Doxygen is now longer used and instead we now use:
- LPEG for comment parsing (see `scripts/luacats_grammar.lua` and `scripts/cdoc_grammar.lua`).
- LPEG for C parsing (see `scripts/cdoc_parser.lua`)
- Lua patterns for Lua parsing (see `scripts/luacats_parser.lua`).
- Treesitter for Markdown parsing (see `scripts/text_utils.lua`).
- The generated `runtime/doc/*.mpack` files have been removed.
- `scripts/gen_eval_files.lua` now instead uses `scripts/cdoc_parser.lua` directly.
- Text wrapping is implemented in `scripts/text_utils.lua` and appears to produce more consistent results (the main contributer to the diff of this change).
Problem:
When a function has multiple signatures, putting its tag at the last one
may make one think that's its only signature.
Solution:
When a function has multiple signatures, put its tag at the first one.
Implement api_keydict_to_dict as the complement to api_dict_to_keydict
Fix a conversion error when nvim_get_win_config gets called from lua,
where Float values "x" and "y" didn't get converted to lua numbers.
- Consistently use the variable CMAKE_BUILD_TYPE to select build type.
- Remove broken `doc_html` target.
- Remove swap files created by oldtest when cleaning.
- Only rerun `lintdoc` if any documentation files has changed.
- Add type annotations, fix most of the type warnings.
- Fix a minor bug on `spell_ignore_files`: nil error when an invalid
spelling is found but the file is not ignored.
Summary: Separate the lint job (`make lintdoc`) to validate runtime/doc,
it is no longer as a part of functionaltest (help_spec).
Build (cmake) and CI:
- `make lintdoc`: validate vimdoc files and test-generate HTML docs.
CI will run this as a part of the "docs" workflow.
- `scripts/lintdoc.lua` is added as an entry point (executable script)
for validating vimdoc files.
scripts/gen_help_html.lua:
- Move the tests for validating docs and generating HTMLs from
`help_spec.lua` to `gen_help_html`. Added:
- `gen_help_html.run_validate()`.
- `gen_help_html.test_gen()`.
- Do not hard-code `help_dir` to `build/runtime/doc`, but resolve from
`$VIMRUNTIME`. Therefore, the `make lintdoc` job will check doc files
on `./runtime/doc`, not on `./build/runtime/doc`.
- Add type annotations for gen_help_html.
Using CMAKE_INSTALL_PREFIX is unreliable as it's a cache variable,
meaning the users previous value will be used if not supplied. Instead,
use the `--prefix` flag which is guaranteed to always work.
Problem: Any preceding luadocs block that define alias types with
`@alias` magic would be prepended to the documentation of functions
that follow, despite the "blank line" separator. For example:
```
--- @alias some.type.between.functions
--- Blah blah long documentation for alias
--- | "foo" # foo
--- | "bar" # bar
--- The documentation that should appear in vimdoc.
function M.function_to_include_in_doc()
...
end
```
then the vimdoc generated for `function_to_include_in_doc` would include
the text from the alias block (e.g., "Blah blah ... for alias").
Solution:
- refactor: Lua2DoxFilter should maintain its own internal state
`generics`, rather than carrying it as a parameter to local helper
functions.
- Add another boolean state `boolean_state` which represents whether
to ignore the current docstring block (magic lines). This flag will
be reset as soon as the block is end.
Note: As expected, there is no change at all in the current docs
generated, because we have been working around and writing luadoc
comments so that such erroneous docstring resulting from preceding
`@alias` blocks won't appear.
- Add section `VIM.LPEG` and `VIM.RE` to docs/lua.txt.
- Add `_meta/re.lua` which adds luadoc and type annotations, for the
vendored `vim.re` package.
- Fix minor style issues on `_meta/lpeg.lua` luadoc for better vimdocs
generation.
- Fix a bug on `gen_vimdoc` where non-helptags in verbatim code blocks
were parsed as helptags, affecting code examples on `vim.lpeg.Cf`,
etc.
- Also move the `vim.regex` section below so that it can be located
closer to `vim.lpeg` and `vim.re`.
Problem: Some lines in the generated vim doc are overflowing, not
correctly wrapped at 78 characters. This happens when docs body contains
several consecutive 'inline' elements generated by doxygen.
Solution: Take into account the current column offset of the last line,
and prepend some padding before doc_wrap().
Since some point INCLUDE_DEPRECATED stopped working as it is usually
turned off when generating an actual vimdoc. This commit fixes this
hidden feature back again (used for devel purposes only).
Problem: main() has too much logic implemented there, too difficult to
read.
Solution: Do more OOP, introduce `Section` dataclass that stores
information about a "section", with documentation and concrete examples
about what each field and variable would mean. Extract all the lines for
rendering a section into `section.render()` pulled out of `main()`.
Problem:
For function definitions to be included in the vimdoc (formatted) and
to be exported as mpack data (unformatted), we had two internal
representations of the same function/API metadata in duplicate;
one is FunctionDoc (which was previously a dict), and the other is
doxygen XML DOM from which vimdoc (functions sections) was generated.
Solution:
We should have a single path and unified data representation
(i.e. FunctionDoc) that contains all the metadata and information about
function APIs, from which both of mpack export and vimdoc are generated.
I.e., vimdocs are no longer generated directly from doxygen XML nodes,
but generated via:
(XML DOM Nodes) ------------> FunctionDoc ------> mpack (unformatted)
Recursive Internal |
Formatting Metadata +---> vimdoc (formatted)
This refactoring eliminates the hacky and ugly use of `fmt_vimhelp` in
`fmt_node_as_vimhelp()` and all other helper functions! This way,
`fmt_node_as_vimhelp()` can simplified as it no longer needs to handle
generating of function docs, which needs to be done only in the topmost
level of recursion.
