Problem:
The documentation flow (`gen_vimdoc.py`) has several issues:
- it's not very versatile
- depends on doxygen
- doesn't work well with Lua code as it requires an awkward filter script to convert it into pseudo-C.
- The intermediate XML files and filters makes it too much like a rube goldberg machine.
Solution:
Re-implement the flow using Lua, LPEG and treesitter.
- `gen_vimdoc.py` is now replaced with `gen_vimdoc.lua` and replicates a portion of the logic.
- `lua2dox.lua` is gone!
- No more XML files.
- Doxygen is now longer used and instead we now use:
- LPEG for comment parsing (see `scripts/luacats_grammar.lua` and `scripts/cdoc_grammar.lua`).
- LPEG for C parsing (see `scripts/cdoc_parser.lua`)
- Lua patterns for Lua parsing (see `scripts/luacats_parser.lua`).
- Treesitter for Markdown parsing (see `scripts/text_utils.lua`).
- The generated `runtime/doc/*.mpack` files have been removed.
- `scripts/gen_eval_files.lua` now instead uses `scripts/cdoc_parser.lua` directly.
- Text wrapping is implemented in `scripts/text_utils.lua` and appears to produce more consistent results (the main contributer to the diff of this change).
Summary: Separate the lint job (`make lintdoc`) to validate runtime/doc,
it is no longer as a part of functionaltest (help_spec).
Build (cmake) and CI:
- `make lintdoc`: validate vimdoc files and test-generate HTML docs.
CI will run this as a part of the "docs" workflow.
- `scripts/lintdoc.lua` is added as an entry point (executable script)
for validating vimdoc files.
scripts/gen_help_html.lua:
- Move the tests for validating docs and generating HTMLs from
`help_spec.lua` to `gen_help_html`. Added:
- `gen_help_html.run_validate()`.
- `gen_help_html.test_gen()`.
- Do not hard-code `help_dir` to `build/runtime/doc`, but resolve from
`$VIMRUNTIME`. Therefore, the `make lintdoc` job will check doc files
on `./runtime/doc`, not on `./build/runtime/doc`.
- Add type annotations for gen_help_html.
