203 Commits

Author SHA1 Message Date
Michael Davis
e2d60932f7 Add NOTICE for erlang test grammar
Apache-2.0 on its own doesn't list the copyright holder since you are
not meant to edit the license text (unlike the MIT for example). So the
license file alone isn't enough, we also need the notice (recently
pushed upstream to tree-sitter-erlang). Also see the NOTICE in
tree-sitter-elixir which is also Apache-2.0 licensed.
2025-09-24 12:16:36 -04:00
Michael Davis
bc014cf19a bindings: Vendor v0.25.9 2025-09-06 19:08:59 -04:00
Michael Davis
cfce9b7c82 bindings: Release v0.2.2 2025-08-31 10:39:15 -04:00
Isaac Mills
e8b04f3d84 feat: Add support for loading tree-sitter-language grammars (#24) 2025-08-31 09:32:03 -05:00
blinxen
029bf646f0 Remove unused dependency (#23) 2025-07-17 15:32:12 -05:00
Michael Davis
bd02eab611 Expand README, link to 25.07 release notes 2025-07-15 15:29:57 -04:00
Michael Davis
9fe0be04c3 bindings: Fix error message for impossible patterns 2025-07-15 15:29:57 -04:00
Michael Davis
da576cf74e bindings: Vendor v0.25.8
This tag only changed things in tree-sitter-cli.
2025-07-14 09:46:17 -04:00
Michael Davis
45c3dce31b Bump tree-house-bindings to v0.2.1 2025-07-12 17:30:42 -04:00
Michael Davis
ca57556f58 bindings: Vendor v0.25.7 2025-07-12 17:06:49 -04:00
Michael Davis
d4a395aca8 highlighter: Fix index into active_highlights in debug assertion
The number of parent highlights can decrease after entering an injection
of a child layer - if those highlights end - so when looking up the
highlights of the current layer we need to be careful to clamp the
parent layer's `parent_highlights` index to the current length of
`active_highlights`.

Co-authored-by: Peter Retzlaff <pe.retzlaff@gmail.com>
2025-07-07 19:31:42 -04:00
Michael Davis
74e37686b0 skidder: Resolve clippy 1.88 uninlined_format_args lint 2025-07-05 10:57:23 -04:00
Michael Davis
0c7d6a94a7 CI: Run everything against the MSRV 2025-07-05 10:56:15 -04:00
Michael Davis
67dbeba728 highlighter: Add a test case for Rust nested vec macros 2025-07-05 10:51:22 -04:00
Michael Davis
e56dc4b261 flake: Fix compatibility with aarch64 macOS 2025-07-05 10:50:58 -04:00
Michael Davis
b2823754c4 bindings: Start changelog notes for next release 2025-06-18 10:46:51 -04:00
Rafael Oliveira
ddc2909fc6 bindings: Add Node::is_extra (#19)
Co-authored-by: Rafael Oliveira <tavares.rafael@luizalabs.com>
2025-06-18 09:43:06 -05:00
Michael Davis
470b507dd8 Bump tree-house version to v0.3.0, add changelog entries 2025-06-16 10:22:22 -04:00
Michael Davis
144f5248b3 highlighter: Fix clippy map_or/is_none_or lint 2025-06-14 21:49:51 -04:00
Michael Davis
a75c71b10b CI: Update MSRV 2025-06-14 21:47:51 -04:00
Michael Davis
fa5a5c7b9d highlighter: Finish layers when all highlights on the layer are done
In the test cases the edoc layer (the backtick and single quote within
the comments) is recognized by `QueryIter` as being finished after it
highlights the inline code block - its only capture. The highlighter
needs to "deactivate" the layer rather than finishing (and removing it)
however, because the code block highlight extends to the next comment.

So when exiting an injection we check that the `QueryIter` considers a
layer to be finished and also that all highlights in the layer are done
as well.
2025-06-14 21:41:20 -04:00
Michael Davis
130363bb88 highlighter: Maintain highlight ordering when captures emit out-of-order
This fixes a case that pops up with some specific CSS highlights which
causes the `active_highlights` stack to become out of order. The odd
highlight is this:

    "#" @punctuation
    ((color_value) "#") @string.special
    (color_value) @string.special

The `#` node is fought over by the first two patterns. Requiring that
the pattern matches `#` in the second pattern causes tree-sitter to
finish its capture after the first pattern, so we capture the child node
`{Node # 9..10}` first and then `{Node color_value 9..13}` - reversed of
the normal ordering.

In this case we need to maintain the ordering of `active_highlights` by
`Vec::insert`ing into the correct position.
2025-06-14 21:35:33 -04:00
Michael Davis
3a8552889c Import CSS test grammar 2025-06-14 21:29:18 -04:00
Michael Davis
9ac17cbcfc minor: Document Highlighter active_highlights 2025-06-14 21:28:05 -04:00
Michael Davis
5c0fd31045 Add a debug assertion for active_highlights ordering 2025-06-14 21:28:05 -04:00
Michael Davis
6128c5a31d highlighter: Eagerly panic when highlight ranges are out of order
The errors were collected into the string but `errors` was never read.
Instead of collecting errors I think it's better for the sake of
debugging to panic eagerly.
2025-06-14 13:04:44 -04:00
Michael Davis
987bba27f9 highlighter: Add a tiny inline markdown case for debugging 2025-06-14 13:01:16 -04:00
Michael Davis
1a053fab41 minor: Eagerly panic when a fixture file can't be read
This has a more direct error message when you typo the filename in a
highlighter test.
2025-06-14 12:59:02 -04:00
Michael Davis
959309a5e8 Bump bindings & highlighter to v0.2.0 2025-06-06 17:10:32 -04:00
Michael Davis
4c61c57fcc Add changelog entries for bindings & highlighter v0.2.0 2025-06-06 17:08:41 -04:00
Michael Davis
a3ff253c7f bindings: Vendor v0.25.6 2025-06-04 12:46:04 -04:00
Skyler Hawthorne
94f3c81d14 add recursive walker (#16) 2025-06-01 12:11:15 -05:00
Michael Davis
07195d921c bindings: Require range and match limit in InactiveQueryCursor::new
Because of the way that cursors are cached, we need to set or reset
the byte range and match limit when re-using from the cache. In most
cases callers should set a byte range and match limit, so rather than
resetting these we require them in `new`.
2025-06-01 13:07:47 -04:00
Michael Davis
f9fb1181ff highlighter: Remove unused query cursor in Syntax::update
This is from an earlier prototype and is unused.
2025-06-01 13:00:52 -04:00
Michael Davis
0241884cf2 highlighter: Consider any skipped injection layers to be modified
This fixes an issue where an edit to a document which removes an
injection layer could eventually cause a panic that `parent_ranges` is
empty in `intersect_ranges`: `Option::unwrap` on a `None` value.

The test case edits a markdown block where a codefence is created
between two HTML comments. Before the edit, the document has a combined
injection for the two HTML comments. After the edit, the second HTML
comment is reinterpreted by the markdown parser as the content of the
codefence - not an HTML comment anymore.

In this case the HTML layer must be re-parsed because its included
ranges have been changed the edit: before it was a range for each
comment and after it is only the first comment. Previously we did not
mark the HTML layer as modified since it was not directly edited. But
this meant that, when running the HTML injection query over this layer
after the edit, we got query matches for the second HTML comment since
the node still existed according to the layer's tree. This second
injection capture did not intersect with any parent ranges since the
layer's ranges no longer included the second HTML comment, causing
`parent_ranges` to be empty and an unwrap of `None`.

The fix is to mark any layers which are not reused as modified. If a
layer is a combined injection and not all layers are reused then the
layer must be re-parsed because its included ranges have changed. All
modified layers are re-parsed. To do this we look at the old injections
for each layer and any that are skipped or discarded rather than reused
have their layer marked as modified.

Note that this does not cause layers which have been removed from the
tree to stick around. The layers are only marked as modified (meaning
that they need to be re-parsed) and not marked as touched (meaning that
they are still valid).

Also note that this is not an issue for adding injection ranges to a
combined injection since we always consider a layer to be modified if
we cannot reuse an injection.

A practical reproduction of this (and the source of the test) is
mentioned in helix-editor/helix#13544.
2025-06-01 12:36:23 -04:00
Michael Davis
7acb839083 minor: Improve readability of test loader skip messages
Be a bit more verbose about what is being skipped. And color the
messages cyan.
2025-06-01 10:28:32 -04:00
Michael Davis
1681ba06a4 highlighter: Consider layers finished if they have no more injections
Previously the `QueryIter` dropped layers which had no remaining
captures but the layer could potentially have more injections. In this
case the `QueryIter` would drop the layer and then recreate it later
when the parent layer entered another injection for this layer. This
could be potentially wasteful if the `QueryIter`'s `LayerState` type has
an expensive `Default` implementation.

We should prefer to emit `QueryIterEvent::ExitInjection` with `Some`
state (meaning that the layer is done and the state can be dropped) only
when the layer will not be entered again.
2025-06-01 10:28:32 -04:00
Michael Davis
f3b47bea1a highlighter: Eagerly drop finished query cursors
We can return a query cursor which has no more captures to the thread-
local cache as soon as it is done so that another caller can use it.
2025-06-01 10:28:32 -04:00
Michael Davis
c9fa54f2eb highlighter: Tombstone finished layers in the QueryIter
This fixes a bug where a layer could be removed from `QueryIter`'s
`active_layers` because it had no more captures and then reinitialized
later, restarting the query cursor and producing duplicate captures.

The new test case exhibits this behavior. Consider this markdown.

    ```rust
    /// `Something`
    /// Anything
    ```

This consists of four layers:

* The root layer, the entire markdown document
* The Rust injection in the codefence
* Two markdown injections within the Rust doc comments. These injections
  are tagged with `injection.combined` so they form one markdown layer.
* Two markdown.inline injections within those markdown injections. Again
  these injections are combined so they form one layer.

Layers are removed from `QueryIter` when they have no more captures. In
this example code the "`Something`" was the only capture in the two
injections of the markdown.inline layer, so after the first injection
was finished, the layer would be removed. Once the `QueryIter` reached
the "Anything" region, though, it would create a new `ActiveLayer` for
the markdown.inline layer and recreate its query cursor. Then the query
cursor would emit the same highlights as it did previously for
"`Something`".

This breaks an invariant that the `Highlighter::next_event_offset` must
be monotonically increasing. The added test case failed with:

    thread 'tests::codefence_rust_doc_comments' panicked at /.../ropey-1.6.1/src/slice.rs:703:23:
    byte_slice(): Invalid byte range 27..12: start must be <= end

Also see helix-editor/helix#13569.

The solution is to "tombstone" any layers which are finished in a new
`HashSet<Layer>` and avoid recreating query cursors for any finished
layers. Outside of this bug, this also prevents us from needlessly
creating query cursors which we know will yield nothing. Query cursors
are reused in a thread-local cache, so by not creating a query cursor we
allow another caller to use the cached cursor.
2025-06-01 10:28:19 -04:00
Michael Davis
a8aa76d5b7 bindings: Vendor v0.25.5 2025-05-29 09:23:29 -04:00
blinxen
ba30877d17 Add missing license files for crates (#14) 2025-05-25 10:27:23 -05:00
Nik Revenco
e10af057c8 feat: add Syntax::layers_for_byte_range (#9)
Co-authored-by: Nik Revenco <154856872+NikitaRevenco@users.noreply.github.com>
Co-authored-by: Michael Davis <mcarsondavis@gmail.com>
2025-05-24 11:08:24 -05:00
Ross Smyth
b9702ee945 Add ability to add a custom version with nix 2025-05-22 10:18:44 -04:00
Ross Smyth
ed7e6634d8 Add skidder-cli to nix flake outputs 2025-05-22 10:18:44 -04:00
Michael Davis
778853ae64 highlighter: Avoid cloning injection properties
Cloning the properties can be avoided and it is relatively expensive in
terms of temporary allocations. The nice part of cloning was being able
to use `unwrap_or_default` to get default injection properties for
injections which don't use any `#set! injection.X` properties but we can
handle this where the properties are accessed instead.

Closes #12
2025-05-22 10:11:22 -04:00
Michael Davis
16e11895a6 cli: Show help output when no arguments are specified 2025-05-19 11:31:27 -04:00
Ross Smyth
f037c85154 Add skidder cli version flag 2025-05-19 11:31:27 -04:00
Michael Davis
4e6f4d294c bindings: Prepare a v0.1.1 release 2025-05-14 16:53:18 -04:00
Michael Davis
779ab28a83 bindings: Patch endian.h to include IllumOS
According to <https://redirect.github.com/mikepb/endian.h/issues/4> this
should be supported and seems to fix the IllumOS build. Also see the
PR for upstream: <https://redirect.github.com/tree-sitter/tree-sitter/pull/4450>.

Co-authored-by: Mike Zeller <mike@mikezeller.net>
2025-05-14 16:48:44 -04:00
Michael Davis
524ef1260f Prepare a v0.1.0 release for the bindings & highlighter 2025-05-13 18:41:00 -04:00