Skip to content

feat(yaml_parser): parse document end token#7936

Merged
l0ngvh merged 1 commit intobiomejs:mainfrom
l0ngvh:multiple-documents
Nov 2, 2025
Merged

feat(yaml_parser): parse document end token#7936
l0ngvh merged 1 commit intobiomejs:mainfrom
l0ngvh:multiple-documents

Conversation

@l0ngvh
Copy link
Contributor

@l0ngvh l0ngvh commented Nov 1, 2025

Summary

Parse document end token .... As the name implied, a document end token marks the end of a document in a stream, allowing a new document to start.

Test Plan

Added new snapshots for multiple documents YAML as well as malformed document end.

Docs

N/A

@changeset-bot
Copy link

changeset-bot bot commented Nov 1, 2025 •
edited
Loading

⚠️ No Changeset found

Latest commit: c48cebc

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions github-actions bot added the A-Parser Area: parser label Nov 1, 2025
@l0ngvh l0ngvh changed the title feat(yaml_parser): parse doc end token feat(yaml_parser): parse document end token Nov 1, 2025
@l0ngvh l0ngvh force-pushed the multiple-documents branch from 8dbc9a0 to c48cebc Compare November 1, 2025 10:29
@l0ngvh l0ngvh requested review from a team November 1, 2025 10:47
@l0ngvh l0ngvh marked this pull request as ready for review November 1, 2025 10:47
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Nov 1, 2025

Walkthrough

This PR refactors the YAML lexer to introduce structured document-end handling and consolidate scope-closing logic. Key changes include renaming close_scope to close_breached_scopes, adding close_all_scopes for final scope drainage, implementing document-end detection via is_at_doc_end() and consume_doc_end(), and unifying trailing trivia handling through consume_trailing_trivia(). The parser's error recovery path has been updated to use unexpected_token instead of the removed malformed_document. Tests are added to validate document-end parsing and edge cases.

Possibly related PRs

Suggested reviewers

  • ematipico

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)
Check name Status Explanation
Title Check ✅ Passed The title "feat(yaml_parser): parse document end token" directly and clearly reflects the main objective of the changeset. The pull request's primary goal is to add parsing support for the YAML document end token (...), which is precisely what the title communicates. The title is specific, concise, and uses the conventional feat() format appropriately. It allows a teammate scanning the commit history to immediately understand the core purpose of this change without ambiguity.
Description Check ✅ Passed The pull request description is clearly related to the changeset. It articulates the core feature (parsing document end tokens), explains the purpose and functionality (marking document boundaries and allowing new documents to start), and references the test plan additions (snapshots for multiple-document YAML and malformed document end scenarios). The description aligns with the actual changes visible in the file summaries, including new test files and lexer/parser modifications. The note about documentation being N/A is also appropriate and transparent.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a35c496 and c48cebc.

⛔ Files ignored due to path filters (3)
  • crates/biome_yaml_parser/tests/yaml_test_suite/err/block/mapping_key_contains_multiple_values.yaml.snap is excluded by !**/*.snap and included by **
  • crates/biome_yaml_parser/tests/yaml_test_suite/err/document/doc_end_with_trailing_tokens.yaml.snap is excluded by !**/*.snap and included by **
  • crates/biome_yaml_parser/tests/yaml_test_suite/ok/document/separated_by_doc_end.yaml.snap is excluded by !**/*.snap and included by **
📒 Files selected for processing (8)
  • crates/biome_yaml_parser/src/lexer/mod.rs (9 hunks)
  • crates/biome_yaml_parser/src/lexer/tests/document.rs (1 hunks)
  • crates/biome_yaml_parser/src/lexer/tests/flow.rs (1 hunks)
  • crates/biome_yaml_parser/src/lexer/tests/mod.rs (1 hunks)
  • crates/biome_yaml_parser/src/parser/document.rs (2 hunks)
  • crates/biome_yaml_parser/src/parser/parse_error.rs (2 hunks)
  • crates/biome_yaml_parser/tests/yaml_test_suite/err/document/doc_end_with_trailing_tokens.yaml (1 hunks)
  • crates/biome_yaml_parser/tests/yaml_test_suite/ok/document/separated_by_doc_end.yaml (1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.{rs,toml}

📄 CodeRabbit inference engine (CONTRIBUTING.md)

Format Rust and TOML files before committing (e.g., via just f)

Files:

  • crates/biome_yaml_parser/src/lexer/tests/document.rs
  • crates/biome_yaml_parser/src/lexer/tests/flow.rs
  • crates/biome_yaml_parser/src/parser/document.rs
  • crates/biome_yaml_parser/src/parser/parse_error.rs
  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
  • crates/biome_yaml_parser/src/lexer/mod.rs
**/*.rs

📄 CodeRabbit inference engine (CONTRIBUTING.md)

Document rules, assists, and their options with inline rustdoc in the Rust source

Files:

  • crates/biome_yaml_parser/src/lexer/tests/document.rs
  • crates/biome_yaml_parser/src/lexer/tests/flow.rs
  • crates/biome_yaml_parser/src/parser/document.rs
  • crates/biome_yaml_parser/src/parser/parse_error.rs
  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
  • crates/biome_yaml_parser/src/lexer/mod.rs
🧠 Learnings (21)
📚 Learning: 2025-10-15T09:22:15.851Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_formatter/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:22:15.851Z
Learning: Applies to crates/biome_formatter/tests/language.rs : Create tests/language.rs defining `HtmlTestFormatLanguage` and implement the TestFormatLanguage trait

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/document.rs
  • crates/biome_yaml_parser/src/lexer/tests/flow.rs
  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
📚 Learning: 2025-10-15T09:25:05.698Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_service/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:25:05.698Z
Learning: Applies to crates/biome_service/../biome_lsp/src/server.tests.rs : Keep end-to-end LSP tests in ../biome_lsp/src/server.tests.rs

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/document.rs
  • crates/biome_yaml_parser/src/lexer/tests/flow.rs
  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
📚 Learning: 2025-10-15T09:24:31.042Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_parser/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:24:31.042Z
Learning: Applies to crates/biome_parser/crates/**/src/lexer/mod.rs : Create a lexer module at crates/<parser_crate>/src/lexer/mod.rs

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/document.rs
  • crates/biome_yaml_parser/src/lexer/tests/flow.rs
  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
  • crates/biome_yaml_parser/src/lexer/mod.rs
📚 Learning: 2025-10-24T21:24:58.650Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_analyze/CONTRIBUTING.md:0-0
Timestamp: 2025-10-24T21:24:58.650Z
Learning: Applies to crates/biome_analyze/crates/*_analyze/**/src/**/lint/**/*.rs : In ### Invalid doc examples, each snippet must use the expect_diagnostic modifier and emit exactly one diagnostic

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/document.rs
  • crates/biome_yaml_parser/src/parser/parse_error.rs
  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
📚 Learning: 2025-10-15T09:24:31.042Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_parser/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:24:31.042Z
Learning: Lexer must implement the biome_parser::Lexer trait

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/document.rs
  • crates/biome_yaml_parser/src/lexer/tests/flow.rs
  • crates/biome_yaml_parser/src/parser/parse_error.rs
  • crates/biome_yaml_parser/src/lexer/mod.rs
📚 Learning: 2025-10-15T09:22:15.851Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_formatter/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:22:15.851Z
Learning: Applies to crates/biome_formatter/tests/spec_tests.rs : In tests/spec_tests.rs, generate tests with `tests_macros::gen_tests! {"tests/specs/html/**/*.html", crate::spec_test::run, ""}`

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/document.rs
  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
📚 Learning: 2025-10-24T21:24:58.650Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_analyze/CONTRIBUTING.md:0-0
Timestamp: 2025-10-24T21:24:58.650Z
Learning: Applies to crates/biome_analyze/crates/*_analyze/**/src/**/lint/**/*.rs : Every code block in docs must declare a language

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/document.rs
📚 Learning: 2025-10-15T09:22:15.851Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_formatter/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:22:15.851Z
Learning: Applies to crates/biome_formatter/tests/** : Create a tests directory containing a specs subfolder and the files spec_test.rs, spec_tests.rs, and language.rs

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/document.rs
  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
📚 Learning: 2025-10-15T09:25:05.698Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_service/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:25:05.698Z
Learning: Applies to crates/biome_service/src/workspace/watcher.tests.rs : Place watcher tests related to workspace methods in src/workspace/watcher.tests.rs

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/document.rs
  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
📚 Learning: 2025-10-15T09:22:15.851Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_formatter/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:22:15.851Z
Learning: Applies to crates/biome_formatter/tests/spec_test.rs : Implement a `run` function in tests/spec_test.rs that wires SpecSnapshot and includes!("language.rs") as shown

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/document.rs
  • crates/biome_yaml_parser/src/lexer/tests/flow.rs
  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
📚 Learning: 2025-10-15T09:22:46.002Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_js_formatter/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:22:46.002Z
Learning: Applies to crates/biome_js_formatter/**/*.rs : For non-mandatory tokens, use the provided helper constructors (e.g., `token`, `space_token`, `dynamic_token`)

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/flow.rs
  • crates/biome_yaml_parser/src/parser/parse_error.rs
📚 Learning: 2025-10-15T09:22:46.002Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_js_formatter/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:22:46.002Z
Learning: Applies to crates/biome_js_formatter/**/*.rs : Do not attempt to fix code; if a mandatory token/node is missing, return `None` instead

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/flow.rs
  • crates/biome_yaml_parser/src/parser/document.rs
  • crates/biome_yaml_parser/src/parser/parse_error.rs
📚 Learning: 2025-10-24T21:24:58.650Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_analyze/CONTRIBUTING.md:0-0
Timestamp: 2025-10-24T21:24:58.650Z
Learning: Applies to crates/biome_analyze/crates/*_analyze/**/src/**/lint/**/*.rs : Avoid avoidable string allocations; compare against &str or TokenText rather than allocating Strings

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/flow.rs
📚 Learning: 2025-10-15T09:22:46.002Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_js_formatter/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:22:46.002Z
Learning: Applies to crates/biome_js_formatter/**/*.rs : When a token is mandatory and present in the AST, use the AST-provided token (e.g., `node.l_paren_token().format()`) instead of emitting a static token

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/flow.rs
  • crates/biome_yaml_parser/src/parser/document.rs
  • crates/biome_yaml_parser/src/parser/parse_error.rs
📚 Learning: 2025-10-15T09:22:15.851Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_formatter/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:22:15.851Z
Learning: Applies to crates/biome_formatter/src/**/*.rs : After generation, remove usages of `format_verbatim_node` and implement real formatting with biome_formatter utilities

Applied to files:

  • crates/biome_yaml_parser/src/parser/parse_error.rs
📚 Learning: 2025-10-15T09:24:31.042Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_parser/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:24:31.042Z
Learning: Applies to crates/biome_parser/xtask/codegen/*.ungram : Nodes for enclosing syntax errors must include the Bogus word (e.g., HtmlBogusAttribute)

Applied to files:

  • crates/biome_yaml_parser/src/parser/parse_error.rs
📚 Learning: 2025-10-24T21:24:58.650Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_analyze/CONTRIBUTING.md:0-0
Timestamp: 2025-10-24T21:24:58.650Z
Learning: Applies to crates/biome_analyze/crates/*_analyze/**/src/**/lint/**/*.rs : Provide informative diagnostics: explain what the error is, why it triggers, and what to do (prefer a code action or a note)

Applied to files:

  • crates/biome_yaml_parser/src/parser/parse_error.rs
📚 Learning: 2025-10-15T09:24:31.042Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_parser/CONTRIBUTING.md:0-0
Timestamp: 2025-10-15T09:24:31.042Z
Learning: Use p.eat for optional tokens, p.expect for required tokens; use .ok() for optional nodes and .or_add_diagnostic(...) for required nodes

Applied to files:

  • crates/biome_yaml_parser/src/parser/parse_error.rs
📚 Learning: 2025-10-24T21:24:58.650Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_analyze/CONTRIBUTING.md:0-0
Timestamp: 2025-10-24T21:24:58.650Z
Learning: Applies to crates/biome_analyze/crates/*_analyze/**/src/**/lint/**/*.rs : Use options/full_options/use_options code block modifiers as specified to demonstrate configuration in docs; keep modifier order consistent

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
📚 Learning: 2025-10-24T21:24:58.650Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: crates/biome_analyze/CONTRIBUTING.md:0-0
Timestamp: 2025-10-24T21:24:58.650Z
Learning: Applies to crates/biome_analyze/crates/*_analyze/tests/quick_test.rs : Quick test lives in biome_js_analyze/tests/quick_test.rs; unignore #[ignore] and set rule filter and SOURCE for ad-hoc runs

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
📚 Learning: 2025-10-26T15:28:00.951Z 
Learnt from: CR
Repo: biomejs/biome PR: 0
File: CONTRIBUTING.md:0-0
Timestamp: 2025-10-26T15:28:00.951Z
Learning: Applies to **/*.rs : Document rules, assists, and their options with inline rustdoc in the Rust source

Applied to files:

  • crates/biome_yaml_parser/src/lexer/tests/mod.rs
🧬 Code graph analysis (3)
crates/biome_yaml_parser/src/parser/document.rs (1)
crates/biome_yaml_parser/src/parser/parse_error.rs (2)
  • expected_directive (18-20)
  • unexpected_token (42-49)
crates/biome_yaml_parser/src/parser/parse_error.rs (3)
crates/biome_parser/src/diagnostic.rs (1)
  • expected_node (481-483)
crates/biome_yaml_parser/src/parser/mod.rs (1)
  • new (24-29)
crates/biome_yaml_parser/src/token_source.rs (1)
  • new (15-20)
crates/biome_yaml_parser/src/lexer/mod.rs (1)
crates/biome_yaml_parser/src/token_source.rs (2)
  • current (59-61)
  • new (15-20)
🪛 YAMLlint (1.37.1)
crates/biome_yaml_parser/tests/yaml_test_suite/err/document/doc_end_with_trailing_tokens.yaml

[error] 1-1: syntax error: expected the node content, but found ''

(syntax)

crates/biome_yaml_parser/tests/yaml_test_suite/ok/document/separated_by_doc_end.yaml

[error] 3-3: syntax error: expected '', but found ''

(syntax)

Comment @coderabbitai help to get the list of available commands and usage tips.

@l0ngvh l0ngvh merged commit 7bbd1b4 into biomejs:main Nov 2, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dyc3 dyc3 dyc3 approved these changes

Assignees

No one assigned

Labels

A-Parser Area: parser

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@l0ngvh @dyc3