Fix lexer edge cases and string handling - #505

JanJakes · 2026-09-08T07:51:38Z

Summary

This PR fixes lexer edge cases and related string handling:

String values: Preserve binary and single-byte charset payloads and correctly decode escaped newlines.
Function names: Exclude trailing whitespace from function-name token ranges under IGNORE_SPACE.
Version comments: Support six-digit versions from MySQL 8.1, preserve five-digit fallback parsing and unversioned numeric content, and recognize vertical tabs as whitespace.
WordPress validation: Mirror the parent wpdb::query() charset checks, retaining its binary-data and prevalidated-query exemptions.
LIKE BINARY patterns: Preserve raw bytes instead of producing NULL during GLOB conversion, and handle escaped newlines.

The lexer fixes cover both PHP packages and the native extension.

Why

The token decoder's UTF-8 regex modifier caused an accidental TypeError for legitimate binary data as well as malformed text. Decoding must preserve bytes; charset enforcement needs column types and SQL mode. Mirroring WordPress's checks restores its expected validation behavior.

Regression tests cover decoding, token ranges, version comments, and pattern conversion. CI expectations remove the test_strip_invalid_text cases fixed by decoding and the two invalid-query tests fixed by WordPress validation.

Remaining limitations

Direct MySQL-on-SQLite writes can still accept invalid text under strict mode; full charset enforcement remains driver work. WordPress's checks retain their exemptions, including tables containing binary columns.
SQLite GLOB can still treat distinct invalid UTF-8 sequences as equal. The LIKE BINARY fix preserves pattern bytes but does not provide fully byte-correct matching.
Legacy multibyte charsets such as Big5 and GBK remain unsupported by the decoder.

coderabbitai · 2026-09-08T07:52:02Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: a4d89563-b551-4294-b3a4-b3460c8a5eab

📥 Commits

Reviewing files that changed from the base of the PR and between 4ade8df and 8cd2a07.

📒 Files selected for processing (11)

.github/workflows/wp-tests-phpunit-run.js
packages/mysql-on-sqlite/src/mysql/class-wp-mysql-lexer.php
packages/mysql-on-sqlite/src/mysql/class-wp-mysql-token.php
packages/mysql-on-sqlite/src/sqlite/class-wp-mysql-on-sqlite.php
packages/mysql-on-sqlite/src/sqlite/class-wp-sqlite-pdo-user-defined-functions.php
packages/mysql-on-sqlite/tests/WP_MySQL_On_SQLite_Tests.php
packages/mysql-parser/src/class-wp-mysql-lexer.php
packages/mysql-parser/src/class-wp-mysql-token.php
packages/mysql-parser/tests/WP_MySQL_Server_Suite_Parser_Tests.php
packages/php-ext-wp-mysql-parser/src/lib.rs
packages/plugin-sqlite-database-integration/wp-includes/sqlite/class-wp-sqlite-db.php

💤 Files with no reviewable changes (1)

.github/workflows/wp-tests-phpunit-run.js

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

📝 Walkthrough

Walkthrough

The change updates MySQL token handling, lexer parsing, LIKE pattern translation, and database query validation. It adds coverage for binary data, escaped newlines, vertical tabs, versioned comments, token ranges, LIKE patterns, and invalid query text.

Changes

MySQL compatibility updates

Layer / File(s)	Summary
Binary-safe token and LIKE values `packages/mysql-parser/src/class-wp-mysql-token.php`, `packages/mysql-on-sqlite/src/mysql/`, `packages/mysql-on-sqlite/src/sqlite/`, `packages/mysql-parser/tests/WP_MySQL_Token_Tests.php`, `packages/mysql-on-sqlite/tests/*`	Token and LIKE escape handling preserves non-UTF-8 bytes and unescapes escaped newlines. Tests cover raw bytes, multibyte values, wildcards, and GLOB metacharacters.
Lexer whitespace and comment parsing `packages/mysql-parser/src/class-wp-mysql-lexer.php`, `packages/mysql-on-sqlite/src/mysql/class-wp-mysql-lexer.php`, `packages/php-ext-wp-mysql-parser/src/`, `packages/mysql-parser/tests/`, `packages/mysql-on-sqlite/tests/mysql/*`, `.github/workflows/wp-tests-phpunit-run.js`	The lexers recognize vertical tabs, support six-digit MySQL version comments, and keep ignored whitespace out of function token ranges. Tests cover version boundaries, whitespace variants, token offsets, corpus configuration, and PHPUnit expectations.
Invalid query text validation `packages/plugin-sqlite-database-integration/wp-includes/sqlite/class-wp-sqlite-db.php`	`WP_SQLite_DB::query()` validates marked queries after charset discovery, records rejected query details, resets validation state, and updates query-count tracking.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~30 minutes

Merge Risk: ⚪ Minimal · up to 8cd2a

This updates MySQL-compatible lexing and binary string handling, fixes token ranges and version-comment parsing, and adds invalid-query validation. The supplied coverage and current implementation summaries show no remaining merge-blocking risk.

Sequence Diagram(s)

sequenceDiagram
  participant SQLInput as SQL input
  participant WP_MySQL_Lexer
  participant WP_MySQL_Token
  SQLInput->>WP_MySQL_Lexer: tokenize comments, whitespace, and function syntax
  WP_MySQL_Lexer->>WP_MySQL_Token: emit token bytes and ranges
  WP_MySQL_Token->>WP_MySQL_Token: unescape token value

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 36 functions across 15 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes: fixes for lexer edge cases and string handling. It is concise and related to the pull request objectives.

Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch lexer-improvements

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-08T07:52:27Z

🤖 Lexer benchmark

Changes to lexer-related files were detected and triggered a benchmark:

Config	Base (QPS)	This PR (QPS)	Speedup
no JIT	70,197	70,109	1.00×
tracing JIT	158,144	157,337	0.99×

Note: Hosted runners are noisy, and absolute numbers vary. Treat the results with caution and verify them locally.

To reproduce locally:

cd packages/mysql-on-sqlite && composer run bench-lexer

mho22

Looks good to me!

The final backslash-stripping step used preg_replace() with the "u" (UTF-8) modifier. Invalid UTF-8 made it return null, causing a TypeError because get_value() must return a string. MySQL literals can legitimately contain non-UTF-8 bytes in binary or single-byte charset payloads. Switch the modifier to "s" (DOTALL). Byte-wise unescaping preserves valid UTF-8 and raw bytes, and also handles a backslash followed by a newline. Legacy multibyte charsets such as Big5 and GBK remain unsupported. Remove 14 obsolete WordPress test failure expectations. Reclassify Tests_DB_Charset::test_invalid_characters_in_query as an expected assertion failure: invalid text is accepted instead of crashing at this step. Charset enforcement remains separate from token decoding.

When resolving a function keyword (SYM_FN), the lexer peeks for a following "(" and, under SQL_MODE_IGNORE_SPACE, skips intervening whitespace first. It skipped by advancing bytes_already_read and never restored it. When no "(" followed, the keyword was emitted as an IDENTIFIER whose length — derived from bytes_already_read in produce() — now covered the trailing whitespace, so the extracted value was e.g. "COUNT " instead of "COUNT". Under this ANSI-style mode a column or table named after a function would resolve to the wrong identifier. Peek with a local index instead of mutating bytes_already_read, so the token's byte range ends at the keyword and the next scan consumes the whitespace.

Recognize six-digit version numbers followed by whitespace starting with MySQL 8.1. Preserve five-digit parsing for older server versions and when the sixth digit is not followed by whitespace. Keep fewer than five leading digits as SQL content. Accept all MySQL ASCII whitespace, including vertical tabs, in the PHP and native lexers. Test version boundaries, whitespace, fallback parsing, and unversioned numeric content. Keep the corpus expectations pinned to MySQL 8.0.38. https://dev.mysql.com/blog-archive/are-you-ready-for-mysql-10/ https://github.com/mysql/mysql-server/blob/mysql-8.1.0/sql/sql_lex.cc

Use wpdb's charset checks to reject invalid text before executing a query, while preserving binary data and prevalidated queries. Keep Query Monitor entries aligned when validation performs charset lookups. Remove the two obsolete failure expectations for the existing WordPress invalid-text tests. Document that full charset enforcement belongs in MySQL on SQLite so it protects all callers, including direct driver use.

Remove UTF-8 validation from LIKE-to-GLOB pattern conversion so raw bytes do not turn the pattern into NULL. Use DOTALL to also unescape backslashes followed by newlines. Cover byte preservation, malformed UTF-8, Unicode, wildcards, escapes, and NULL in helper and query tests. Document the remaining GLOB limitation: distinct invalid UTF-8 sequences can match as equal.

## Release `3.0.2` Version bump and changelog update for release `3.0.2`. **Changelog draft:** * Fix schema reconstruction with native numeric results ([#506](#506)) * Fix lexer edge cases and string handling ([#505](#505)) * Preserve index prefix lengths and order in SHOW CREATE TABLE primary keys ([#500](#500)) * Align savepoint handling with MySQL semantics ([#496](#496)) **Full changelog:** v3.0.1...release/v3.0.2 ## Next steps 1. **Review** the changes in this pull request. 2. **Push** any additional edits to this branch (`release/v3.0.2`). 3. **Merge** this pull request to complete the release. Merging will automatically build the plugin ZIP, create a [GitHub release](https://github.com/WordPress/sqlite-database-integration/releases), and deploy to [WordPress.org](https://wordpress.org/plugins/sqlite-database-integration/).  ## Summary by CodeRabbit - **Bug Fixes** - Improved schema reconstruction when handling native numeric results. - Corrected lexer edge cases and string processing. - Fixed index prefix lengths and ordering in `SHOW CREATE TABLE` output for primary keys. - Improved savepoint handling to better match MySQL behavior. - **Release** - Updated the SQLite integration to version 3.0.2.

Brings in WordPress#496 (MySQL savepoint semantics), WordPress#500 (index prefix lengths in SHOW CREATE TABLE) and WordPress#505 (lexer edge cases and string handling). The one conflict is in the engine's transaction and savepoint handling: WordPress#496 reintroduced direct connection->query('BEGIN IMMEDIATE'/'COMMIT'/'ROLLBACK') plus its own $in_transaction and $savepoint_names tracking, while this branch routes transaction control through WP_SQLite_Connection_Interface so that remote backends can implement it themselves. Resolved by keeping upstream's semantics and this branch's dispatch: the new savepoint-name bookkeeping (MySQL replaces a savepoint on name reuse where SQLite shadows it, RELEASE and ROLLBACK TO drop the savepoints created after the named one, and an unknown name raises the new exception) is preserved, but the statements go through begin_transaction()/commit()/rollback()/savepoint()/ release_savepoint()/rollback_to_savepoint(). Upstream's engine-level $in_transaction polyfill is dropped as dead state, because inTransaction() on this branch already delegates to the connection. Verified: packages/mysql-on-sqlite 1080 tests, 1429272 assertions, no failures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

JanJakes force-pushed the lexer-improvements branch 2 times, most recently from f695a70 to 4ade8df Compare September 8, 2026 10:13

JanJakes marked this pull request as ready for review September 8, 2026 10:23

JanJakes requested a review from mho22 September 8, 2026 11:40

mho22 approved these changes Sep 9, 2026

View reviewed changes

Comment thread packages/mysql-parser/tests/WP_MySQL_Server_Suite_Parser_Tests.php

JanJakes added 5 commits September 9, 2026 11:51

JanJakes force-pushed the lexer-improvements branch from 4ade8df to 8cd2a07 Compare September 9, 2026 09:51

JanJakes changed the title ~~Fix lexer string values, token ranges, and version comments~~ Fix lexer edge cases, WordPress validation, and LIKE BINARY Sep 9, 2026

JanJakes changed the title ~~Fix lexer edge cases, WordPress validation, and LIKE BINARY~~ Fix lexer edge cases and string handling Sep 9, 2026

mho22 approved these changes Sep 9, 2026

View reviewed changes

JanJakes merged commit 69340fe into trunk Sep 9, 2026
35 checks passed

JanJakes deleted the lexer-improvements branch September 9, 2026 12:48

batonac mentioned this pull request Sep 9, 2026

Sync with upstream trunk (2 commits) Avunu/sqlite-database-integration#10

Open

JanJakes mentioned this pull request Sep 10, 2026

Release 3.0.2 #507

Merged

Navigation Menu

Uh oh!

Fix lexer edge cases and string handling - #505