Lexer
=====

The CrossGL lexer turns source text into ``(token_type, text)`` pairs consumed
by ``crosstl.translator.parser.Parser``.

Token ordering
--------------

Token patterns are stored in an ``OrderedDict``. More specific patterns must
appear before broader patterns because the combined regular expression reports
the first matching named group. For example, compound operators such as ``+=``
and ``<<=`` are listed before their single-character forms, and identifiers are
listed near the end.

Keyword handling
----------------

Identifier text is checked against ``KEYWORDS`` after token matching. This lets
the lexer share one identifier regex while still emitting parser-friendly token
types such as ``STRUCT``, ``COMPUTE``, ``VEC4``, and ``SAMPLER2D``.

Comments and whitespace
-----------------------

Whitespace is skipped. Comment tokens are preserved so parser helpers can
consume them consistently with ``skip_comments()`` while still allowing source
frontends to retain comment awareness if needed.

Errors
------

If no token pattern matches the current character, the lexer raises a
``SyntaxError`` with a line, column, source line, and caret pointer. This keeps
syntax errors actionable before parsing begins.

Debugging
---------

Use ``Lexer(code).debug_print()`` to print token indexes, token types, and raw
text. This is useful when adding grammar support or diagnosing a parser
lookahead bug.