Mini-rust in Rust 004: Refactoring our lexer

Refactoring our lexer, so all the awful things we did to make the tests pass are behind us. More info: https://artificialworlds.net/blog/2024/11/22/mini-rust-in-rust-004-refactoring-our-lexer/