Little Lexer

A tiny very very simple Lexer/Parser class.

Feed it an array of [regex,char] pairs and it scans an input string returning a string of char's matching the tokens found.

Since it outputs a string, it's output can be used as it's input! Thus exactly the same (very simple) class can be used as a lexical scanner and a parser!

What makes this such a very very very cute idea (apart from the fact it so very small) is... Traditionally Lexical scanners such as flex yield a stream of tokens id's (usually an int or an enum) and the token (usually a string). I have chosen (this is the smart bit) to make each token id a character (which limits the number of possible token id's a bit, but certainly not for any practical use), and not return a stream of token id's and tokens, but an array.

Now an array of token id's is just an array of char which is just a string!

WAKE UP! This is VERY cute! Thus the output of the lexical scanner is just a string which can be used as the input to a lexical scanner.

Thus we can use the very same class as both a lexical scanner _AND_ a parser!

And all this in a mere teensy 44 lines of non-blank non-comment lines of Ruby!

I told you it was cute.

See the built in tests at the end of the file for examples.

You can down load it from the project page

Contact