Lexer state

Lexer state cannot be avoided when implementing a lexer for Perl and Ruby.

In Perl/Ruby, regular expression starts and ends with '/', for example:

/perl|ruby/

As we all know, '/' is the divide operator.

So when the lexer meets a '/' character, it can not tell whether it is the start of a regular expression or a divide operator without the help of the context.

Same thing for "<<", it can be the left shift operator, or the start of "here doc":

<<EOF
whatever
EOF