Presumably, the tokenizer doesn't know anything about the meaning of the tokens it's extracting. It's the parser's job to attach meaning to tokens.