undefined | Better HN

0 pointstetromino_3y ago0 comments

> It's a toy problem with no real consequences.

It's an extremely simplified but recognizable version of a real problem: natural language input tokenization, which of course has real consequences.

0 comments

5 comments · 4 top-level

bluepizza3y ago· 1 in thread

This problem was solved by medieval monks by introducing white spaces.

tetromino_OP3y ago

Many languages still don't put whitespace between some or all words (Chinese, Japanese). Many languages tend to use long compound words which are better represented as sequences of tokens rather than one token (German, Russian). Many inputs are dirty (coming from OCR, for example) so whitespace can be unreliable. And even in clean English text, inflected words may be better tokenized as sequences of tokens rather than a single token.

glitchc3y ago

The problem there is that natural language processing is a highly specialized field, not something one can expect a generic SRE to know unless it's explicitly mentioned in the posting. I'm willing to bet the posting said nothing about NLP.

I could ask a detailed question on cryptography, robotics or reverse-engineering, and it would be just as (ir)relevant to the SRE. Computer science is a much broader field than your niche area. Arbitrary goal-posts are arbitrary.

reedf13y ago

Ah yes, I whack out four or five natural language input tokenizers before lunch most days.

rimliu3y ago

Then this problem is solved by a gazillion of libraries already, no?

j / k navigate · click thread line to collapse