undefined | Better HN

0 pointsfenomas11y ago0 comments

What do you mean by saying that Unicode breaks people's names? The problem isn't that there's anything wrong with Unicode, the problem is that it's not possible to unambiguously convert text from SJIS to unicode and back again, because SJIS has some duplicate mappings for historical reasons (compatibility with different pre-existing encodings as I recall). The same would presumably be true of converting SJIS to any other encoding that didn't have the same duplicated code points.

0 comments

3 comments · 2 top-level

jessaustin11y ago· 1 in thread

Why does Unicode get to decide what "duplicate" means in this context? If Mr. Smythe is addressed as "Mr. Smith" in a letter from his bank, is he going to be mollified by the explanation that Unicode considers his preferred representation to be a duplicate?

fenomasOP11y ago

Unicode didn't decide anything like that, the characters were duplicates per SJIS itself. SJIS intentionally has duplicate mappings in order to be compatible with vendor encodings that put the same character in different places.

Basically, the characters in Unicode are a superset of what you'd find in any single Japanese encoding. The problem is that it mashes many of them together with characters you'd find in C or K encodings.

phkahler11y ago

>> because SJIS has some duplicate mappings for historical reasons

I think that's the problem. Sounds like unicode has some duplicate mapping. Never mind the "historical reasons", it's broken.

j / k navigate · click thread line to collapse