undefined | Better HN

0 pointsTheLoneWolfling12y ago0 comments

My "ideal" string would be to store it as a UTF-8 rope, with the additional restriction (doesn't change the interface any) that all characters within a node in the rope have the same length. (You can use overlong encodings internally if it makes sense (One single-byte character in a bunch of longer characters), which is a microoptimization that will in some cases save a few bytes.)

I'd also treat a character + combining characters as a single character.

0 comments

2 comments · 1 top-level

weinzierl12y ago· 1 in thread

>that all characters within a node in the rope have the same length [...] I'd also treat a character + combining characters as a single character.

The problem with this is that Unicode doesn't restrict the number of combining marks. If your hypothetical library wants to offer full Unicode support, your "nodes of same length" idea wouldn't work.

Of course an implementation which makes an arbitrary restriction wouldn't be unusual. In fact, I'm not aware of any application that supports an arbitrary number of combining marks even if the standard allows it.

When it comes to standard conformance UAX15-D3 [1] is probably the closest we could get. It'd require 128 byte per character.

[1] http://unicode.org/reports/tr15/#Stream_Safe_Text_Format

TheLoneWolflingOP12y ago

The length of each node is not the same; each character within a node is the same number of bytes long.

So you end up with one node in the rope that has one logical character - that is some absurd number of bytes long. (In reality there'd be a maximum of 2^8-1 or 2^16-1 or something bytes per character)

j / k navigate · click thread line to collapse