undefined | Better HN

0 pointsCyph0n3y ago0 comments

My bad - not English, but ASCII. The assumed max tweet size is in bytes rather than (UTF) characters.

0 comments

I specifically assumed a max tweet size based on the maximum number of UTF-8 bytes a tweet can contain (560), with a link to an analysis of that, and discussion of how you could optimize for the common case of tweets that contain way fewer UTF-8 bytes than that. Everything in my post assumes unicode.

Tepix3y ago

Did you consider URLs? They don't seem to count against tue size and can be very large indeed (like 4k)

modeless3y ago

URLs are shortened and the shortened size counts against the tweet size. The URL shortener could be a totally separate service that the core service never interacts with at all. Though I think in real twitter URLs may be partially expanded before tweets are sent to clients, so if you wanted to maintain that then the core service would need to interact with the URL shortener.

Cyph0nOP3y ago

Thanks for clarifying. I missed the max vs. average analysis because I was focused on the text. Still, as noted in the Rust code comment, the sample implementation doesn’t handle longer tweets.

KingLancelot3y ago

UTF-8 is 1-4 bytes per codepoint dude, not 1-2.

280 * 4 = 1120, not 560.

Dylan168073y ago

Twitter doesn't allow 280 of any codepoint. The more complicated ones count double against your limit.

1 more reply

Dylan168073y ago

That size in bytes is based on the max size in UTF-8 and UTF-16. Codepoints below U+1100 are counted as one "character" by twitter and will need at most 2 bytes. Codepoints above it are counted as two "characters" by twitter and will need at most 4 bytes. Therefore 560 bytes, and it supports all languages.

Side note, this is more pessimistic than it needs to be, if you're willing to transcode. The larger codepoints fit into 20-21 bits, and the smaller ones fit into 12-13 bits.

Cyph0nOP3y ago

I was referring to the comment in the Rust code, not the analysis.

j / k navigate · click thread line to collapse

0 comments

trishume3y ago

Tepix3y ago

Did you consider URLs? They don't seem to count against tue size and can be very large indeed (like 4k)

modeless3y ago

Cyph0nOP3y ago

Thanks for clarifying. I missed the max vs. average analysis because I was focused on the text. Still, as noted in the Rust code comment, the sample implementation doesn’t handle longer tweets.

KingLancelot3y ago

UTF-8 is 1-4 bytes per codepoint dude, not 1-2.

280 * 4 = 1120, not 560.

Dylan168073y ago

Twitter doesn't allow 280 of any codepoint. The more complicated ones count double against your limit.

1 more reply

Dylan168073y ago

Side note, this is more pessimistic than it needs to be, if you're willing to transcode. The larger codepoints fit into 20-21 bits, and the smaller ones fit into 12-13 bits.

Cyph0nOP3y ago

I was referring to the comment in the Rust code, not the analysis.

j / k navigate · click thread line to collapse