undefined | Better HN

0 pointskazinator4y ago0 comments

I don't entirely buy the argument because in a regular ASCII control sequence like ESC[5A, you have the same problem. All those characters have a role outside of the signaling, so if either side is in an unexpected state, they get misinterpreted. This is just the risk of in-band signaling.

Of course, if the terminal is ignorant of UTF-8 then this is a nonstarter, because whenever CIS occurs as a continuation byte, it will be misinterpreted. If the terminal is ignorant of UTF-8, why would you send UTF-8 to it, though? If it's going to be interpreting UTF-8 as some branch of ISO Latin, the display will be a mess.

If the terminal and host do handle UTF-8, then this CSI signaling is just an extension of the state machine. It's also nice and simple that, in the absence of any data loss or synchronization error, the CSI code is unambiguously not part of any valid UTF-8 character (except as the second or subsequent byte where the receiver is in the right state to interpret it that way).

In my experience, it's the terminal->host direction where you get mixups, whereby the terminal generates some escape sequence like for an arrow key, but the host is not in the right state, and interprets part of it as data. This is exacerbated by a situation in which the host supports ESC as a UI command. CSI solves the abmbiguity between the control sequence start and ESC just being ESC.

0 comments

2 comments · 1 top-level

duskwuff4y ago· 1 in thread

The fundamental problem with mixing C1 controls with UTF-8 is that it forces the terminal emulator to break layering. It can't run a UTF-8 decoder first, because that'll turn the C1 controls into replacement characters, and it can't run a terminal sequence decoder first either, because that'll treat a lot of the UTF-8 continuation characters as control sequences. And what you're likely to find if you start using C1 controls is that support for them in terminal emulators is often incomplete and/or buggy. Handling them correctly in conjunction with UTF-8 text is difficult, and many terminals just don't bother.

The ambiguous nature of ESC in the terminal->host direction (as you put it) is unfortunate, but is difficult to fix. Some terminals (like iTerm) can be configured to use C1 controls for function keys, but my experience has been that a lot of software fails to recognize these sequences, making it impractical to use.

kazinatorOP4y ago

The fundamental problem with TCP/IP is that it forces the stack to break layering. In the same frame of bytes, you have a confusing mix of ethernet addressing, IP header, and a payload of application data, all from totally different pieces of software in the system. Even the data itself is fragmented, with some session wrapping around content that are done by different application stacks.

> It can't run a UTF-8 decoder first, because that'll turn the C1 controls into replacement characters, and it can't run a terminal sequence decoder first either because that'll treat a lot of the UTF-8 continuation characters as control sequences.

It has to have a state machine which recognizes the combined language of UTF-8 sequences and control sequences. Which is the approach you would take anyway, even with C0 controls.

That combined language is an unambiguous, regular set, so you could code it with your eyes closed.

Starting in an initial state, the legal inputs are: ASCII character, Unicode character, or escape sequence headed by CSI. This is decidable from reading exactly one byte value with no further lookahead.

That's just one way. You can in fact follow a layered approach whereby the terminal decodes everything with UTF-8 before analyzing it for control or data.

For instance, say we decode UTF-8 into integer code points. A valid character decodes into its implied code point. An invalid byte like CSI can decode into some reserved range like U+DCxx. The higher layer of the terminal's firmware then looks for values in that U+DCXX range: that's where it finds the CSI.

I have years of experience with this exact encoding scheme, which I baked into the text I/O streams of a programming language.

For instance, oh, /proc/self/environ is NUL-separated, right? No problem:

  1> (file-get-string "/proc/self/environ")
  "CLUTTER_IM_MODULE=xim\xDC00XDG_MENU_PREFIX=gnome-\xDC00LANG=en_CA.UTF-8\xDC00;D
  [...]
  -go:@/tmp/.ICE-unix/2065,unix/sun-go:/tmp/.ICE-unix/2065\xDC00GTK_IM_MODULE=ibus
  \xDC00_=/usr/local  /bin/txr\xDC00"

The NULs are rendered into \xDC00 codes. This is called the "pseudo-null" character in the terminology of this language, and has a symbolic name: #\pnul:

  2> #\xDC00
  #\pnul

We can split the data on it to recover the list of environment entries:

  3> (take 4 (spl #\pnul *1))
  ("CLUTTER_IM_MODULE=xim" "XDG_MENU_PREFIX=gnome-" "LANG=en_CA.UTF-8"
   "DISPLAY=:1")

In a similar way, any other invalid byte can be used for framing, if the data in between is all valid.

j / k navigate · click thread line to collapse