undefined | Better HN

0 pointspdonis6y ago0 comments

> The distinction in Python 3 is between text (str) and bytes.

Feel free to s/Unicode/str/ in what I posted if you prefer that terminology. The problem is still the same.

An example of the problem: Python's standard streams (stdin|out|err) in Python 2 are streams of bytes, but in Python 3 they're streams of Unicode (or str if you prefer that terminology) characters. The problem is twofold: first, if my standard streams are hooked to a console, Python can't always properly detect the encoding of the bytes coming from the console, so it can give me the wrong Unicode characters; second, if my standard streams are hooked to pipes, there is no encoding it can pick that is right, since the bytes aren't even coming from a console (where at least there is some plausible argument for saying the user meant to type Unicode characters, not bytes). What Python 3 should have done was keep the standard streams as bytes, since that's the only common denominator you can rely on, and then let the application decide how to decode them if it decides it needs to, just as in Python 2.

0 comments

2 comments · 1 top-level

takeda6y ago· 1 in thread

I believe the behavior is correct though. Python uses encoding specified through LANG/LC_* which is the encoding that supposed to be used, and all properly behaved applications use it.

If your application works on binary data, you can use sys.stdin/out/err.buffer to get binary version. Most people will use it for text, so the defaults make sense. Personally I would like if there was no automatic conversion when using files/network/pipes etc. but I guess that would make it more confusing for new users, and would be unnecessary boilerplate for most use cases.

pdonisOP6y ago

> Python uses encoding specified through LANG/LC_

Yes, that's the best you can do, but it's still not always correct. I agree that it should be, but "should be" and "is" aren't always the same.

> If your application works on binary data, you can use sys.stdin/out/err.buffer to get binary version.

Yes, but there are still standard library functions that will use the regular streams, and that might conflict with what your application is doing. There is no way to tell Python as a whole "use binary streams everywhere because they are pipes for this application".

> Personally I would like if there was no automatic conversion when using files/network/pipes etc.

That would work if (a) Python could always detect that condition (it can't) and (b) the entire standard library adjusted itself accordingly.

> I guess that would make it more confusing for new users, and would be unnecessary boilerplate for most use cases.

Python 2 worked fine with the standard streams being binary, and applications wrapping them to decode to Unicode when necessary. Python 2.7 even back ported the TextIOWrapper and similar classes to make the wrapping as simple as possible. A similar approach could have been taken in Python 3 (binary streams and a simple wrapper class), but it wasn't.

j / k navigate · click thread line to collapse