undefined | Better HN

0 pointsSohcahtoa826y ago0 comments

> Both rust and go decided to go with Unicode support that is largely based around utf-8 with free transmutations to bytes and almost free conversions from bytes. This could have been Python 3 but that path was dismissed without a lot of evidence.

Do you mean that if you have bytes, but you want to send them to a function that expects a string, then it would automatically interpret the bytes as UTF-8?

If so, that violates the "Explicit is better than implicit" part of the Zen of Python. Encoding/Decoding bytes to/from strings shouldn't happen automatically because doing so means you have to make an assumption about the encoding.

0 comments

3 comments · 1 top-level

masklinn6y ago· 2 in thread

> Do you mean that if you have bytes, but you want to send them to a function that expects a string, then it would automatically interpret the bytes as UTF-8?

No, the types are separate and not implicitly converted P2-style, however "unicode strings" are guaranteed to be proper UTF8 so encoding to UTF-8 is completely free, and decoding from UTF8 just requires validating.

Python's maintainers rejected this approach because "it doesn't provide non-amortised O(1) access to codepoints", and while Python 3 broke a lot of things they sadly refused to break this one completely useless thing, only to have to come up with PEP 393 a few years later.

Sohcahtoa82OP6y ago

Ah, that makes sense. Thank you for the clarification.

Lev1a6y ago

To add to your earlier dialog partner, here are the doc pages for the relevant Rust functions/methods, embedded with runnable examples:

https://doc.rust-lang.org/std/string/struct.String.html#meth...

https://doc.rust-lang.org/std/primitive.str.html#method.as_b...

https://doc.rust-lang.org/std/str/fn.from_utf8.html

Also, as explained in those docs, if and when you are absolutely sure that the Vec or slice of bytes is valid UTF-8, you could use the following "unsafe" methods to not incur the overhead of validation (warnings in the docs):

https://doc.rust-lang.org/std/string/struct.String.html#meth...

https://doc.rust-lang.org/std/str/fn.from_utf8_unchecked.htm...

j / k navigate · click thread line to collapse

0 comments

3 comments · 1 top-level

masklinn6y ago· 2 in thread

> Do you mean that if you have bytes, but you want to send them to a function that expects a string, then it would automatically interpret the bytes as UTF-8?

Sohcahtoa82OP6y ago

Ah, that makes sense. Thank you for the clarification.

Lev1a6y ago

To add to your earlier dialog partner, here are the doc pages for the relevant Rust functions/methods, embedded with runnable examples:

https://doc.rust-lang.org/std/string/struct.String.html#meth...

https://doc.rust-lang.org/std/primitive.str.html#method.as_b...

https://doc.rust-lang.org/std/str/fn.from_utf8.html

https://doc.rust-lang.org/std/string/struct.String.html#meth...

https://doc.rust-lang.org/std/str/fn.from_utf8_unchecked.htm...

j / k navigate · click thread line to collapse