This is a huge mistake. You would never expect something like bitCast to do this.
I don't understand this approach. Why change something so simple and low level to be complicated and high level?
Just don't allow casting to u24, as it makes no sense unless you define u24 to be u32 sized as I think c standard does.
I think this approach as an idea is bad but at least just add another built-in that implements this higher level idea to not break a simple expectation and current behavior?
The reason u32->u24 casting must be well defined is because some hardware (e.g. many GPUs, microcontrollers) only have floating point multipliers. A 24 bit unsigned integer (stored in a 32 bit register) can be losslessly converted to a 32 bit float by the hardware, multiplied, then converted back.
This is much faster than doing 32 bit multiplication in software, however, you still need to tell the compiler about this constraint.
Interpreting u24 like it is actually 24 bits sounds like programming in crazy land since it is not 24 bits in any relevant architecture afaik.
They didn't allow []u24 with a similar rationale as far as I can remember. I agree with this as someone programming at this level should be able to understand there is no real u24 layout and they should use []u32. Going with the same magical rational they went with here, compiler should generate unaligned u24 loading code when you use []u24 since it is "logically 24 bits"
Zig natively supports arbitrary bit-width integers, the ABI is defined and you could simply think it as a slice of the next larger backing integer.
The[3]u8 to u24 bitCast will simply be backed by a 32bit int, using the same ABI. As you have u1 - u65535, sometimes it can be multiple words.
The 24 Bits (3 Bytes) [3]u8 to u24 example is exactly related to utf-8 that covers all the languages but excludes the emojis.
There are very valid use cases when you want to limit utf-8 to U+0000-U+FFFF, and it is valuable if your language allows you to make those decisions.
Remember, in zig packed structs are just integers and integers are just a group of logically consecutive bits.
Arrays like []u24 do not have the same ABI, arrays are not bit/byte packed, are not universally LSB across archs etc..
The compiler isn't producing unaligned code, don't confuse the abstraction with the concrete implementation. And yes [8]u1 and [8]u8 are exactly the same size and shape, even though they are arrays.
My current project is parsing ELF/Macho files, I can easily have zero allocations in my hot path with zig, the same is far more challenging in C, so I am biased, especially with zig allowing methods on structs.
And yes, I do use that crazy casting to 0xdeadbeef and other ascii metadata that is in those files.
To be clear here, I am not trying to prove you wrong, this is one of the places zig is very different and (IMHO) useful. Especially with streaming data or where you have network ordering etc... It is so nice to only cast what you need to but it does take a little while to wrap your head around how this interacts with buffers which are not your native endianness. At least for me, once I figured out to separate the shape of those data streams from their values it was super useful.
Citation please - every single GPU in the literal world supports integer arithmetic for operating on tid, gid, etc.
https://www.amd.com/content/dam/amd/en/documents/instinct-te...
> V_MUL_U32_U24
>,Multiply two unsigned 24-bit integer inputs and store the result as an unsigned 32-bit integer into a vector register. D0.u32 = 32'U(S0.u24) * 32'U(S1.u24)
> Notes
> This opcode is expected to be as efficient as basic single-precision opcodes since it utilizes the single-precision floating point multiplier. See also V_MUL_HI_U32_U24.
Nvidia GPUs used to do the same thing and theres a umul24 intrinsic if you care to use it.
https://stackoverflow.com/questions/5544355/cuda-umul24-func...
This is super-super-niche since it basically only applies to 32-bit integer multiplication.
You likely won't run into it unless you're doing high performance embedded systems or GPU programming on non-NVDIA cards, and for some unknowable reason, your workload does a 32-bit integer multiplication in the hot path.
> it became allowed to use @bitCast to reinterpret a [3]u8 as a u24
This cant't make sense unless u24 is defined to be 24bits in the first place. It is just silly to allow something like this. It would make so much more sense to me if they started disallowing this or just even print a deprecation notice for it for one release version.
> Useful for larger integers than int16_t while saving 25% over a 32-bit value
You can't even do []u24 in zig as far as I can remember and understand anyway so this is only happening in a packed struct context.
C doesn't mandate padding but C compilers allow having pointers and arrays of irregular _BitInt types as far as I can understand.
In this [1] document, in Abi considerations section, it writes that it is defined to have next-power-of-two layout size.
Also here (for RISCV) [2] it seems like it is defined with next-power-of-two layout.
Also the document here (for x86_64) defines it similarly [3]
[1] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2709.pdf
[2] https://github.com/riscv-non-isa/riscv-elf-psabi-doc/issues/...
[3] https://gitlab.com/x86-psABIs/x86-64-ABI/-/tree/master?ref_t...
It's worth remembering that zig is a ~hll that should be platform agnostic. suppose someone built a byte-chip with a 24 bit word. the "new" zig way of doing things will be more portable and slot right in, and support 32- and 16-bit datatypes just fine.
Once you see that the fact somebody has a u24 in their code is between them and the compiler alone.
As others probably noted byte casting (keeping the same endianess) is what unions are for.
Is there at least some sort of @transmute or something ? If Zig wants to say "bitCast" means this odd operation, but provides the thing most people actually want under some plausible name that's just an extra thing to learn which seems OK.
Yeah, if your architecture doesn’t support 24-bit int it maps to 32-bits. But it also declares that the numbers you’re storing should never be larger than 2^24. It’s about type safety, and also run time checks in safe mode I believe. Bitcasting three bytes to a 24-bit type makes just as much sanse as casting 4 bytes to 32-bit. Theres zero reasons to introduce arbitrary artificial constraints on what you can do based on details of (most of) the underlying architectures, which doesn’t even matter for the operation you’re performing.
Of course you could also go crazy and store data in 24 bit blocks in your SRAM. That kind of ruins the 8 bit and 16 bit reads though.
I'm no Zig expert, but if you want endian-dependent semantics I'd assume either @ptrCast or a packed union would do the job.
const MyUnion = packed union {
full: u16,
bytes: [2]u8,
};
const value: u16 = 0x55aa;
const in_union: MyUnion = @bitCast(value);
const without_union: [2]u8 = @bitCast(value);
std.debug.assert(without_union[0] == in_union.bytes[0]);
std.debug.assert(without_union[1] == in_union.bytes[1]);
> Converts a pointer of one type to a pointer of another type. [1]
[1] https://ziglang.org/documentation/master/#toc-ptrCast
So it is not the same.
You could use it to define a function that implements bitCast. Which defeats the purpose of having any @bitCast intrinsic instead of using @mempcy for everything
> You could use it to define a function that implements bitCast. Which defeats the purpose of having any @bitCast intrinsic
Yes, and this is one reason @bitCast was changed to have different semantics that are not trivially achieved with @ptrCast.
I was inclined to agree with you, but what decided it for me is that Zig has another mechanism for "reinterpret bytes". It's exposed on the stdlib as std.mem.asBytes, but this is literally a wrapper for the following:
@ptrCast(@alignCast(ptr));
Also worth noting: packed structs in Zig are already defined as logically little-endian: the first field is of low significance, the second is above that, and so on. So this makes `@bitCast` consistent with an existing convention of treating integers as logically little-ended, without regard to how they're actually arrayed in memory.
Plus it stands to make low-level bit-twiddling, using oddly-sized integers, optimize better. I like that, especially when what we trade for that is: nothing. Nothing at all, this is a pure win.
I'd even guess it's that rare language update which silently fixes buggy code, where someone figured "well, basically everything is little-endian already" (or just didn't think about it), and now that code works properly on big-endian machines.
I think endianness is the footgun that Zig is solving, rather than Zig being the one introducing a footgun when you deal with endianness.
It is not feasible for someone to write endian portable code in a language like Zig without understanding what endianness is imo. Regardless of how they change @bitCast there will be other cases that break this like doing @ptrCast + @memcpy.
Also this breaks currently written code that is endian portable and uses @byteSwap like it is done in most other programming languages that do these things.
> As a general rule, the new semantics tend to match the behavior of the old semantics on little-endian targets.
They've basically said that bit casting is going to be little endian. This simplifies things for the 100% of people that are on little endian machines, while making the code still work for the 0% of people (rounded to the nearest 0.0000001%) that are using big endian machines.
Don't get me wrong, I love Zig and I think it's a great C replacement, but I'm very confused on why C3 or Odin rarely get any attention at all, despite being in the same C-replacement crowd.
But still surprised at what Zig does better than these other projects? Is Andrew much better at marketing/promoting the language? He's very hard to dislike.
Sometimes its just right time, right place. But also, Zig has received attention via projects like Ghostty, TigerBeetle, and Bun (prior to rewrite of course)
C3 uses :: for namespaces, that makes it a competitor with C++ more than C. Equally Odin's syntax is more at home among python, not systems programming.
The appeal of Zig is it feels like C. To many people, this is a downside. C is very very scary to them. But for people who feel at home in C, it's not a downside.
Additionally, the selling point for both are "c replacement" where the selling point of Zig is "good systems programming language" C is only mentioned by it's users as a heuristic.
If 2 groups are trying to replace a language that people are running away from, and that's their best selling point... I'd assume they're less likely to be as successful as a different language just trying to be as good as it can be.
I've even stopped comparing Zig to C, IMO, it does a disservice to both. And I say that as someone who likes C.
Full disclosure, I need to spend a bit more time with both odin and c3 to know exactly how this compares. But the reason I keep writing Zig, and still love it, is how simple it is. Zig is aggressively insistant on simplicity at the expense of functionally or comfort. The only other high level language I know of that is as aggressive about it's design simplicity is infact C. While I assume it's an accident when C does it, it's definitely not an accident in Zig.
Can I do that in C3 or Odin?
Proper enums, proper tagged unions, and often reading the docs can allow the AI to distinguish T * to one of
1. [*]T
2. [:0]T
3. ?T
4. *T
And these are just the most common ones. If you know it’s a read only pointer/array then you can add the const modifier
There is a mountain of code written in C that you can simply include in Zig without a wrapper dependency and without having to create the wrapper yourself.
Doesn't matter as neither will see significant adoption.
His politics don't matter to me. Hell, if the politics of technologists dictated whether I used their products, I'd have to go live in the wilds, without any tech. :-)
Or the Java guys who wrote bloated apps that wasted CPU cycles on garbage collection instead of writing in C++, like God intended?
Or the Fortran / Cobol guys who wrote in those God-damned, wasteful, useless high-level languages, instead of using assembly, like a proper programmer should?
though, to be fair, the middle layer itself is composed of this same work. so it's fractal, or turtles all the way down.
Those who viewed code as a means to build something else, are happy to switch to LLMs if they can build that something faster/cheaper.
Whereas, those who liked coding for its own sake, don't want to use LLMs, and fear for their jobs and their happiness.
Unfortunately for the latter group, we're moving to a world where most development is done by LLMs, and only cutting-edge or hobbyist work is done manually. E.g., Japanese artisanal wood-working and joinery is beautiful and elegant... but modern carpentry doesn't build that way.
Now I get to fix things created by managers who enjoyed building things they still don't understand.
I wonder, these arbitrary-width integers... Is it actually even really worth it? My intuition is to prefer manually packing/unpacking things instead (in any language, even C that has bit width for struct fields), because it gives me a better mental picture of the code that is actually generated. Particularly for something like an signed odd-bit integer - what kind of code gets generated for sign-extension, a presumably common operation?
Does anybody have other experiences with them, one way or the other?
They're situationally useful, especially when performance isn't an enormous concern. That u729 example above came from a variant sudoku solver I wrote to aid developing new puzzles (easy to check the rough magnitude of the solution space for whatever idea I was mulling over and examine how restricted the board actually was -- just an intermediate step in puzzle design). It's not optimal (hard on the icache, can be hard on registers, other issues abound), but it's dead simple to use, and the assembly isn't terrible, beating all the normal solvers I saw floating around. It's a nice point on the laziness/correctness/good-enough-perf pareto curve.
Another comment mentioned this, but they're great in packed structs for representing weird numeric entities (I think I have a logarithmic number system floating around which does that).
One thing the language does quite a lot is use them to guard against certain classes of human error at compile time. It doesn't perfectly make impossible actions unrepresentable, but shoving a full u32 into a shift argument usually doesn't make sense, so the types are constrained to be smaller.
https://ziglang.org/documentation/master/std/#std.bit_set.St...
Sometimes it's just more clear to work with integers than other representations. Most situations with a state space of N bits have meaningful integer representations, where arithmetic functions on those representations are also meaningful.
For example, CRCs can be written as the remainder from long division of the message by the polynomial. Defining nontrivial cyclic permutations is also much more straightforward as functions on integers than on bitsets.
If I had to steel-man the idea, I'm pretty sure the integer-based solution has better codegen with many kinds of sparse, comptime-known masks. I think you're right though, StaticBitSet looks better.
The bus-width is a generic parameter and can be below or above 64 bits (depending on the emulated system). With arbitrary-width integers the high level code remains the same no matter what the bus-width is, and from looking at the compiler output, as long as bit operations don't straddle the underlying 64-bit integer boundary, those bit operations are just as efficient as working on a simple 64-bit int.
Also AFAIK LLVM supports arbitrary-width integers since pretty much forever, Zig just 'exposed' them in the language (as later did Clang via _ExtInt(N), which is now deprecated in favour of C23's _BitInt(N)).
The other nice usage (also in emulators) is for chip registers and counters, those often have odd widths (like 5 bits), and writing those as u5 instead of u8 in the code is just nicer since it matches the chip documentation, and when reading the code it's immediately clear that this u5 is a 5-bit counter or register.
Obviously there are ways around pretty much everything, but it’s nice to have first class language support for bit slices.
I like them, they're nicer than C's bitfields: The order isn't implementation-defined, and the types remember their range rather than being converted to a power-of-two size upon read. (Maybe that's possible with C23 _BitInt(n), I haven't tried if those work in bitfields)
e.g. https://github.com/zml/zml/blob/33ced8fa078b3c7c8c709bd526ae...
Example: shifting more than the width of the shifted integer is illegal behavior in Zig: therefore, the, what, shiftand? let's go with that, the shiftand for a u64 must be a u6 or smaller.
Sounds annoying? No, it's great! Check this out:
const CodeUnitKind = enum(u2) {
low,
high,
follow,
lead,
};
const CodeUnit = packed struct(u8) {
val: u6,
kind: CodeUnitKind,
};
mlugg, please don't apologize for creating something I actually want to read. I'm drowning in low effort garbage, the in depth technical explanation is a refreshing breath of fresh air.
Might as well apologize for creating a language without a garbage collector, sure most people are unwilling to think, but some of us like nice things and are actually willing to apply effort.
Much more important, thanks for not just the devlog, and explaining the changes. But also; thanks for fixing/improving this!
I appreciate all the work you've put in, I really enjoy watching the the language I like constantly improve.
Every one of them is faster than the others too lol! Mold for one tries really hard to be GNU ld and to be useful as an independent linker most have to - I guess Zig/Go ones are purpose built so at least those don't duplicate GNU ld compatibility.
I wouldn’t call that endian-agnostic. It’s explicitly picking little-endian.
It also makes things look weird for beginners. I know how it works, but in the
test "bitcast [2]u3 to @Vector(3, u2)"
in: [300]u8;
out: [800]u3;
out = @bitCast(in);Aside from that: I'm not familiar with how standard base64 deals with endianness, so I'm not sure if it would match that, but this `@bitCast` would certainly give you a base64 encoding. But it would probably emit pretty terrible code to do that---our lowering of `@bitCast` isn't really optimized for moving around huge amounts of data in one operation! (But maybe LLVM would surprise me.)
Of course! I guess it was too early to do the maths correctly... :)
All this to say that Zig just keeps cleaning up and giving well-defined semantics to warts I learned to live with in C.
No jokes aside, these posts are the best advertisements of the language.
And I truly like their AI stance.