> I _think_ allocating a slice of contiguous bytes and using unsafe pointers should work fine as long as you are very cautious about structs/vars with pointers into the buffer getting freed by the GC

Go's GC is conservative, so I don't think you need to take any special caution in that regard. I would expect that you just need to take care that your casts are correct (e.g., that you aren't casting overlapping regions of memory as distinct objects).