Could it be that macOS reports GPU memory as main memory on M1? It would have been useful to look at the GPU memory usage on the Intel test.
I don't know much about the M1 graphic stack, but I remember being able to malloc a buffer in the userspace and passing to a Metal kernel as a texture not so long after Metal was announced.
I imagine that on the M1, the driver asks for a chunk of the main memory to the kernel, whereas on Intel the driver asks the GPU to allocate a chunk from the GPU memory.