You do need the buffer to never underrun though, no?
That having been said, if this works reliably, it's fantastic. The ESP32 is pretty great, but controlling WS2812b LEDs from a Pi Zero would be extremely convenient as well.
It only matters if the underrun is in the middle of a message, and the messages are short enough to fit in the buffer whole. I'm sure there are already libraries for driving WS2812b from Pi, such as from Adafruit. Certainly the WS2801 library works.