As you get to small image sizes, the relative size of the quantization table and the Huffman encoding table becomes significant. You can easily just pick a standard version of these, leave them out of the image (which is no longer a valid JPEG), and then put them in at the destination to make a valid JPEG again.

I'm not finding a source, but I think I remember the tables being in the 1-2kB range.

Cheap MJPEG USB cameras do this. Their streaming data is like JPEG but without the tables since it would take up too much bandwidth.