I'm not actually sure. In retrospect, I'm not sure what ‘DCT without quantizing’ really means, since the output of the cosines are probably real numbers? I guess the interpretation would be quantized to however many steps to reproduce the original result when inverted (and rounded).
In lossless JPEG it seems they omitted the DCT primarily for this reason: It not being a lossless operation to begin with, if you actually want to store the result. What other lossless codecs often do is store a lossy version such as that produced by a DCT, alongside a compressed residual stream coding the difference (error).
In either case, it's important to note the distinction between reordering and compressing; reordering tricks like DCT can reorder entropy without affecting the number of bits required to store them, but the simple fact of having reordered data can make the resultant stream much easier to predict.
For example, compare an input signal like this one:
FF 00 FF 01 FF 02 FF 03 FF 04 ...
By applying a reordering transformation to move all of the low and high bytes together, you turn it into
FF FF FF FF FF .. 00 01 02 03 04 ..
which is much more easily compressed. As for whether that's the case for (some suitable definition of) lossless DCT, I'm not sure.