Of course herbie is already good step here, but it is still somewhat niche.
The obvious thing would be making sure that it's easy to get a unambiguous, natural representation of what a float value actually is. (Similar to what C printf %a specifiers produce, rather than something like Ryu float-to-decimal, which is useful for what a float value means for human-readable purposes.)
0X1.999999999999AP-4
+0X1.999999999999AP-3
=0X1.3333333333334P-2
≠0X1.3333333333333P-2
is ugly (and could be improved[0]), but far more understandable (and less superstition-inspiring) than: 0.1 + 0.2 = 0.30000000000000004 ≠ 0.3
0: At the very least, please make it easy to get "0xA.BCDEFp+1", with capital "ABCDEF" and lowercase "xp", so the digits look like digits and the punctuation looks like punctuation.[1] https://solvespace.com/index.pl
There are hundreds of numerical algorithms in there, and we have some bugs that might be related to this kind of implementation error.
But Herbgrind is a heavyweight dynamic binary analysis using the Valgrind framework, so the slowdown it imposes on the runtime might be prohibitive for debugging some programs which take a while to compute their results.