I kind of agree with you, but then, since it would fit for integers, I can’t think of a better name and could live with some generalization of “off by one”. I’m not a mathematician but maybe something like “off by n; n -> lim 0”
I’m not sure. Since it is related to the distinction in mathematics between open and closed intervals (except on computers we have floats in instead of reals), maybe “interval error” or “endpoint error”. But not “bounds error”, of course.
EDIT: Or maybe “comparison error”. I like that the best. But calling it an off-by-one seems wrong, especially as what we usually mean by that is something more like not handling the last iteration of a loop correctly.
Amoung my friends when we used to do ACM programming competitions we called it “Off by equals error” since it was actually a fairly common issue in those types of programs and it was useful to have a name for it.