> I may be misunderstanding your argument, but if it's that of a simple offset, then only the one starting from 0 forms a monoid (a group without an inverse to each element). Though, of course, you could redefine the + operation...

Yes, agreed, there is other algebraic structure that can tell the difference, but Peano arithmetic by itself cannot.

> I think I’m missing something here. PA defines x * 0 = 0 for all x. So while we could take (Z+, 1, ++) as a model of it, we would be imposing a completely different definition of multiplication than the usual. Would this not be simply choosing to label 1 as 0 and work from there?

Despite the name, in the usual mathematical meaning of the term, Peano arithmetic does not define arithmetic at all, only the successor operation, and everything else is built from there. Once we have those, for the model (Z_{\ge 0}, 0, ++), we certainly usually do define x0 = 0 for all x; and, you're right, if for the model (Z_{\ge 1}, 1, ++) we defined x1 = 1 for all x (as no-one could stop us from doing), then we'd just be dealing with "0 by another name." But it might be equally sensible, if our model of Peano arithmetic is (Z_{\ge 1}, 1, ++), to define x1 = x for all x, in which case we recover the expected arithmetic.