It isn't as if there is an actual line connecting two dots across space.
It's just, there's a set of "points" (there doesn't need to be any things about these "points" other than that they are distinct. You could think of them as urelements I guess.), and another set of "edges", which are each a set of two elements from the first set.
A pair of "points" are "connected" by an "edge" iff the set comprised of those two "points" is in the second set.
This doesn't need a notion of space to work, unless you need like, an idea of space to have collections of things. So unless "the collection of odd numbers" needs an idea of space, then graphs don't need space.