Z-order curve usage to decrease dimensionality to 1 (opens in new tab)

Here is a writeup on Google's S2 library for considering addressing the surface of the Earth as 1D, using Hilbert Curves.

http://blog.christianperone.com/2015/08/googles-s2-geometry-...

And 2015 HN thread: https://news.ycombinator.com/item?id=10066616

Usually a Morton ordering is used for things like improving average memory locality of N-dimensional data (e.g., loop iteration order, data layout, ...). But it is average locality that is being improved, because of huge jumps between many neighbors.

In a small number of dimensions, without knowing what search algorithm is being used, this is just more work than comparing the original values. It doesn't mention what "k-NN algorithm" is being used, beyond brute force search.

Lossily compressing N-dimensional data (from 2 to 1000s of dimensions) into a representation that requires fewer bits can be done via quantization as well, either scalar quantization, vector quantization (aka k-means) or product quantization, if your data has known statistics.

It also matters if you are building a static data structure that is queried many times, versus one that needs continual updating.

As the first commenter on the site pointed out, the Hilbert Curve is probably a better choice (https://en.m.wikipedia.org/wiki/Hilbert_curve)

Space-filling curves have been studied as a way to reduce N-dimensional space to 1-D. They are very useful for applications like maximizing sequential access in a N-D datastore. A recent SIGMOD paper analyzed space-filling curves impact on data access: http://dl.acm.org/authorize.cfm?key=N37709

QUILTS: Multidimensional Data Partitioning Framework Based on Query-Aware and Skew-Tolerant Space-Filling Curves Shoji Nishimura (NEC Corporation); Haruo Yokota (Tokyo Institute of Technology)

It discusses C-Curve, Z-Curve, and Hilbert curves.

Here's a great write up on how you can use Z-curves to do multidimensional sorting using redis otherwise 1 dimensional sorted set datastructure. It's one of the best hands on examples on a way to solve real problems using this tech within your stack.

https://redis.io/topics/indexes

Can someone help explaining why this hash method could improve distance calculation for k-NN? What does it improve compared with Geohash or k-d tree structure?

I've used Z-order curves and related curves in Neuroevolution research. It allows conversion of an adjacency matrix to a spatial subtrate representation that preserves locality as it grows (Thus going from 1 dimension to 2 or 3). This technique is an alternative for HyperNEAT or ES-HyperNEAT and experiments demonstrate higher performance that ES-HyperNEAT on the modular retina task problem.

If you're tempted to use space filling curves (z-curves, hilbert curves, etc.) you should take a look at simple multi-dimensional data structures as an alternative.

As a couple of people have mentioned in this thread, space filling curves aren't great at preserving locality (i.e., two points that are "close together" in two dimensional space might end up being "far apart" in one dimensional space, and vice versa). A k-d tree is easy to code up and, in general, will be more efficient for queries like k-NN than dimensionality reduction because it's better at preserving locality.

There are also good libraries for multi-dimensional data structures for pretty much any mainstream language.

AKA a geohash (for the lat/lon example), but can easily be extended to other bounded dimensions.

This stuff mostly goes over my head, but I'd like to understand it. In the original method, why is op concatenating the two positions into one number, rather than just storing the lat/lon pair and using something like Euclidean distance?

As the commentators on the website mentioned, this method does not preserve the closeness of the points (when going from 2D to 1D), so it is unclear how can it help the author.

So how would https://xkcd.com/195/ look using a z-curve instead?