This New Algorithm for Sorting Books or Files Is Close to Perfection (opens in new tab)

Exactly. How much space do I leave around the colors considering recent book colors?

I considered doing that once. When sorted by subject/author you can end up with a mix of tall books and short books on each shelf in a bookcase, which means that the vertical spacing of the shelves has to be enough to accommodate tall books on every shelf. That can waste a lot of space.

Sort by height and you might be able to reduce shelf spacing enough to make room to add a shelf to the bookcase.

Right. Not sorting each one by absolute height, but sorting into "bins" so that you have books of similar height, which then can be subsequently sorted for placement in a shelf of appropriate clearance.

Even more, if your bookcase has adjustable shelves (for example, via dowels), you can compute the optimal shelf arrangement and book placement, totally customized to your books.

And if you want extra credit, also take into consideration the width of each book, so you know when you reach the shelf size.

I do not admit of having spent hours on this many years ago, nor of still having the code solving it somewhere.