That would be reasonable to do even if you don't know the internals very well. The index types (called "index access methods") are well-isolated (API-wise) from most of the rest of the code. Technically, you could even add a new index access method as an extension without even touching the core.

The exception to both of those things is the WAL recovery code, which can't be written as an extension and also requires more knowledge of the rest of the system.

But the WAL recovery code is not too bad for those that know it. So if you did the rest, then I wouldn't be surprised if someone jumped in to write it or help you write it.

"A minor feature I'd like to see is to allow ORDER BY on UPDATE statements to avoid deadlocks."

One solution here is to do a "SELECT ... FOR UPDATE" first. Unfortunately, that is redundant (and does redundant work).