> But at some point he was confused why many of his buy/sell orders weren't being executed despite being open for several minutes (hours?) his algo would make bids far away from the current spread, anticipating movement

* Did his levels actually get crossed? It could be that he never had the opportunity to get filled at the price he hoped for.

* What was his queue position? Stock exchanges tend to be price time ordered. If others had submitted orders at the same price before he had, they would have priority when getting filled.

* Apropos the prior point, was his broker actually submitting orders when he sent them? Some brokers may avoid showing deep out of the money orders, which could have affected his queue position.

> He finally concluded that some institutional traders must have access to sub-penny ordering, despite it being against regulation.

There's nothing wrong with this. Reg NMS rule 612 permits sub-penny price improvement [0].

[0] https://www.sec.gov/divisions/marketreg/subpenny612faq.htm#q...