You would need to position the clothes rod not so close to the wall. There is another common design of closets that this design omits - adjustable depth of rod, via repositioning of the rod flanges

https://www.lowes.com/pd/Project-Source-1-5-16-in-Flange-Set...