ESP32 RGMII (32x) -> PHY (32x RTL8211F) -> Slave switch (6x RTL8367) -> master switch (1x) -> magnetics (for the external port). You’ll probably want a better IC for the master switch so I can’t name one of the top of my head but this would be a relatively simple, if large, PCB.
The hard part I think is verifying that all the PHYs and switches will work correctly without magnetics on a board to board connection.
I am way out of my depth here. I actively avoided analog design in college, and it kind of shows. :-(
There isn’t much analog design here, except the high speed digital signals for which you can just follow some basic rules of thumb (and IC routing guidelines!). You need to length match the busses and correctly route the differential pairs between the PHY and switch. If you know how to use Altium (I assume Kicad has similar features) you can do pin and part swapping with some clever placement to route only a few of the busses as short as possible, then clone them as rooms until you’ve got the number of nodes you need. The last bit will be routing the master switch that connects the slave switches and the external port.
If you’ve never done it before I’d be realistic and try to hit 100mbit first, maybe gigabit interconnection between the switches. The routing you’ll need at those slower speeds will be a lot more forgiving.