Plenty of objects in the browser API contain references to things that could be used to defeat the compartmentalization.
If one were to enumerate all properties on window and document, how many would be objects with a reference back to window, document or some API not on the allowed list?