I was not a big Amiga user, but on the Atari ST we had a graphics abstraction API (VDI) from day one. Had the notion of a virtual device and drawing surface. All done through a proper syscall TRAP system, too. I am given to understand that the AmigaOS syscall API was in the form of JMPs.
So when the Falcon and TT Etc came along with full 16-bit 256 colour SVGA-like graphics, anything properly written GUI "just worked"
Games and the like, yes, had to fall back to a video compat mode.