Going more in the computational geometry direction, in addition to some interesting algorithms in the Graphics Gems series:
O'Rourke, "Computational Geometry in C 2e". Deals with the basics in a principled way. Start here for fundamentals like a good algorithm for intersection of two lines, or inside/outside polygon tests (don't depend on garbage blog posts for well studied fundamentals like this). The book's webpage is https://www.science.smith.edu/~jorourke/books/compgeom.html
Ericson, "Real-Time Collision Detection". Deep dive into practical collision detection algorithms.
And for Shaders, check out Inigo Quilez and ShaderToy:
- https://www.youtube.com/@InigoQuilez
- https://iquilezles.org/
- https://www.shadertoy.com/
You should be able to get your hands dirty pretty quickly implementing cool things on ShaderToy.