"Real-time Rendering" by Akenine-Tomas Möller et all is one of the standard references here.

https://www.amazon.com/Real-Time-Rendering-Fourth-Tomas-Aken...

Peter Shirley's free Ray Tracing series is incredibly good as well: https://raytracing.github.io/books/RayTracingInOneWeekend.ht...

And the outstanding piece in rendering is the Physically Based Rendering book by Pharr et all (which won an Oscar): https://www.pbrt.org/

None of those are math references as such, but the thing is with graphics it's not based on first principles (well, ray tracing is kinda getting there but still has lots of artistic stuff rather than science stuff), but rather on a bag of tricks. And the math without the "bag of tricks" context may not be that clear.

Plenty, depends if you want books,

https://www.amazon.com/OpenGL-Shading-Language-Cookbook-high...

https://www.amazon.de/Graphics-Shaders-Practice-Michael-Bail...

https://www.amazon.de/Real-Time-Rendering-Fourth-Tomas-Akeni...

https://www.amazon.de/Physically-Based-Rendering-Theory-Impl...

Or grab some online content instead,

http://www.pbr-book.org/

https://developer.nvidia.com/gpugems/GPUGems/gpugems_pref01....

https://developer.nvidia.com/gpugems/GPUGems2/gpugems2_insid...

https://developer.nvidia.com/gpugems/GPUGems3/gpugems3_pref0...

https://developer.download.nvidia.com/CgTutorial/cg_tutorial...

Naturally most of the stuff I linked to, doesn't use necessarily GLSL ES, however it is relatively easy to translate the concepts and ideas into it, given how most shading languages happen to be either a C or C++ based dialect.

I would also add Real-Time Rendering[2] and Real-Time Collision Detection[3] to the list of absolutely essential game development books.

[1]https://www.amazon.com/Engine-Architecture-Third-Jason-Grego... [2]https://www.amazon.com/Real-Time-Rendering-Fourth-Tomas-Aken... [3]https://www.amazon.com/Real-Time-Collision-Detection-Interac...