My advice for people interested in graphics would be to jump in and get started with a tutorial on the web and then get a book or two to learn more. My recommendations would be:
Introduction To Ray Tracing - It focuses on ray tracing, but many of the topics (vectors, matrices, shading, view transforms, etc.) are applicable regardless of the actual rendering method, and it does a good job explaining all of it.
https://www.amazon.com/Introduction-Tracing-Kaufmann-Compute...
Real Time Rendering - This also covers all the math, but focuses on rendering at a higher level and covers more ground.
https://www.amazon.com/Real-Time-Rendering-Third-Tomas-Akeni...
Finally, Computer Graphics: Principles and Practice - This is a great general reference for more specific topics. It's like Real Time Rendering expanded to cover more topics and going into more depth. Not a beginner reference, but great to have when you need more information on something.
https://www.amazon.com/Computer-Graphics-Principles-Practice...
Edit: I'd also add: I wouldn't bother learning much OpenGL/WebGL to begin with (except shader programming in GLSL, since there's no good alternative abstraction for that). If you end up liking working with 3D graphics, go back and learn some about it since it'll help you understand performance concerns better—but meanwhile, knowing it is just an optimization you don't need yet. It's true three.js is built on top of it, but the significant principles you need to use three.js effectively fall under real-time rendering, not OpenGL.