My advice for people interested in graphics would be to jump in and get started with a tutorial on the web and then get a book or two to learn more. My recommendations would be:
Introduction To Ray Tracing - It focuses on ray tracing, but many of the topics (vectors, matrices, shading, view transforms, etc.) are applicable regardless of the actual rendering method, and it does a good job explaining all of it.
https://www.amazon.com/Introduction-Tracing-Kaufmann-Compute...
Real Time Rendering - This also covers all the math, but focuses on rendering at a higher level and covers more ground.
https://www.amazon.com/Real-Time-Rendering-Third-Tomas-Akeni...
Finally, Computer Graphics: Principles and Practice - This is a great general reference for more specific topics. It's like Real Time Rendering expanded to cover more topics and going into more depth. Not a beginner reference, but great to have when you need more information on something.
https://www.amazon.com/Computer-Graphics-Principles-Practice...
Also, if people are looking for an even bigger list of graphics books the Real-Time rendering blog has a comprehensive list:
All wonderful texts and can tell you everything you want to know about doing 3D graphics in software. They won't help at all (generally) for GPU based graphics sadly.
[1] http://www.amazon.com/Computer-Graphics-Principles-Practice-...
[2] http://www.amazon.com/Principles-Interactive-Computer-Graphi...
[3] http://www.amazon.com/Computer-Graphics-3rd-Alan-Watt/dp/020...
[4] http://www.amazon.com/Real-Time-Rendering-Third-Tomas-Akenin...
If you're looking for something lower level, the book you probably want is the white book - Computer Graphics: Principles and Practice. I have the 2nd edition from 1992, which is the still the standard intro graphics textbook for many CS departments. Though Amazon says there will be a 3rd edition coming out at the end of this year!
http://www.amazon.com/Computer-Graphics-Principles-Practice-...
Another highly recommended book (also recommended in another comment here) is Real-Time Rendering, but I've only used bits and pieces from this one, so I don't know how good it is for folks just starting out. Still probably one you'll want to add to your shelf if you continue on in the field.
http://www.amazon.com/Real-Time-Rendering-Tomas-MOller/dp/15...
Oh, and also also, head over to YouTube with some snacks and a drink, sit down, and watch the weekly Overgrowth game developer videos from Wolfire Games. It's both inspirational to see what other people are doing, and a great demo of concepts that you'll read about in GEA, such as animation blending: http://www.youtube.com/playlist?list=PLA17B3FAA1DA374F3&...
Edit: I'd also add: I wouldn't bother learning much OpenGL/WebGL to begin with (except shader programming in GLSL, since there's no good alternative abstraction for that). If you end up liking working with 3D graphics, go back and learn some about it since it'll help you understand performance concerns better—but meanwhile, knowing it is just an optimization you don't need yet. It's true three.js is built on top of it, but the significant principles you need to use three.js effectively fall under real-time rendering, not OpenGL.