If you're looking to find out how a physics engine works you should check out Box2D, it's an open source 2D physics engine. https://github.com/erincatto/Box2D
Game Engine Architecture(non affiliated)
http://www.amazon.com/Game-Engine-Architecture-Jason-Gregory...
It's extremely readable for an 800 page textbook (I actually read it cover to cover), and covers a wide range of topics including basic software best practices and challenges unique to game development, introductory 3D math, rendering, animation, collisions / physics, and basic gameplay. The author speaks from a wealth of experience from having built several real game engines in the industry.
[0] http://www.amazon.com/Game-Engine-Architecture-Jason-Gregory...
http://www.amazon.com/Game-Engine-Architecture-Jason-Gregory...
Anyway, I would suggest picking a small area to fiddle with at a low level, and come up with a small project as a goal for learning it. Maybe just AI at first... text graphics, but create agents that wander in a virtual world. Or maybe just start with a basic 3D scene and try getting lighting going. Think about what seems the coolest to you, then just try to work that little part out and build a proof of concept.
I might try the Open GL books to get started with graphics. I would look around for books like these: http://www.amazon.com/Game-Engine-Architecture-Jason-Gregory... (although I haven't read that one, so I can't vouch for it's quality).
http://www.amazon.com/Game-Coding-Complete-Third-McShaffry/d...
Those two will clean up most of the biggest newbie mistakes, especially the Gregory one.
I also suggest this book for collision detection:
http://www.amazon.com/Real-Time-Collision-Detection-Interact...
However, I haven't found the book that can adequately explain integrating collision into the rest of the game. It's a tricky subject which I'm still learning and might write about someday.
[1] https://www.amazon.com/Advanced-Animation-Rendering-Techniqu...
[2] https://www.amazon.com/Game-Engine-Architecture-Jason-Gregor...