I would like to second jeffool's sentiment. I've gone down this path a bit (without really succeeding), and although it's cool to imagine what it's like to do all that really low level manipulation, it takes a REALLY long time to get anything satisfying out of it. Just doing basic 3D collision detection in a simple scene isn't trivial unless you have experience. Physics simulation is a whole beast on it's own, but pretty necessary to make a fun 3D type game.

Anyway, I would suggest picking a small area to fiddle with at a low level, and come up with a small project as a goal for learning it. Maybe just AI at first... text graphics, but create agents that wander in a virtual world. Or maybe just start with a basic 3D scene and try getting lighting going. Think about what seems the coolest to you, then just try to work that little part out and build a proof of concept.

I might try the Open GL books to get started with graphics. I would look around for books like these: http://www.amazon.com/Game-Engine-Architecture-Jason-Gregory... (although I haven't read that one, so I can't vouch for it's quality).

http://www.amazon.com/Game-Engine-Architecture-Jason-Gregory...

http://www.amazon.com/Game-Coding-Complete-Third-McShaffry/d...

Those two will clean up most of the biggest newbie mistakes, especially the Gregory one.

I also suggest this book for collision detection:

http://www.amazon.com/Real-Time-Collision-Detection-Interact...

However, I haven't found the book that can adequately explain integrating collision into the rest of the game. It's a tricky subject which I'm still learning and might write about someday.