"Linux 3-D Graphics Programming" 
That explains how to build a 3d engine almost from scratch and it does explain 2D math (cross product, etc) and also 3D stuff of course (matrix operations, perspective projection, etc).
It's a bit dense in that it pretty much builds everything from zero, but it's a very nice way to go through everything needed to work with 3D graphics.
You can check the content table in Amazon I think.
The second book "Advanced Linux 3D Graphics" is a bit less interesting for me since it talks a lot about Blender and other software which it doesn't really go with the theme of the first one, but there's still a lot of good info in there.
Get dozens of book recommendations delivered straight to your inbox every Thursday.