Years (eons?) ago I bought something similar. Two book guide through math which I still find useful to this day:

this: http://www.amazon.com/Mathematics-Physical-Sciences-Robert-L...

and this: http://www.amazon.com/Further-Mathematics-Physical-Sciences-...

Vector, Matrix, Affine transformations, quaternions - this can all be managed in a week or two for solid understanding, even if one's math skill is at advanced arithmetic (pre algebra) level, since it is Linear algebra after all. It really is not that hard, and of utter importance in graphics.

No excuses: http://www.khanacademy.org

There is a book Math for game programmers or something, I forgot, but these are the basic tools you have as a graphics programmer. I'd argue you need to expand knowledge into physics if you want to be a good graphics programmer (optics), and without some calculus, namely integration, forget about advanced graphics programming. You could do it, but you won't understand it - you'll end up waiting for others to code examples and publish non-scientific papers in order to understand what you need to do. Thus, you will always lag behind cutting edge and won't be able to read published papers and implement/expand them.

I have kind of a bad memory, so I have these two books I find very helpful when I want to refresh my pre-calculus knowledge: http://www.amazon.co.uk/Mathematics-Physical-Sciences-Robert... and http://www.amazon.co.uk/Further-Mathematics-Physical-Science...