Sigh, the canonical reference on 3D graphics "in software" has been the Foley and Van Dam tome "Computer Graphics: Principles and Practice in C" [1], followed closely by "Principles of Interactive Computer Graphics" [2] by Newman and Sproul. Followed closely by the Graphics Gems series, and then Watt's eponymous "3D Computer Graphics" [3] and "Real Time Rendering" [4] by Moller and Haines.

All wonderful texts and can tell you everything you want to know about doing 3D graphics in software. They won't help at all (generally) for GPU based graphics sadly.

[1] http://www.amazon.com/Computer-Graphics-Principles-Practice-...

[2] http://www.amazon.com/Principles-Interactive-Computer-Graphi...

[3] http://www.amazon.com/Computer-Graphics-3rd-Alan-Watt/dp/020...

[4] http://www.amazon.com/Real-Time-Rendering-Third-Tomas-Akenin...

Any book on computer graphics would be a good start for the graphics side.

I like 3D Computer Graphics by Alan Watt as an introduction and overview http://www.amazon.co.uk/3D-Computer-Graphics-Alan-Watt/dp/02...

and Real-time Rendering for more advanced subjects http://www.amazon.co.uk/Real-time-Rendering-Tomas-Akenine-Mo...

The session from Assembly 'The Basics of Demo Programming' is online and may also be useful http://www.youtube.com/watch?v=TbcZyAO6K7c or have a look at The Demo Effects Collection http://demo-effects.sourceforge.net/

But it depends what platform you want to use C-64 An Introduction to Programming C-64 Demos http://www.antimon.org/code/Linus/ and http://codebase64.org/doku.php

Processing http://www.processing.org/

WebGL Browserscene: Creating demos on the Web http://www.youtube.com/watch?v=XZLqwXdXjqY

Deconstructing a browserscene demo http://www.youtube.com/watch?v=pOjWOA-iPnA

I have some more bookmarks on pinboard http://pinboard.in/u:z303/t:demoscene/