I think the RSA algorithm is fascinating, just because of the impact it has had on the industry. Imagine a world without public-key encryption. Crazy right? (Obviously there would be other ways to achieve public-key encryption if RSA had never come along... but you get my point).

While we're on the subject: This book is a great read: http://www.amazon.com/Nine-Algorithms-That-Changed-Future/dp...

I'm going to get Think Complexity from Allen B. Downey.

http://www.amazon.com/Think-Complexity-Science-Computational...

Or:

http://www.amazon.com/Nine-Algorithms-That-Changed-Future-eb...

Seconding Dream Machine. Three more:

The Information: A History, A Theory, A Flood http://www.amazon.com/The-Information-History-Theory-Flood/d...

Turing's Cathedral: The Origins of the Digital Universe http://www.amazon.com/Turings-Cathedral-Origins-Digital-Univ...

Nine Algorithms That Changed the Future: The Ingenious Ideas That Drive Today's Computers http://www.amazon.com/Nine-Algorithms-That-Changed-Future/dp...