I can't really recommend it, since half (or more) of it is about Doron Swade navigating the politics of a London Museum in order to get the thing built. But there is an interesting history of Babbage.
[1] http://www.amazon.com/The-Difference-Engine-Charles-Computer...
Others have listed some great, entertaining reads already:
Hackers,
Soul Of A New Machine (which won a Pulitzer),
Cringley's PBS series Triumph Of The Nerds (available on YouTube),
Where Wizards Stay Up Late
Some not mentioned so far (as I write):
The ancient, online Jargon File is a large glossary that captures a lot of early computer subculture through its lexicon. Eric S. Raymond maintains it today, but it originated way back in the 1970s: http://www.catb.org/jargon/html/
"American Experience," on PBS, did a stellar documentary on the origins of Silicon Valley and the pervasive startup mentality there. It's all about the rise of the semiconductor industry, starting with transistors. Watch online: http://www.pbs.org/wgbh/americanexperience/films/silicon/
Dropping LSD was, it turns out, crucial to the origins of personal computing! This I learned from Jaron Lanier and Kevin Kelly, who recommended John Markoff's What The Dormouse Said: http://www.amazon.com/What-Dormouse-Said-Personal-Computer-e...
The Difference Engine: Charles Babbage and the Quest to Build the First Computer is a short book but also a fun read. Doron Swade, technology historian and assistant director of London's Science Museum, races to build a copy of Charles Babbage's "difference engine" before the anniversary of said machine; he tells his travails in building it while giving Charles Babbage's story at the same time: http://www.amazon.com/Difference-Engine-Charles-Babbage-Comp...
No one has mentioned books covering the dark side of hacking. There are some great reads out there, and infosec is a crucial part of computer history.
CYBERPUNK: Outlaws and Hackers on the Computer Frontier covers Kevin Mitnick, the Chaos Computer Club, and Robert Tappin Morris (who, somewhat inadvertently, wrote the first Internet worm). Mitnick disputes his section of the book, but it's fascinating nonetheless. Worth it for the Morris part alone: http://www.amazon.com/CYBERPUNK-Outlaws-Hackers-Computer-Fro...
The Cuckoo's Egg: Tracking a Spy Through the Maze of Computer Espionage by Clifford Stoll is a fun read. Stoll is an astronomer by trade, and his analytical thinking can be an inspiration: http://www.amazon.com/The-Cuckoos-Egg-Tracking-Espionage/dp/...
The Watchman is a true crime thriller you won't be able to put down. The author set out to write a book on Mitnick but wound up detouring to do a story on Kevin Poulsen, who is now an excellent infosec writer at Wired. You will not believe what Poulsen does in this book. http://www.amazon.com/Watchman-Twisted-Crimes-Serial-Poulsen...
The Hacker Crackdown by acclaimed sci fi author Bruce Sterling is a great work on an infamous cross-country bust of many hackers. You'll get a look into the BBS subculture, Phrack Magazine, and the phreaker scene. http://www.amazon.com/Hacker-Crackdown-Disorder-Electronic-F...?
And let's not forget gaming:
Masters of Doom: How Two Guys Created an Empire and Transformed Pop Culture gives a great history of ID Software and the origins of the FPS: http://www.amazon.com/Masters-Doom-Created-Transformed-Cultu...
http://www.amazon.com/Difference-Engine-Charles-Babbage-Comp...
https://www.amazon.com/Difference-Engine-Charles-Babbage-Com...
(unfortunately overshadowed in Google Search results by a William Gibson book of the same name)
It gives a lot of color on Babbage, but yes the conclusion was that Babbage design basically worked, and could have been built. There were errors in his drawings that they had to correct, but nothing fundamental.
The group at the Science Museum spent over 6 years doing this! This is the group that holds most of his papers, drafts, and unfinished machines.
Although there are a couple things I want to follow up on. They weren't that specific about what computation they did. And does it still work today? It was extraordinarily finicky. It produced a lot of bit errors, as did mechanical computing devices that came later, which sort of defeated the purpose (it was supposed to calculate tables of logarithms and such with higher accuracy than humans.)