I would highly recommend getting this for the shelf: http://www.amazon.com/Analysis-Regression-Multilevel-Hierarc...
It's one of the most readable books on data analysis I've come across and does a great job presenting both frequentist and Bayesian techniques with tons of R sample code.
There are a lot of advantages and nice things in Python, but I do think folks tend to toss out R a bit too casually. Each tool has areas they excel in. I don't even do particularly complex analysis, but have run into areas where Python is woefully lacking in fairly common (social science) models.
I'm surprise nobody's mentioned Knuth, though maybe that goes without saying?
As for probability and statistics, I haven't really found anything (at an advanced) level that I've been happy with. Maybe it's because my background is in statistics, so it's a perception bias (I see the flaws more easily than with other subjects), but I think that most statistics textbooks are pretty rotten.
There are really only two that I'd recommend, and only one at a high level. Gelman & Hill is a great introduction to computational statistics at a high level, while still very readable (and enjoyable!)
http://www.amazon.com/Analysis-Regression-Multilevel-Hierarc...
Other than that, the only truly stellar statistics textbook I've ever seen was the one I used in my intro class in high school. It's sad, but it's a very true comment about the current state of most statistics textbooks (that I can find).
Morgan and Winship's Counterfactuals and Causal Inference: Methods and Principles for Social Research [1] is also really good. Be sure to get the second edition; it's much better than the first.
[0] http://www.amazon.com/Analysis-Regression-Multilevel-Hierarc...
[1] http://www.amazon.com/Counterfactuals-Causal-Inference-Princ...