I'd suggest one of the 'Learning OpenCV' editions for that. They (or atleast the first edition which is 8 or 9 years old now) cover a lot of OpenCV algorithms and more importantly, the underlying intuition/math. It's the kind of book you need to make time for and study, but will make you feel good when you do :D. There's a 3rd edition [1] now, which I haven't read but has one of the original authors as a co-author and the TOC seems as comprehensive as the 1st edition.

If you develop a taste for CV theory and math, then I suggest Richard Szeliski's [2] book. It's very readable.

Neither book covers any of the latest deep learning stuff. You have to look elsewhere for that.

[1] http://shop.oreilly.com/product/0636920044765.do

[2] http://szeliski.org/Book/

Before this post inspires people to go on a crazy I-can-do-computer-vision-the-hard-way binge and impulse-buy O'Reilly's (excellent) "Learning OpenCV", the second edition is being released sometime this month so you may want to hold off:

http://shop.oreilly.com/product/0636920022497.do