If you just throw everything into a neural network, then you won't really understand the breadth of the problems you're solving, and you'll be therefore ignorant of the limitations of your hammer. While NNs are incredibly useful, I think a deep understanding of the core problems is essential to know how to use NNs effectively in a particular domain.
After getting a grip on those concepts, Szeliski's Computer Vision: Algorithms and Applications (http://szeliski.org/Book/) had some really amazing coverage of CV in practice. Mastering OpenCV (https://www.amazon.com/Mastering-OpenCV-Daniel-Lelis-Baggio/...) was very useful when actually implementing some algorithms.
Good Luck !
Get dozens of book recommendations delivered straight to your inbox every Thursday.