You're not asking about anything about computer vision, if you're asking just about Caffe and Tensorflow. You're not even asking about the whole of machine learning: you're just asking about neural networks.
If you haven't done significant mathematical stuff by yourself (like, do all the problems in a math book by yourself without anyone else telling you what to do), go get a degree. MS in CS is probably best, those are surprisingly easy to get into.
Note that all of these will deal nearly exclusively with the mathematics. Otherwise, you are following tutorials and wouldn't need any actual knowledge to do things. Some of the mathematics is a year old: some of the mathematics is 50 years old, some of it has been around since Gauss.
If you haven't done significant mathematical stuff by yourself (like, do all the problems in a math book by yourself without anyone else telling you what to do), go get a degree. MS in CS is probably best, those are surprisingly easy to get into.
Otherwise, Haykin's Neural Networks and Learning Machines is best in my opinion (http://www.amazon.com/Neural-Networks-Learning-Machines-3rd/...). Bengio's Deep Learning book is the most current by far (http://www.deeplearningbook.org/). Otherwise, you read papers.
Note that all of these will deal nearly exclusively with the mathematics. Otherwise, you are following tutorials and wouldn't need any actual knowledge to do things. Some of the mathematics is a year old: some of the mathematics is 50 years old, some of it has been around since Gauss.