Since the above isn't finished I'll toss out a recommendation. I snagged this a book[0] on a whim from a university library. I'm usually more of a skimmer when it comes to textbooks but I accidentally read it cover to cover. Really helped me wrap my mind around the basic math behind modern cryptography.

[0]: https://www.amazon.com/Mathematical-Cryptology-Computer-Scie...