Oded Goldreich's "Foundations of Cryptography" books are the rigorous, theoretical oriented books used in many grad-level crypto courses: https://www.wisdom.weizmann.ac.il/~oded/foc-book.html

Neil Koblitz has "A Course in Number Theory and Cryptography", which I haven't read, but is often used: https://www.amazon.com/Course-Number-Cryptography-Graduate-M...

Also, Boneh and Shoup is good and mentioned in other comments: https://toc.cryptobook.us/