For relativity, there is the venerable General Relativity for Mathematicians.

https://www.amazon.com/General-Relativity-Mathematicians-Gra...

> A theoretical physicist friend likes to say, only partly in jest, that there should be books titled "______ for Mathematicians", where _____ is something generally believed to be difficult (quantum chemistry, general relativity, securities pricing, formal epistemology)

At least one of these does exist:

https://www.amazon.com/General-Relativity-Mathematicians-Gra...

Given how the author seems to be most familiar with geometry by the kind of examples that she or he gives, I'm surrpised that they're not already familiarw with this book.