As far as the math is concerned, 3D Math Primer is the book I always recommend starting with:

https://www.amazon.com/Math-Primer-Graphics-Game-Development...

When you say "modern 3D rendering", do you mean modern native APIs like Vulkan, Direct3D 12 and Metal? In that sense, modern GPU programming becomes about scheduling and co-operative buffer management, along with writing programs to be parallel. There's no great guide for this to my knowledge.

If you mean everything above the graphics API, e.g. modern lighting and shading techniques, there is a wide range of choices. PBR-inspired material response models are now dominant, and I recommend Naty Hoffman's slide deck as an excellent overview of the subject [0]. I can go into more details here if wanted. Haines's Real-Time Rendering is a pretty good reference. Get familiar with math, geometry and trigonometry. I recommend "3D Math Primer for Graphics and Games" by Parberry and Dunn [1] if you need a refresher.

[0] https://blog.selfshadow.com/publications/s2015-shading-cours... [1] https://www.amazon.com/Math-Primer-Graphics-Game-Development...