I've not read this myself but I've seen The Audio Programming Book[0] cited frequently in "Where do I start with audio programming?" questions.

[0]https://www.amazon.com/Audio-Programming-Book-MIT-Press/dp/0...