Amit Singh’s excellent book MacOS Internals: A Systems Approach [1] is a bit outdated now but still recommended if you’re interested in the workings of a non-Linux commercial OS.

[1] https://www.amazon.com/Mac-OS-Internals-Approach-paperback/d...

The UNIX on NeXT and derived OS is just there mostly for compatibility purposes, and not much else.

http://www.nextop.de/NeXTstep_3.3_Developer_Documentation/

https://www.amazon.com/Mac-OS-Internals-Approach-paperback/d...

https://www.amazon.com/Mac-OS-iOS-Internals-Apples/dp/111805...