RE your questions at the bottom, some good reading materials are:

- https://sre.google/sre-book/monitoring-distributed-systems/

- https://cloud.google.com/architecture/devops/devops-measurem...

- https://www.oreilly.com/library/view/distributed-systems-obs...