This is one of those boring books that will never be a NY bestseller, while containing some timeless gems of wisdom worth multiple NY-bestseller shallow books each: https://www.amazon.ca/Managing-Design-Factory-Donald-Reinert...

For example — how you build a feedback loop and why it's more important than writing elaborate guides, docs and business processes.

Or how you give a simple yet meaningful metrics to your tests and learning process.

I was a project manager on web development efforts at IBM when I read that book and "Managing the Design Factory"[0] around the same time. The combination of the two completely changed how I approached managing projects generally.

[0]http://www.amazon.com/Managing-Design-Factory-Donald-Reinert...

There is a large body of theory, evidence and even mathematical proof that supports why agile methodologies, such as Scrum or Kanban can improve the effectiveness of teams in terms of throughput/productivity. Systems, information, decision, risk and queuing theory when correctly applied to product design and development can yield transformative results, in a similar way that companies like Toyota transformed manufacturing process using similar principles to optimise for just-in-time production vs economies of scale.

Unfortunately, you are unlikely to find such information in most of the literature around Agile methodologies, blogs or the army of consultants that roam the earth much less your average Scrum Master. Cargo-culting is rife in most the companies I have seen, even where the adoption of agile principles and methodologies have a significant positive effect on a team's effectiveness. I'd have to classify myself as a cargo-culter in the beginning too when I first exposed to working in Agile team taking an eXtreme Programming approach some years ago.

Unfortunately, herein lies the problem. For many people it does work (whether you like it or not) and as you say, it takes on the characteristics of a religion - people experience "better" and are just happy to keep following a recipe for "success", without understanding WHY. Obviously, this leads to people trying to replicate such "success" in a different context and when things go wrong they don't know how to fix it - they don't have a deep enough understanding of how to re-apply the principles and and normally they are optimising for the WRONG thing.

There are valid proven theories behind why working in iterative cycles, limiting work-in-process, shipping working code in the production regularly and self-organising teams lead to better results - you just won't hear why from most of the Agile evangelists in this world but there are answers out there if you look for them. Instead your typical Scrum Masters obsess about far more trivial and unimportant things like how to phrase the title of a story card, retrospective formats, burn down charts and velocity points.

If you can be bothered to look a bit deeper, then just avoid using the "A" word in your searches. ;-)

Here is one book, if you can get past the rather dry and academic natural to get you started. http://www.amazon.com/Managing-Design-Factory-Donald-Reinert...

P.S. If you want a really good laugh, do a bit of research on how "Waterfall" came to be and how ridiculous the whole Waterfall vs Agile thing really is.