There's a couple really good books that delve into this complex issue [1], [2]. The TLDR is that early 20th century American cities were quick to level entire neighborhoods to bring in modern infrastructure. It was swift and brutal, wiping out 50 to 100 years of history without much thought. All replaced with boring concrete roads and bland architecture.
So in the 60s and 70s, activists changed city laws to empower local residents with the ability to delay or out-right block housing projects.
What you're seeing now is the result of 40-50 years of cities stuck going through very tedious processes to get anything moved forward.
The problem with empowering any local resident is that you see only the people who want to stop something at the hearings and all it really takes are a hand-full. The rest of the residents are impartial or just too busy to care.
[1] https://www.amazon.com/Neighborhood-Defenders-Participatory-...
[2] https://www.amazon.com/Golden-Gates-Fighting-Housing-America...
Katherine Levine Einstein was on The Weeds podcast a few weeks ago and gave a really good summary of her findings. She doesn't have an answer, but it at least frames the situation in a way that seems solvable.
Boston University's Katherine Levine Einstein explains the dysfunctional politics behind America's housing crisis.
https://www.amazon.com/Neighborhood-Defenders-Participatory-...
"Since the collapse of the housing market in 2008, demand for housing has consistently outpaced supply in many US communities. The failure to construct sufficient housing - especially affordable housing - in desirable communities and neighborhoods comes with significant social, economic, and environmental costs. This book examines how local participatory land use institutions amplify the power of entrenched interests and privileged homeowners. The book draws on sweeping data to examine the dominance of land use politics by 'neighborhood defenders' - individuals who oppose new housing projects far more strongly than their broader communities and who are likely to be privileged on a variety of dimensions. "
