If you want to dive deeper into this, Jane Jacobs's "Death and Life of the Great American Cities" is an incredible and very approachable book on urban planning: https://www.amazon.com/Death-Life-Great-American-Cities/dp/0...
If you're generally interested in answers to the questions you raise, you might enjoy reading Jane Jacobs' The Death and Life of Great American Cities. (https://www.amazon.com/Death-Life-Great-American-Cities/dp/0...)
Also read up on all the work Singapore is doing to increase density without ruining their economy or environment while improving livability.
And some further reading on the dynamics of city growth: http://journals.plos.org/plosone/article?id=10.1371/journal....
I love thinking about this stuff, complex and fascinating :)
If you walk down a street in, say, Amsterdam, or down one in St Louis, it's two very different experiences - Amsterdam teeming with life, it feels close, safe and small. You can bike anywhere in 30 minutes; the layout of the city itself encourages pedestrian traffic. The marginal cost of popping into the bakery next to the wine store is 30 seconds, so the streets are lined with small mom & pop stores that care, genuinely care about how their street is doing, that the sidewalk is welcoming..
St Louis is a city replanned for cars - you do not move in St Louis but by automobile. The marginal cost of going to two stores instead of one is at least ten minutes, so you always go to Schnucks and call it a day. Biking from one point to another is a lost cause - I literally saw a woman crush the skull of a bicyclist with her SUV in St Louis just last month, he's dead now because fuck bike lanes.
And Amsterdam is three times as populous as St Louis.
Cannot recommend https://www.amazon.com/Death-Life-Great-American-Cities/dp/0... enough for this - urban planning has a massive impact on our lives.