This isn't mysterious or hard, and I've written comments about it before (https://news.ycombinator.com/item?id=7073079). There are two ways to lower the cost of something: increase the supply or decrease demand. It's quite easy to build more units on a given parcel of land but SF has mostly chosen not to do so (http://www.theatlanticcities.com/housing/2013/10/san-francis...).
The real problem starts with voters, since they're the ones electing the mayor, city council, and other officials, who respond to voter preferences against building stuff but who also deal with voters complaining about high rents. They're in a situation similar to the ones Bryan Caplan describes in The Myth of the Rational Voter, which is a fascinating book that deserves not just to be read but reread (http://www.amazon.com/The-Myth-Rational-Voter-Democracies/dp...).
Even if polling shows a lack of discontent, it does not mean they were voting rationally. The emotional fear machine of terrorism will always sway towards totalitarianism policies. But if the majority of those citizens were honestly questioned about having their phones (and their entire families) monitored, I doubt they would be for it.
Irrationality and logical fallacies are flourishing on both ends of the spectrum (citizens and politicians), and there are few vocal voices in between correcting the bullshit.
What's missing are leaders (aka media) evaluating and promoting policies based on rationality reasoning rather than what emotional ploys that sell.
[1] http://www.amazon.com/Myth-Rational-Voter-Democracies-ebook/...
We have politicians who have misunderstand and politicise economics. Similar to how evolution is somehow now a liberal theory.
New Keynesian economics is not liberal or conservative. It says, in hyper-simplified form, save (conservative) in good times, and spend (liberal) in bad. Right-wing Central Europe is one of the few in recent memory to have implemented both legs. Politically conservative trade theory, about which there is consensus amongst economists, earned left-wing Paul Krugman his Nobel Prize.
Myth of the Rational Voter [1] explores issues economists have put to rest but the public refuses to accept there is consensus on. Similar to how the masses insist on perceiving "hackers".
[1] http://www.amazon.com/Myth-Rational-Voter-Democracies-Polici...
http://www.amazon.com/Myth-Rational-Voter-Democracies-Polici...
Other than that, I defer to the "Democracy is two wolves and one sheep deciding what to eat for dinner" quote.
Perhaps we let too many people vote already? The average American isn't just a little uninformed, but is provably wrong about a number of issues, like what the government spends its money on, who holds the power to do various things, and the resources and governments of our neighbors. Heck, I'd bet that a non-zero percentage of voters don't even understand how the marginal tax rate works.
We don't let unqualified people fix our cars, work in our hospitals, or teach our kids, so why should they have the ability to make our political decisions?
Society has plenty of ways to accredit people to be 'knowledgeable' about these subjects (Basic civics test -> high school diploma -> college degree -> Degree in econ/poli sci -> masters -> phd, or perhaps business owners, workers in gov't service, etc).
We already treat voting as a privilege, not a right, (It can be taken away, as with felons) why not weight votes based on your investment in learning??
Obviously this plan is harebrained in that it would massively disadvantage everyone other than upper class white people, and probably lead to a civil war, but it's interesting to think that voting is probably the one place in society where are an objectively recognized expert in the field (I'm thinking more on the Econ side than the Poli Sci side) has exactly the same say as someone who just picks one at random.
The whole idea that some privileged group has the ability to say whether I can vote makes me a little uneasy, so personally I think I'd give a firm no to the idea.
If you're interested in the idea, the economist Bryan Caplan wrote a book called 'The Myth of the Rational Voter' that discusses the common fallacies and provides hard evidence. http://www.amazon.com/dp/0691138737?tag=bryacaplwebp-20&...;
He also has the single ugliest web page that i've ever seen for an academic (a high bar indeed): http://www.bcaplan.com/
Pigouvian taxes on high-carbon fuels are ridiculously unpopular. The median voter (http://www.amazon.com/Myth-Rational-Voter-Democracies-Polici...) does not perceive the TCO of cars, the fuel supply chain, and parking spaces (http://www.amazon.com/High-Cost-Free-Parking-Updated/dp/1932...). So CAFE standards and other kinda crappy, second- and third-best workarounds get used instead.