They're explicitly not political, lesswrong is a website/community and rationality is about trying to think better by being aware of normal cognitive biases and correcting for them. Also trying to make better predictions and understand things better by applying Bayes' theorem when possible to account for new evidence: https://en.wikipedia.org/wiki/Bayes%27_theorem (and being willing to change your mind when the evidence changes).
It's about trying to understand and accept what's true no matter what political tribe it could potentially align with. See: https://www.lesswrong.com/rationality
For more reading about AGI:
Books:
- Superintelligence (I find his writing style somewhat tedious, but this is one of the original sources for a lot of the ideas): https://www.amazon.com/Superintelligence-Dangers-Strategies-...
- Human Compatible: https://www.amazon.com/Human-Compatible-Artificial-Intellige...
- Life 3.0, A lot of the same ideas, but the other extreme of writing style from superintelligence makes it more accessible: https://www.amazon.com/Life-3-0-Being-Artificial-Intelligenc...
Blog Posts:
- https://intelligence.org/2017/10/13/fire-alarm/
- https://www.lesswrong.com/tag/artificial-general-intelligenc...
- https://www.alexirpan.com/2020/08/18/ai-timelines.html
The reason the groups overlap a lot with AGI is that Eliezer Yudkowsky started less wrong and founded MIRI (the machine intelligence research institute). He's also formalized a lot of the thinking around the goal alignment problem and the existential risk of discovering how to create an AGI that can improve itself without first figuring out how to align it to human goals.
For an example of why this is hard: https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden... and probably the most famous example is the paperclip maximizer: https://www.lesswrong.com/tag/paperclip-maximizer
As far as I'm concerned, you sort of have to address the big, tough points in Bostrom's Superintelligence[1], and probably Yudkowsky's List of Lethalities[2]. They have to do with intelligence explosions, with instrumental convergence, and with orthogonality of goals, and all kinds of deceptive behavior that we would expect from advanced AI. Throw in Bostrom's "Vulnerable World" thought experiment for good measure as well[3]. If you're not addressing these ideas, there's no point in debating. Strawmanning "AI will kill us all" out of contexte will indeed sound like wacko fear-mongering.
What surprises me is that everybody's familiar with the "paperclip maximizer" meme, and yet I'm not hearing any equivalently memey-yet-valid rebuttals to it. Maybe I'm missing it. Please point me in the right direction.
Aaronson certainly does not address the core theoretical fears. Instead we get:
> Would your rationale for this pause have applied to basically any nascent technology — the printing press, radio, airplanes, the Internet? “We don’t yet know the implications, but there’s an excellent chance terrible people will misuse this, ergo the only responsible choice is to pause until we’re confident that they won’t”?
We did not have any reason to believe that any of these technologies could lead to an extinction-level event.
> Why six months? Why not six weeks or six years?
Implementation detail.
> When, by your lights, would we ever know that it was safe to resume scaling AI—or at least that the risks of pausing exceeded the risks of scaling? Why won’t the precautionary principle continue for apply forever?
The precautionary principle does continue to apply forever.
On the "risks of scaling": we're hearing over and over that "the genie is out of the bottle," that "there's no turning back," that the "coordination problem of controlling this technology is just too hard."
Weirdly pessimistic and fatalistic for a bunch of "utopic tech bro" types (as Sam Altman semi-ironically described himself on the Lex Fridman podcast, where, incidentally he also failed to rebut Yudkowsky's AI risk arguments directly).[4]
Where's the Silicon Valley entrepreneurial spirit, where's the youthful irrational optimism, when it comes to solving our human coordination problems about how to collectively avoid self-destruction?
There are a finite number of humans and heads of state on earth, and we have to work to get every single one of them in agreement about a non-obvious but existential risk. It's a hard problem. That's what the HN crowd likes, right?
The people opposed to the Future of Life letter (or even the spirit of it) seem to me to be trading one kind of fatalism (about AI doom) for another (about the impossibility of collectively controlling our technology).
We simply must discount the view of anybody (Aaronson included) employed by OpenAI or Facebook AI Research or whose financial/career interests depend on AI progress. No matter how upstanding and responsible they are. Their views are necessarily compromised.
[1] https://www.amazon.com/Superintelligence-Dangers-Strategies-... [2] https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... [3] https://www.ted.com/talks/nick_bostrom_how_civilization_coul... [4] https://youtu.be/L_Guz73e6fw?t=3221