Anthropomorphically, you could say that Perl retreated to its base of support among sysadmins, (Common) Lisp took shelter in big-think research organizations, and Java reinvented itself as New Cobol.
But that doesn't really satisfy either, because it doesn't tell us why nobody tried to similarly re-purpose Lisp. That's especially odd, in fact, because now Everybody Knows that a lisp core is really simple, and Lisp was still on the radar in academia, where people can be rewarded for re-implementing old ideas.
The problem with the AI winter argument is that it's half an answer that satisfies you enough to stop asking questions. If you invoke the collapse of commercial AI to explain Lisp's current state of disfavor, you then have to explain why the collapse of the dotcom bubble didn't have a similar effect on Perl or Java.
But that doesn't really satisfy either, because it doesn't tell us why nobody tried to similarly re-purpose Lisp when DARPA et al lost funding. That's especially odd since Everybody Knows that a lisp core is really simple, and Lisp was still on the radar in academia, where people (called grad students) can be rewarded for re-implementing old ideas.
I think there is a pretty good explanation (with evidence, even!), but it's not a satisfying one in this community: the end of the Cold War caused a major reallocation of scarce resources away from expensive one-off projects for the government, and towards mass-market stuff for consumers and businesses (see below), and Lisp suffered from some initial disadvantages in the new environment:
* Limited availability on PC platforms
* Windows Apps were sexier. The internet was sexier.
* Glut of new CS students while the bubble was inflating were mostly taught in Java. (Many of them are sensitive about arguments that lend further support to the idea that their education has turned out to be less valuable than they thought at the time. That may just be my bias showing, though. I think that Math should be relabeled CS, and that "programming" is for autodidacts and trade-schoolers).
* CS is a new field, and industry tends to dumb down research in new fields for a generation or so after disruptive technical innovations. That way, they get enough of the low hanging fruit that it further disruptive innovation is unlikely and their capital is safer. See below, particularly the chapter on the relationship between GE and MIT before WWII.
Social arguments are hard to make here though, because so many people are so hooked on the "weird and scary and only for elitist pricks" narrative. That's kind of ironic, because high level corporate executives, if they heard the current public stand on lisp -- "it may be really productive, but it's too foreign for most people and I don't feel like learning it" -- could probably force adoption from above, especially now that cost-cutting is the order of the day.
Fresh book recommendations delivered straight to your inbox every Thursday.