I don't think its a question of whether there is an essential reality or not, but rather whether we have access to essential reality. Donald Hoffman makes a strong game theoretical argument for how natural selection chooses effective presentations of reality rather than necessarily accurate ones [1]. Based on your level of interest in this conversation I'd expect you would really enjoy that book!

The game is certainly easier than the mind—I like it as an example because most of us have a hands-on knowledge of what the qualitative experience of "playing a game" is like. But the game still only emerges because the game developer, computer manufacturer, and player jointly give it a emanating system into which it can emerge. On its own, the raw game data doesn't really mean anything at all—if the bitstream of Diablo IV washed up on the beach, there's nowhere in that data encoding the experience of killing Diablo for the first time. One wouldn't even recognize it as something that could be decoded into such an experience [2].

I agree with you that the "why" is tough. Why have a conscious experience? Why have a sense of self at all? Why experience emotions rather than have them be—like you described—a purpose-driven internal signal? And then you get into theories like Internal Family Systems, which has empirical support at least within a prescriptive context if not necessarily a descriptive one [3].

The whole thing is a mess. A great, big, beautiful mess.

[1: https://www.amazon.com/Case-Against-Reality-Evolution-Truth/...] [2: https://benjamincongdon.me/blog/2021/02/21/Three-Layers-of-I...]

I think it's more about the approach taken for researching the hard problem of consciousness (what you basically pointed out; what explains the phenomenon of subjective experience for conscious entities?).

Most researchers are doing exactly what you said, looking at it from the angle of "what arrangement of matter" gives rise to consciousness?. This model follows our intuitions and should definitely be explored; I don't think anyone is arguing the contrary.

But there are others, like the cognitive psychologist Donald D. Hoffman, who are doing research from the complete opposite side of this problem: what if consciousness is a fundamental property of the universe? He explains this far better than I can (pre-caffeinated) in his book The Case Against Reality[1], which I'd recommend you read if you're interested in this topic (he's also done quite a few interviews which can be found on youtube et al[2]).

Perhaps similar to you, I (a materialist) used to have what I can only call a "gut reaction" to this idea - panpsychism just seemed like "new age bullshit" that was impossibly wrong. But I'm becoming more open, and honestly the science being done is really fascinating[3].

[1] https://www.amazon.com/Case-Against-Reality-Evolution-Truth/... [2] https://youtu.be/4HFFr0-ybg0 [3] http://cogsci.uci.edu/~ddhoff/Chapter17Hoffman.pdf