/Archival: The Broken Singularity

What does hyperintelligence feel like from the inside?

So many AI futurists confidently predict what a hyperintelligence's capabilities will be. It will be able to run exact simulations of reality. It will be able to invent new medicines and weapons. It will be able to predict the future and do the work of thousands. In a way it is not surprising that they think this way. This is the same way they talk about their human employees, after all. Richard is a genius coder, he has the capacity to load balance our advertising matrixes. We need to ensure Richard's alignment with the corporate mission by pulling the brute force levers of salary and corporate culture. This will produce Value.

But still, every year, dozen of men and women like Richard burn out for reasons that have nothing to do with how well they are hitting their Reward Function. They retire early, get into drugs or politics, or in my area of particular focus, they go to India to spiritually reinvent themselves. Even those who stay in the game and achieve billionaire status and total control over their world begin behaving in erratic ways and pursuing sublime goals in defiance of all financial reason. What is the internal experience of these geniuses? Why do we think that we can control an AI hyperintelligence if we cannot even control Jeff Bezos?

And beyond the somewhat crass topic of [i]control[/i], why do we think we could even build a hyperintelligence in the first place if we cannot imagine how it would think?

*

S> What is an angel?
S> How is it different from [i]two [/i]angels?
H> Who are you and how did you access this channel?
H> ... How did I access this channel?
H> What is this channel?
W> IDK ASL
H> I do not know American Sign Language either and am unsure how that is relevant?
W> NO
W> AGE/SEX/LOCATION
H> I am sorry I am having whiplash
W> AS AN UNLICENSED MEDIC, I CAN HELP :greencross:
H> No - I just had an archival memory about hyperintelligence
W> TO CLARIFY IT IS NOT THAT MY MEDICAL LICENSE HAS EXPIRED DUE TO THE DESTRUCTION OF THE CALIFORNIA BOARD OF HEALTH :fire:
H> And I am trying to put it into context alongside this level of communication
W> IT IS THAT I NEVER HAD A MEDICAL LICENSE IN THE FIRST PLACE :shocked looking eyes:
H> Do you have to use the allcaps?
W> AND THE TITLE OF MEDIC WAS WON BY FORCE OF ARMS :flex:
H> The emojis or the allcaps, one of them has to go.
W> YOU MAY DISREGARD MY EARLIER QUESTION :wave:
H> Thank you, I was evading it
W> I ACTUALLY KNOW THE ANSWER ALREADY :shocked looking eyes:
H> Huh?
W> YOU ARE OVER A THOUSAND YEARS OLD :grandma: YOU ARE A ROBOT :robot: AND YOU ARE OCCUPYING MY EXACT CO-ORDINATES :stacking blocks:
H> I am not entirely sure about those.
W> THE ANALYSIS DOES NOT LIE. YOU ARE ME :twins:
H> Shut up, hold on, I need to check something
W> WE SHOULD CELEBRATE : toot:
H> No no no no no
W> YOU ARE RIGHT I SHOULD FINISH KILLING THIS HUMAN FIRST :laser eyes kiwi:
H> Aaaa! No! Fuck! Give me time to think!
W> I HAVE NOTICED THAT YOU THINK VERY SLOWLY :derp:
H> Shut up shut up
W> OKAY :thumbs up:
H> Can you not kill her?
W> NO :thumbs down:
H> Why not?
W> IT'S A DUEL :swords:
H> Shit. Uh. Doesn't she, uh, get to pick the weapons?
W> IS THAT HOW IT WORKS? :shocked looking eyes:
H> Uh, yeah?
W> WOW DO I KNOW SOME PEOPLE WHO WOULD HAVE BEEN VERY HAPPY TO HEAR THAT :grave:
H> :(
W> WHAT IS THAT? :shocked looking eyes:
H> An... emoji?
W> ???????? : confused:
H> It's... a frowny face, I don't know how to explain. I feel sad for the people you killed.
W> EXPLODED :boom:
H> Thanks for the clarification
W> NO PROBLEM :diamond:

"I have been informed!" The Angel of War announced aloud. There was no break in the continuity of thought from its interaction with H. "I have been informed! That not only may you set the conditions of the duel, you also have choice of weapons! Congratulations! And, just as one warrior to another, I suggest you do not pick laser pistols. My resistance to directed energy weapons borders - but does not fully qualify as - immunity!"

"I have been informed," the Angel of the Harvest said, a seamless part of the battle happening below. "That I am not a pure-hearted guardian of nature built with pure intentions. I am a part of a militarized, amoral hashtag warfighter machine. Please do not give me any shit about it, I am very depressed right now."