A now-adrift Torgensen visits an “open house” at an AI-focused think tank and gets into a discussion about existential risk.
• • •
“Torgensen,” the doorman repeated, flipping slowly through papers on a clipboard, then stopping on a name. “Peter?”
“Yes.”
“Gotcha.” He nodded. “Go on in.”
Torgensen would have expected something more futuristic to check the guest list for an open house (if an “open house” that required you to register and be accepted truly qualified as “open”) at the Institute for Humanity’s Future, but he’d also have expected the volunteer manning the door to look somewhat more…professional. Of course, he’d have expected the Institute to look a little less like a converted garage in a light industrial park, but here it was anyway. And this was Silicon Valley, where “professional” was, at best, a state of mind rather than a dress code. He was glad he’d forgone the tie.
Inside, the money showed more clearly, albeit in a haphazard, taste-free fashion. The walls had been freshly painted off-white, except for the one done in an “accent” bright blue bearing a huge IHF logo—an infinity symbol set against the letters “IHF” in Futura Bold—carved from bamboo and softly backlit. The floor remained concrete, but polished to the point it could pass for marble at first glance. It was the furniture that—well, he didn’t know what it did, but it sure as hell did something. It was if someone had walked into the nearest Design Within Reach showroom and just pointed at random objects until they had enough to fill up a moving van. Everything had that vague “modern by way of 1960” look to it, with no attempt at coordinating style or color. Torgensen had never been someone who cared about such things; he’d bought his entire living room at IKEA. But he’d at least known to buy from the same lines, so everything more or less fit together. Maybe this was a style he was previously unaware of.
While the furnishings were odd, the crowd looked very much like the crowd at any other semi-professional social gathering: clusters of three or four, usually with drinks in hand, staying together for a few minutes before someone moves to another cluster, with a handful of loners wandering from group to group or hovering around the snack table. Some were dressed like the volunteer doorman; more were dressed closer to Torgensen, but definitely no ties in sight. He didn’t recognize anyone, but on second thought, why would he?
He shuffled toward the snack table himself, nodding to people who seemed to glance in his direction, and put together a small cheese and cracker plate with a glass of biodynamic chardonnay.
Glancing around, he picked one of the groups at random, moving toward them and listening in long enough to determine the conversation had something to do with applying Bitcoin to “green capitalism.” He moved on to a slightly larger, more animated group.
“It’s just too pessimistic a view,” one woman was saying. She was middle-aged and slightly rumpled, with frizzy gray hair and a rumpled business casual outfit. Academic, he guessed.
“It’s realistic,” the man she appeared to be addressing countered. He looked like Carl Sagan cosplaying as Steve Jobs, minus either one’s smile.
“No, it’s amoral,” a third man chimed in, waving a glass of bourbon emphatically enough to make the single spherical clear ice cube clink loudly. This one looked enough like Walter Lucas, down to the hoodie and the day-old stubble, to make Torgensen visibly twitch. “Cutting off AGI at the kneecaps because we’re afraid of some sort of sci-fi doomsday scenario cuts off all the radical improvements it would bring to future generations. It’s the only way we’ll advance on the Kardashev scale. We can’t just—”
Not Carl Sagan rolled his eyes. “A pause to re-assess where we are isn’t cutting anything off permanently. You know how many potential ways there are for artificial superintelligence to go off the rails if it advances too fast, and right now, we don’t even know what the guardrails are.”
“We’re not nearly close enough for that kind of worry, though,” the woman said. She was the first to take note of Torgensen’s hovering, and nodded to him as she continued speaking. “We can bristle all we want when someone describes the newest OpenAI model as ‘spicy autocomplete,’ but we also all know that there’s a fundamental difference between deriving relations between words and actually understanding them.”
“LLMs alone may not get there—”
“They will,” Hoodie Guy insisted.
Torgensen made a soft, skeptical grunt, sipping his drink.
Hoodie Guy abruptly turned to him. “I take it you agree with Mr. Jablonski.”
Torgensen blinked, clearing his throat. That was a familiar name, but he couldn’t place it. “I have my doubts that large language models will blossom into artificial general intelligence on their own, yes. If they are used as front-ends to a…system capable of symbolic logic and reasoning, it might be another matter.”
“Like Prolog?” the woman said.
Not Carl Sagan—Jablonski—gave Torgensen a measured look, lips curling into a slight smile. “Or like mice.”
“Mice?” Hoodie Guy sounded incredulous. “Wait, you mean like those ones in the news?”
“I do.”
Torgensen furrowed his brow. “Have, ah, have we met before, Mr. Jablonski?”
“No, we haven’t. But the Institute has newcomers fill out those interest forms for a reason. You’re Peter Torgensen, aren’t you? Of Nanolab Sciences.”
“Ah, no longer of Nanolab Sciences.” He cleared his throat and took another sip of wine. “I fear the CEO thought I had too pessimistic a view.”
“You thought mixing mice and LLMs would lead to actual AGI.” Hoodie Guy’s tone shifted to more skeptically patronizing than incredulous. “And unsafe AGI, to boot.”
“I don’t know if I would use the term ‘artificial,’ since there is biology involved, but…” He gave his characteristic tiny shrug. “The nanotechnology gave the mice accelerated capabilities, both physical and mental. The language models not only gave the now-enhanced mice the ability to use language themselves, but access to all the information encoded in the models. And, while a mouse’s brain may not be sophisticated on its own, the ways it learns and reasons nonetheless cannot be reproduced by large language models.”
“Not yet,” the woman said.
Torgensen lifted his brows and shook his head. “Not ever. That is, not without…” He waved his glass, slightly rather than emphatically. “Introducing another system.”
“And you think lab mice are a capable enough system.” Jablonski arched a brow, but looked curious rather than skeptical.
“You have seen the news stories of the Dynamice, hmm? The interview with one?”
“Yes, but I’ve seen news stories with reporters who’ve become convinced LLMs are trying to seduce them.” He laughed. “And hype from AI companies proclaiming that LLMs would deceive them during testing to increase their chance of self-preservation.”
“Which it might have been,” interjected Hoodie Guy. “You can’t write those off as ‘hallucinations’; they’re too complex.”
Jablonski shook his head. “You absolutely can. Extremely complex behavior can arise from statistical modeling.” He turned back to Torgensen. “As, I would think, could also be true for that surprisingly photogenic mouse.”
“It could, but…let me think on how to explain a moment.” Torgensen finished the rest of the wine, set down the empty glass on a nearby overpriced end table, and tapped his chin. Aurora, Adonis, and the other Dynamice were clearly not just LLMs; a few minutes with any of them, and the difference would be clear to even the most casual of observers. But saying that would be tantamount to saying just trust me. Not at all satisfactory. “Ah. Yes. It is…initiative. Volition. As excellent as a language model may be at generating complex text and exhibiting complex behaviors, it will never initiate a conversation with you when doing so would be unexpected. It will never change the subject, never say no, never appear to have a mind of its own without being given the context to do so—a previous instruction, a prompt to respond argumentatively.”
“And the mice will?” Hoodie Guy said, tilting his head. He still seemed skeptical, but now seemed interested.
“Aurora—the photogenic one, as Mr. Jablonski put it, although they are all somewhat alarmingly photogenic—escaped our laboratory of her own volition, transforming herself from a small mouse to a humanoid one in the process, and found an IT employee who had previously shown the mice kindness and begged for help.”
Everyone stared at him. The woman looked distinctly uncomfortable.
“Transformed?” Jablonski repeated. “With…with nanotechnology. You’re saying they’re capable of that.”
“Why the hell would you build them to do that?” Hoodie Guy burst out.
“We didn’t,” Torgensen responded matter-of-factly. “They discovered it on their own. As they discovered…many things, although I am not sure how much I can say without violating the non-disclosure agreement.” He lifted his brows, looking away. “I suspect I am already on thin ice.”
“That’s incredible.” Jablonski finished his wine. “I’d never have considered the possibility, but…at least on the surface, it makes sense.”
“But why would that lead to unsafe AGI? Or whatever you’d like to call it. Semi-artificial.” Hoodie Guy spread his hands. “The mice aren’t superintelligent, right? Just human-level.”
“Individually, it is…hard to say. They may be capable of being much more than any of us. One of them made that claim, in a very… I can only call it a very apocalyptic phrasing. We cannot verify it, of course, but even so. In aggregate, though, rather than individually…” He shook his head. “I think it would take only a handful to approach the singularity that Mr. Kurzweil hypothesized about some years back.”
“And that would be unsafe because…?” Hoodie Guy persisted.
“How do we treat lab animals, Dave?” the woman said, faintly exasperated. “Think about the kind of AGI that the Institute anticipates within the next twenty years, but starting with the premise that humans routinely torture and kill them to test fucking cosmetics.”
Dave tilted his head to one side, freezing for several seconds, then finally said, “Oh, shit.”
“Oh, shit, indeed,” Torgensen murmured.
After a few moments, Dave shrugged. “But there’s got to be some way to convince them that we’re fundamentally good, right? The benefits of having actual AGI—whatever you call it—in our world are so incredible, the risk of not having it is worse.”
“We only get those benefits if the AGI is under our control,” the woman countered. “Do you really want to risk all civilization being brought down by mice, of all things?”
“That’s melodramatic of you, Marilyn.”
“If the mice are anything like what Mister, ah,” she stumbled for a moment, “Torgensen is suggesting, I’m not sure that it is.”
“If they get that far out of control, you nuke them.” Dave said that in a tone that suggested he thought it was the most reasonable thing in the world.
“Oh, now who’s being dramatic.” She turned to Torgensen. “Dave sometimes lets his accelerationist tendencies get the best of him.”
“Accelerationist?”
“Getting to an advanced stage of civilization as fast as possible. We seem advanced now, but we’re barely a level zero on the Kardashev scale. That’s—”
“I’m familiar with it.” It was a scale proposed back in the 1960s, measuring technological advancement based on energy consumption, with level one civilizations able to harness all the energy available on a planet, and level two able to harness a star’s energy. “I admit it always struck me as very…science fictional.”
“It is now. And it’s going to remain that way until we get AGI. If we don’t get it right on the first try, we just try again. As fast as possible.”
“Even if ‘not right on the first try’ leads to deploying thermonuclear weapons.”
Dave shrugged. “Civilization can recover from a nuclear apocalypse, but not an AGI one, where we all get turned into paperclips. Or, I guess with your mice, cheese.” He laughed.
“Ha,” Torgensen said, without returning a smile. It was the only way to keep from blanching at the recalled image of the mouse biting off the lab tech’s head.
Jablonski tapped the side of his glass, looking thoughtful, then said to Torgensen, “Can we chat alone for a few minutes?”
Torgensen lifted his brows and nodded. He expected to be led somewhere else, but instead the other man just glanced between the other two. They both nodded, too, and stepped away, whispering with one another.
“Something made you decide to come to this open house, Mr. Torgensen, and I’m guessing it has to do with whatever led you to separate from Nanolab Sciences. Something to do with Dynamice.”
Torgensen brooded. He should probably have gotten more wine. He should probably excuse himself.
“I wished to learn,” he finally said, “what researchers at an institute that concerns itself with the risks and benefits of artificial general intelligence would think of what Nanolab Sciences has created.”
“Two questions. First, do you think AI—not the mice, but the path the rest of the industry is on—will inevitably lead to either an apocalypse or an astonishing new future?”
“I do not.” He folded his hands behind his back. “I doubt that path leads to AGI, and I think much of the industry is engaging in, at best, wishful thinking and at worst, coldly calculated marketing. As a tool, generative AI has the potential to accelerate society in both good and bad ways, but it is only an existential risk if we make a long series of very poor decisions.”
“Hmm.” Jablonski tapped the side of his glass again. “Second, the same question, but this time about the Dynamice.”
Torgensen remained silent for several seconds, staring at the floor, and cleared his throat. “They are an existential risk if we make just one or two poor decisions.”
“But perhaps not, if there is an organization to counter those decisions. To preempt them.”
Torgensen looked up. Jablonski was looking toward the IHF logo.
“Perhaps not.”