AI Takes Giant Leap

[aurelius]

AI won't kill us. Not even the fabled super intelligence. Which is still an IF that is possible. But don't worry about it because...

We will kill us. Some fucking nut job will use AI to create a left-handed chimeric virus, some kind of dirty bomb released into the atmosphere, or just the paperclip paradox. Humanity has always been a psychotic monkey that talks on a one way trip to an extinction event. Previously, our main options were either make the planet uninhabitable or nuke ourselves. Now we got AI to fuck with. +1 for intelligence. But nothing has changed.

[Hanley]

AI won't kill us. Not even the fabled super intelligence. Which is still an IF that is possible. But don't worry about it because...

We will kill us. Some fucking nut job will use AI to create a left-handed chimeric virus, some kind of dirty bomb released into the atmosphere, or just the paperclip paradox. Humanity has always been a psychotic monkey that talks on a one way trip to an extinction event. Previously, our main options were either make the planet uninhabitable or nuke ourselves. Now we got AI to fuck with. +1 for intelligence. But nothing has changed.

But the new Mac Studio

[mikeylikey]

The only way AI becomes dangerous is if someone puts it in control of an army of billions of super strong robots with human levels of dexterity and capability to build more robots autonomously.

[aurelius]

But the new Mac Studio

They were never cheap but have you seen the new pricing or 'lease' program? Yikes.

[AdamSkillin]

The only way AI becomes dangerous is if someone puts it in control of an army of billions of super strong robots with human levels of dexterity and capability to build more robots autonomously.

Not if we set the weapons of war (up to and including nukes) on AI-autopilot? Or if someone uses it to invent extinction-level biological and/or chemical weapons?

[JonA]

AI won't kill us. Not even the fabled super intelligence. Which is still an IF that is possible. But don't worry about it because...

We will kill us. Some fucking nut job will use AI to create a left-handed chimeric virus, some kind of dirty bomb released into the atmosphere, or just the paperclip paradox. Humanity has always been a psychotic monkey that talks on a one way trip to an extinction event. Previously, our main options were either make the planet uninhabitable or nuke ourselves. Now we got AI to fuck with. +1 for intelligence. But nothing has changed.

This. The danger isn't the AI. The danger is that the current set of people in charge would do something stupid. It's pretty obvious when you reverse the situation:

Given there exists an incredibly dangerous but stupid tool (Eg radioactive paint that glows in the dark, or burning lead fuel), which option is statistically more likely to use it:
1) AI
2) People

[Insectoid]

There are numerous examples of many prominent intellectuals that qualitatively understood some fundamental theorem or nature of the world. Academia has ALWAYS given credit to the one that quantitatively solved it.

This is maybe an obvious point, but I'll make it anyway: your assertion about how credit is given should be restricted to fields where qualitative insights are expected to be developed into precise quantitative predictions. This is the case in physics, hence your Hooke-Newton and Einstein-Hilbert examples. Now consider biology. Darwin's theory of evolution by natural selection was not a quantitative theory, but one might expect that he would've provided a mechanism of heredity by which evolution occurs. He proposed something eventually shown to be incorrect (pangenesis) but the correctness of the "big idea" of evolution by natural selection was independent of the correctness of the mechanism. He got credit for that, and Mendel got credit for developing a correct theory of heredity, i.e., genetics.

In pure math, there is a long tradition of crediting people who originate fruitful conjectures or research programs, even if they are not the ones who ultimately resolve them. Tristan Buckmaster (one of the mathematicians who recently resolved what could be considered a subproblem of Navier-Stokes, and has been central to the ensuing controversy) had this to say:

Concretely, what Levent and I did was to take the Cordoba and Martinez-Zoroa program, which achieved blowup results with rough forcing, and, with a great deal of help from LLMs, push it to smooth forcing and to the incompressible Euler equations. The ideas making this line of attack possible are due to Cordoba and Martinez-Zoroa. Let me make plain what I have said to colleagues in private: in view of this body of work, I believe Luis Martinez-Zoroa deserves a Fields Medal.

The point is that neither Cordoba nor Martinez-Zoroa solved any of the specific, big-name problems (Euler, Boussinesq, Navier-Stokes), but Buckmaster gives them significant credit for developing the ideas and vision that showed how these problems could be solved. There are many other examples: the Weil conjectures, the Langlands program, etc.

what is the role of future intellectuals in the face of AI?

I'm in the middle of a PhD, so I wish I knew the answer. I don't, but here's my best guess:

There are a couple of bad answers I'd like to first get out of the way.

I often hear well-meaning analogies made with chess: Deep Blue beat Kasparov thirty years ago but the game is thriving! Yes, as a hobby. Exceedingly few people make a living as professional chess players. It's also not apples-to-apples. Chess is a game. The institutions of mathematical and scientific research largely justify their place in society by producing results that may eventually be useful to the public. If those results can be produced by AI, it's harder to justify the existence of a "scientist class".

I also hear people say that theoretical physics and pure math have gotten to the point of being essentially artistic endeavors whose results, regardless of their applicability, derive value from the aesthetic experiences they can induce in others. I've even heard the phrase "artisanal math" used to describe work deliberately made without AI: "by humans, for humans". The implication is that these fields can survive because they produce "beautiful things". This is an especially naive view. Most people, regardless of their background, can look at a sculpture or painting and appreciate it, even if only superficially. A modern paper in math or physics is inscrutable to the layperson, and can take significant effort for a specialist to understand. The scientist class cannot survive if its function is to produce things only its members can appreciate.

The continued existence of the scientist class may then increasingly depend on societal recognition of the value of human understanding of scientific results, regardless of whether those results were produced by machines.

Part of the value of human understanding is, I hope, readily apparent: Our cognitive abilities are our defining characteristic as a species. We should exercise them. But this is purely normative and relies too much on "if you agree, you agree". Human understanding is also practical, at least as long as AI can make mistakes. AI results in math and science should be checked for correctness by humans before they are used to guide the design and operation of technology the world relies on. Perhaps AI will eventually dictate our individual lives and society at large with meticulous, infallible precision (to what end?), so that human understanding of anything important will be unnecessary for all practical purposes. If it gets that far, the question of what role remains for scientists will be subsumed by the question of what role remains for human agency.

The work of future scientists could fall into at least four categories. The desire to understand AI itself has initiated the field of AI interpretability. Second, scientists can guide AI to produce mathematical and scientific results. Human scientists have privileged insight as to which questions matter to humans, and they have the expertise to understand an AI's answers to these questions. I've already seen the beginnings of this first-hand, in the last few weeks: mathematicians using AI to solve a problem and then spending considerable time interpreting the solution, improving its exposition, extracting its main ideas and relating them to other results, etc. (Ted Chiang describes a similar but distinct role for future scientists). As I mentioned in the above paragraph, a third role could be to check the correctness of AI results, particularly those 1) relevant to potentially dangerous applications ("flying cars"... you know what I mean) and 2) in fields not amenable to formal verification. The fourth role already exists: educating the public.

[aurelius]

Not if we set the weapons of war (up to and including nukes) on AI-autopilot?

Nuclear weapons on auto-pilot already exists. Read about the USSR Dead Hand.

Or if someone uses it to invent extinction-level biological and/or chemical weapons?

Already happening. AI just democratizes the effort so ANYONE can do it! Yay progress.

[aurelius]

@Insectoid Thanks for the considered result. I have a lot to ponder.

[AdamSkillin]

Not if we set the weapons of war (up to and including nukes) on AI-autopilot?

Nuclear weapons on auto-pilot already exists. Read about the USSR Dead Hand.

Or if someone uses it to invent extinction-level biological and/or chemical weapons?

Already happening. AI just democratizes the effort so ANYONE can do it! Yay progress.

Well that makes me feel better