The Sweetest Lesson

  • by David Spuler, Ph.D.

The Sweetest Lesson

Is it the bitter lesson? Sure, the idea is to brute-force a lot more GPUs in 100 trillion weight models. There’s going to be plenty of computation power needed over the next few years as we advance the state-of-the-art models in both training and inference.

But the bitter lesson actually has two factors, of which brute-force hardware is only one part. Let’s see if they both fit:

    1. Brute-force computation, and

    2. Algorithms that differ.

Can you see the mismatch? There needs to not only be brute-force computations, but also the abandonment of human-based reasoning in favor of simpler number crunching. At first, this also seems like a match, because those tricky reasoning algorithms will probably be gone. It’ll just be GPUs and matrices and lots of number crunching.

But, then, inspiration.

Maybe we’re not going to use a fancy reasoning algorithm on top of all those LLMs. The path to intelligence may not be a better “controller” that manages all of those low-level computations. So, the most advanced AI algorithms won’t be using human-style reasoning. But, you know wnat, it’s kind of weird, because here’s the thing:

    We already are!

There’s this whole theory of neural networks that forms the basis of AI theory. For decades, it was a kind of backwater in computing research, where the results were not very impressive. Gradually, it grew in importance as the results from Machine Learning models, which is old-school AI theory, started to be used in real-world activities, like recommending which movies you should watch. Eventually, the advent of hyperscale GPUs led to the GPT series of LLMs, and the rest you already know.

Hence, this is not the bitter lesson. The whole of AI theory in the computing industry is about neural networks, which means this:

    Copying the human brain.

It doesn’t stop there at copying the basic human architecture. The AI industry is effectively copying all the workarounds and extensions that we use to make humans smarter:

  • Tools — e.g., computer usage by LLMs.
  • Data sources — e.g., LLMs now search the internet.
  • Training — it’s like sending LLMs to High School, where they read a textbook.

In summary, AI models are based on not just copying human brains, but also imitating all of the ways that humans learn. And it’s not just the main neural network algorithm that’s copied from a carbon brain. It’s all of the workarounds, too. If we ever get to a truly “intelligent” model, and I do think it’s an “if” not a “when” for the achievement of “Artificial General Intelligence” (AGI), then it’s going to be achieved by copying everything we know about human intelligence.

And that’s the sweetest lesson of all.

 

Sweetest Lesson AI Book:

• Online: Table of Contents

• PDF: Free PDF book download

• Buy: The Sweetest Lesson: Your Brain vs AI

The Sweetest Lesson: Your Brain Versus AI The Sweetest Lesson: Your Brain Versus AI: new book on AI intelligence theory:
  • Your brain is 50 times bigger than the best AI engines.
  • Truly intelligent AI will require more compute!
  • Another case of the bitter lesson?
  • Maybe it's the opposite of that: the sweetest lesson.

Get your copy from Amazon: The Sweetest Lesson