Stephen Wolfram on ChatGPT and AI

I was recently talking with someone about LLMs like ChatGPT and mentioned a piece by Stephen Wolfram (whom I’m becoming a bigger and bigger fan of) called What is ChatGPT Doing… and Why Does it Work?. This is a great read. There are parts that are incredibly dense, but there are also sections that really walk you through the iterative steps of building towards a complex LLM, showing how each piece is layered on to solve a problem. I bought this as a Kindle book before realizing it was freely available as an article on his website!

Stephen Wolfram also has another interesting writing called How to Think Computationally about AI, the Universe and Everything. If you find his explainer on ChatGPT enlightening, this is a great follow-up. One specific excerpt that stood out:

The big achievements of AI in recent times have been about making systems that are closely aligned with us humans. We train LLMs on billions of webpages so they can produce text that’s typical of what we humans write. And, yes, the fact that this works is undoubtedly telling us some deep scientific things about the semantic grammar of language—and generalizations of things like logic—that perhaps we should have known centuries ago.

That resonates so much with learning about embeddings: it feels like we’re stumbling upon these capabilities and being amazed they work, but haven’t yet truly understood why, or what it reveals about how our own brains work.

By the way, if you’re not familiar with Stephen Wolfram, he’s the creator of Mathematica as well as Wolfram|Alpha, which deserves much more credit for how impressively capable it is.