Something Weird Is Happening with LLMs and Chess

I came across an article titled Something weird is happening with LLMs and chess. It delves into the surprise discovery that when presented with a list of chess moves, certain LLMs were surprisingly good at suggesting strong next moves and winning games.

Well, that turned out not to be quite the full story: it was actually isolated to one specific model checkpoint, while newer and larger models performed much worse. It’s a bit of a mystery, but the author added an update at the bottom with deeper testing and theories on why this happened.