Gwern shows that while LLMs are terrible at arithmetic, they are able to invent novel non-trivial mathematical concepts and demonstrate understanding of it.

Gwern explains how the LLM is able to do this by identifying mathematical concepts using the same pathways used for modelling language.

It's a fascinating dive into how LLMs work conceptually and is a further display of emergent, surprising capability.