Artificial intelligence systems may be good at generating text, recognizing images, and even solving basic math problems—but when it comes to advanced mathematical reasoning, they are hitting a wall.

There weren’t calculators or computers in medieval Europe. But there were math duels. Mathematicians would gather in public squares and pose tricky math problems to each other. Then they raced to ...