Apple researchers conduct study on AI’s reasoning ability in mathematics: What they found

Apple researchers, questioning the formal reasoning capabilities of large language models (LLMs), particularly in mathematics, conducted a study to understand the reliability of the reported metrics. They found that LLMs exhibit noticeable variance when responding to different instantiations of the same question.