LLMs are solving MCAT, the bar test, SAT etc like they’re nothing. At this point their performance is super human. However they’ll often trip on super simple common sense questions, they’ll struggle with creative thinking.
Is this literally proof that standard tests are not a good measure of intelligence?
That’s a common misunderstanding.
LLMs have billions of neurons, and we can see firsthand how information travels along their neural pathways and, yeah, it looks a whole lot like they’re thinking. If anything, we have more concrete proof that LLMs think than that humans think.
They do think, it’s just that they don’t have short term memory. They can only remember things linguistically, by talking and then listening to their own output. It’s an artifact of how we’ve set them up to interact with the world. Many humans use a similar thought process for certain problems (e.g. talking out loud to a rubber duck). Sure, there are other ways humans think too (e.g. visual/spatial), but linguistic thought is still valid.