Model output can look like human reasoning however they often ignore these intermediate steps and the output tokens are often filler designed to allow more context to load. There something there that's half way to reasoning because it's loading that related training data but it not a connected chain of thought as we do. I read this article about it a few weeks ago and it summarizes the current research.
https://www.quantamagazine.org/is-ai-reasoning-right-for-the-wrong-reasons-20260731/
Thanks that sounds interesting, will have a listen. Mapping human cognition to what's likely represented with the stuff stored in neurons is very tricky but also quite intriguing.