I like this comment on Slashdot in the above link:

LLMs don’t have an understanding of anything. They can only regurgitate derivations of what they’ve been trained on and can’t apply that to something new in the same ways that humans or even other animals can. The models are just so large that the illusion is impressive.

So true.