Well, not sure what you mean by hallucinating, but there are some LLMs (chatgpt3.5-turbo-instruct) that could play chess at a good amateur level (1750 elo), and by studying them, some people found parameters in the model that seemed to represent the current state of the chess board (as in, they corresponded to the game, and changing them artificially made the LLM play as if the game as always been that way).
Well, not sure what you mean by hallucinating, but there are some LLMs (chatgpt3.5-turbo-instruct) that could play chess at a good amateur level (1750 elo), and by studying them, some people found parameters in the model that seemed to represent the current state of the chess board (as in, they corresponded to the game, and changing them artificially made the LLM play as if the game as always been that way).