A recent study has shown that the GPT-4 language model underlying the chatbot has become much dumber as of June 2023 than the same model was in March 2023. At the same time, the GPT-3.5 language model, on the contrary, has improved in most tasks, although it still has problems.
Researchers at Stanford University asked the chatbot various questions and evaluated the correctness of the answers. It was not about some abstract things, but quite specific ones. For example, the AI had to answer whether the number 17 077 was prime. To better understand the process of AI “thinking” and improve the result, the chatbot was asked to describe its calculations step by step. Usually, in this mode, AI is more likely to answer correctly.
However, it did not help. While back in March, GPT-4 model gave the right answer in 97.6% of cases, in June, the figure dropped to 2.4%! That is, the chatbot almost never answered the question correctly. In the case of GPT-3.5, the figure, on the contrary, increased from 7.4% to 86.8%.
Code generation has also deteriorated. Scientists created a dataset with 50 simple LeetCode tasks and measured how many GPT-4 answers were performed without any changes. The March version successfully coped with 52% of the problems, but when using the June model, this figure dropped to 10%.
By the way, users have been complaining about the decline in ChatGPT‘s “mental capacity” in recent weeks. It is still unclear why this happened and whether OpenAI will do anything about it.
