AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models
Zihan Chen, Lei Nico Zheng, Di Zhu
Don't read falling human ratings as proof that a model got worse. Fixed-version tests can rise over time. Compare models head-to-head at the same moment, and track the rating standard itself.
Users say AI assistants are getting worse. Nobody has checked whether the people doing the rating are changing instead of the models.
Method: Researchers studied 1,260,967 blind votes on Chatbot Arena, where people pick the better of two anonymous models or say both are bad. They only used model pairs whose versions stayed fixed. For the same unchanged pairs, the share of 'both are bad' votes rose by 1.71 percentage points a month, from a mean of 17.1%. The rise held within the same voters and across most controls. The ranking between the stronger and weaker model stayed flat.
Caveats: Based on one public platform with self-selected voters. Results may not carry to paid users or to other rating setups.
Reflections: How much of the rising standard comes from users comparing against newer products they have just used? · Would scheduled re-anchoring with fixed reference answers keep absolute scores stable over time?