Model Evaluation and Threat Research is an AI research charity that looks into the threat of AI agents! That sounds a bit AI doomsday cult, and they take funding from the AI doomsday cult organisat…
The research explicitly showed that the anecdotes were flawed, and that actual measured productivity was the inverse of what the users imagined. That’s the entire point. You’re just saying “nuh uh, muh anecdotes.”
I said it needs to be measured. But few teams are going to do that, they’re building products not case studies.
This study is catnip for the people who put “AI” in scare quotes and expect those of us who use it to suddenly realize that we’ve only been generating hallucination slop. This has not been the lived experience of those of us in software development. In my own case I’ve seen teams stop hiring because they are getting the same amount of work done in less time. But those are anecdotes, so it doesn’t count.
The research explicitly showed that the anecdotes were flawed, and that actual measured productivity was the inverse of what the users imagined. That’s the entire point. You’re just saying “nuh uh, muh anecdotes.”
I said it needs to be measured. But few teams are going to do that, they’re building products not case studies.
This study is catnip for the people who put “AI” in scare quotes and expect those of us who use it to suddenly realize that we’ve only been generating hallucination slop. This has not been the lived experience of those of us in software development. In my own case I’ve seen teams stop hiring because they are getting the same amount of work done in less time. But those are anecdotes, so it doesn’t count.
It’s entirely possible to measure metrics.
Enjoy your slopware.