Even the greatest artificial intelligence agents are pretty hopeless at on-line freelance work, in accordance to an experiment that challenges the thought of AI changing workplace employees en masse.
The Distant Labor Index, a brand new benchmark developed by researchers at information annotation firm Scale AI and the Middle for AI Security (CAIS), a nonprofit, measures the potential of frontier AI fashions to automate economically beneficial work.
The researchers gave a number of main AI brokers a spread of simulated freelance work and located that even the greatest might carry out lower than 3 % of the work, incomes $1,810 out of a doable $143,991. The researchers checked out a number of instruments and located the most succesful to be Manus from a Chinese language startup of the identical title, adopted by Grok from xAI, Claude from Anthropic, ChatGPT from OpenAI, and Gemini from Google.
“I ought to hope this offers far more correct impressions as to what is going on on with AI capabilities,” says Dan Hendrycks, director of CAIS. He provides that whereas some brokers have improved considerably over the previous 12 months or so, that does not imply that this may proceed at the identical charge.
Spectacular AI advances have led to hypothesis about AI quickly surpassing human intelligence and changing huge numbers of employees. In March, Dario Amodei, CEO of Anthropic, recommended that 90 % of coding work would be automated inside a matter of months.
Earlier waves of AI have impressed misplaced predictions about job displacement, for instance regarding the imminent replacement of radiologists with AI algorithms.
The researchers generated a spread of freelance duties by verified Upwork employees. The duties span a spread of labor together with graphic design, video enhancing, sport growth, and administrative chores like scraping information. They mixed an outline of every job with a listing of recordsdata wanted to carry out the work and an instance of a completed mission produced by a human.
Hendrycks says that whereas AI fashions have gotten higher at coding, math, and logical reasoning lately, they nonetheless wrestle to use completely different instruments and to carry out advanced duties that contain quite a few steps. “They do not have long-term reminiscence storage and may’t do continuous studying from experiences. They cannot decide up expertise on the job like people,” he says.
The evaluation provides a counterpoint to a benchmark of financial work provided in September by OpenAI known as GDPval, which purports to measure economically beneficial work. In accordance to GDPval, frontier AI fashions akin to GPT-5 are approaching human skills on 220 duties throughout a spread of workplace jobs. OpenAI did not present a remark.
Disclaimer: This article is sourced from external platforms. OverBeta has not independently verified the information. Readers are advised to verify details before relying on them.
