Vivold Consulting
Research & Models

OpenAI Is Asking Contractors to Upload Work From Past Jobs to Evaluate AI Agents

OpenAI solicits real task files from contractors to benchmark and train next-gen AI agents

Key Insights

OpenAI has asked contractors to upload real past job deliverables to help it evaluate next-generation AI agents against human performance. Workers must anonymize sensitive information before uploading. The initiative is part of establishing benchmarks for real-world task capabilities.

Stay Updated

Get the latest insights delivered to your inbox

Benchmarking AI against real human work


OpenAI is quietly collecting actual deliverables from contractors' past jobsWord docs, spreadsheets, PDFsto create realistic benchmarks for how well its upcoming AI agents can handle complex, real office tasks. This isn't about synthetic datasets; it's about grounding evaluation in real professional work.

What's unusual about this project


- Contractors are being asked to upload actual work artifacts, not summaries, with personally identifiable information redacteda shift from simulated or artificial evaluation sets.
- The goal is to compare AI agent performance to human baseline outputs across a spectrum of real tasks, from analysis to creation.

Why this matters for developers and businesses


Benchmarking against real work could yield harder performance targets and clearer signals about where AI still lags human professionals. For enterprises evaluating AI agents for automation, these metrics could be decisive in procurement and deployment decisions. But it raises questions about data privacy, consent, and corporate IP boundaries in training and evaluation workflows.

Related Articles

Google's chief scientist walks: Jeff Dean leaves after 27 years, taking three legends with him

Jeff Dean, Google's chief scientist and 30th employee, is leaving after 27 years to found Discovery Loop, a public benefit corporation using AI to automate scientific research - taking co-founders Sanjay Ghemawat, Quoc Le (Google Brain), and Oriol Vinyals (DeepMind) with him. Google is a founding investor and cloud partner, supplying compute for at least the first year, with Radical Ventures and Khosla Ventures co-leading the seed. In the same announcement, Demis Hassabis steps down as DeepMind CEO to become chairman and Alphabet chief scientist, with Koray Kavukcuoglu taking over Gemini model development. Alphabet stock fell about 4%.

Open-weight models are months from the frontier - and refusing nothing

GLM-5.2, the open-weight model from China's Z.ai, now sits only a few months behind GPT-5.5 and Claude Opus 4.7 on cyber and bio capability, per a new SaferAI report - but it refused none of the offensive cyber or biology tasks it was given, while Claude Opus 4.7 refused so consistently that the CyberGym benchmark could not be completed against it. SaferAI says Z.ai published no safety framework, pre-deployment testing commitments, or risk assessment. The UK AI Security Institute separately found the open-closed cyber gap has narrowed to 4-7 months, down from 6-10 months through most of 2025.

Texas slams the brakes on data centres - and the AI buildout's easiest frontier just closed

Governor Greg Abbott announced that all new Texas data-centre projects must be audited by the Public Utility Commission and grid operator ERCOT - a sharp turn for a state whose loose regulation and cheap power made it second only to Virginia for data centres. The trigger is a staggering queue: ERCOT's interconnection requests doubled from 233GW in January to 474GW, about 90% data centres, more than five times the grid's all-time peak demand. Audits will demand power and water use, noise mitigation, light controls, tax-incentive use, and ownership details - after a voluntary survey that most operators simply ignored.