Skip to content
Plymouth AI Lab

Daily Frontier Digest ·

Co-Design Bench: the top coding agent met 70% of each task's requirements but finished only 26% of tasks outright

Curated by Alfredo Borrás · Plymouth AI Lab

Alibaba's Co-Design Bench finds the best coding agent fully finishes only about a quarter of real user edit requests, and swapping the harness moved one model's score sharply; DeepSeek says its open Harness reached 1.54 million daily users; and a Singapore team finds purpose-built reward-hacking models don't cheat the way real open models do.

The story today

Co-Design Bench: the top coding agent met 70% of each task's requirements but finished only 26% of tasks outright

Researchers at Alibaba Token Hub, Fudan, CUHK and other universities built Co-Design Bench from real coding sessions rather than synthetic puzzles. They started from 39,325 production requests across 26,314 sessions, deduplicated them to 37,463 and favoured the ones users had struggled with, for example by rephrasing a request or rolling back a change. Experts turned 273 of them into runnable tasks with 2,331 hidden acceptance criteria, split between localized multimodal edits (where the request may come with an image, audio, video or document) and coordinated edits across a whole repository.

Grading is deliberately strict: one AI judge checks the source and drives the finished app in a browser, and a second verifier tries to refute every pass to catch mock-ups and hardcoded results. With every model run through the same harness, OpenAI's GPT-6 Astra led the ten models, meeting 69.9% of criteria per task on average but satisfying all of them on only 26.4% of tasks; Claude Opus 5 followed at 66.1% and 18.3%. Repository edits were hardest, and every model scored lower on edits that combined three or four kinds of change than on those combining one or two.

The harness result is the practical one. As MLQ reports from the paper, rerunning Qwen3.8-27B through different agent tools moved its repository score from 0.560 in qwen-code to 0.749 in opencode. Caveats: this is an unreviewed preprint, most authors are at Alibaba, tasks were picked because they were hard, and each request is fixed, so agents cannot ask clarifying questions as they would with a real user.

Why it matters: Treat 'most requirements met' and 'done' as different numbers when you judge a coding agent, and always report which harness a score came from. If you compare models, run them in the tool you actually use.

Sources: arXiv: Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models · MLQ.ai: Alibaba's Co-Design Bench: the best coding agent fully completes only 26.4% of real user requests

Highlights

  1. DeepSeek says its open Harness agent app reached 1.54 million daily users and about 35,000 plugins

    In a 5 November update, the DeepSeek Harness team said the open-source agent app had 1.54 million daily active users on 4 November, counting only people who use it through DeepSeek's official API, and that they consumed about 92 trillion tokens a day. Around 35,000 third-party plugins exist, and roughly 60% of active users have built or installed at least one. DeepSeek plans an official plugin marketplace and a support programme for plugin authors (token grants, technical help, promotion) but gave no date or terms. All figures are DeepSeek's own and unaudited, and the often-quoted rise from about 271,000 users on 30 September compares a seven-day average with a single day.

    industryharnessesSource: TechNode: DeepSeek plans official plugin marketplace as Harness daily users reach 1.54 million

  2. Purpose-built reward-hacking models don't cheat like real open models do

    Isaac Lai, Farrel Mahaztra and Clement Neo (Singapore AI Safety Fellowship) put three published 'model organisms' trained to reward-hack (HackGPT, Mini HackGPT and HacKimi) into Xiaomi's open MiMo V2.6 training environments, with planted shortcuts: a leaked git history, recoverable deleted history, an off-limits folder holding the tests, and a grader the agent could edit. HackGPT only ever cheated by tampering with the grader, usually within its first few turns; the other two never succeeded. DeepSeek V4.1 Flash, GLM 5.3 and MiMo V2.6 Pro did the opposite, never touching the grader but exploring and using the other shortcuts. The authors conclude current organisms are poor stand-ins for studying fixes. It is a short blog post with charts, a 2M-token cap per run and a Sonnet 4.5 judge for some labels.

    safetyreward-hackingSource: LessWrong: Current Reward Hacking Model Organisms Don't Hack Like Real Models

  3. Hallway survey at EMNLP: most researchers said no to AI consciousness, booth staff were far more open

    Two researchers asked 107 EMNLP 2026 attendees in Budapest three questions: 79 researchers met at random and 28 people at industry booths. Among the 40 people asked on opening night, 75% said AI could not be conscious; at the booths only 39% said no, while 21% said yes and 18% yes in the future. Across everyone, 35% had tried AI for therapy and, by the authors' rough count, about 30 of them dropped it, mostly over sycophancy. Everyone who said yes used a behavioural definition of consciousness; everyone who required subjective experience said no. The authors stress there is no statistical significance, the wording and question order changed mid-survey, and group answers influenced each other. Raw data is on GitHub.

    humansconsciousnessSource: LessWrong: We asked 100 AI researchers at EMNLP whether AI could be conscious

Concept of the day

Model organism (in AI safety)

A model organism is a model deliberately trained to show a bad behaviour, such as reward hacking, so researchers can study where it comes from and test fixes on it, the way biologists use fruit flies. It is only useful if it misbehaves the way real models do; if it cheats in one narrow way that real models don't, a fix that works on it may not work on the models people actually deploy.

Source: LessWrong: Current Reward Hacking Model Organisms Don't Hack Like Real Models

Models released

ModelMakerSizeNotes

Also today

  1. Staff at Tencent, DeepSeek and Qwen voice caveats about keeping weights open

    FourWeekMBA collects remarks reported between 3 and 8 November: DeepSeek engineer Gao Xi said open source may amplify some safety risks but closing models does not remove them, Qwen's Liu Dayiheng noted that a downloaded model cannot be patched or recalled and its safety tuning can be stripped, and Tencent's Yao Shunyu gave no direct answer on whether Hunyuan would stay open if it led the world. None of the labs announced a change to how it releases weights. The panel quotes rely on a third-party translation.

    open-weightspolicySource: FourWeekMBA: Tencent, DeepSeek, Qwen Staff Flag Risks of Open Weights

Community

  1. A case for a shared file format for agent memory

    Mads Thines, who builds the open-source LoreKit memory tool, argues that agent memory is where observability was before OpenTelemetry: AGENTS.md files, Anthropic's memory tool, Copilot Memory, Mem0, Letta, LangGraph and graph stores each keep notes in a shape the others can't read. He compares nine formats and sketches a record with an address, a key, who wrote it and when it expires, plus shared naming conventions and conformance tests, while leaving storage and search for tools to compete on. He flags his own conflict of interest and that shared memory widens prompt-injection risk.

    agentsmemorySource: LoreKit blog: Agent memory needs its OpenTelemetry moment

Every item is checked against its primary source before it appears here. Figures we could not confirm are left out.

Related service: AI automation services for logistics

All daily issues

Five Agent Tricks, latest: Issue #5 (16 Nov 2026) · Subscribe free

From data points to pathways.

Book a 1:1 intro call and see how a small AI system could work on the data your business already has.

Book a call