{"data":[{"status":"published","verifiedAt":"2026-10-24","featured":true,"slug":"swe-bench-verified","name":"SWE-bench Verified","organization":"SWE-bench Team","category":"engineering","description":"从真实 GitHub issue 出发，检验 agent 能否理解代码库并提交正确的修复。","version":"Verified · 500 tasks","taskCount":"500","metric":"% Resolved","environment":"容器化代码仓库","tags":["真实仓库","问题修复","执行验证"],"maintenance":"active","maintenanceEvidence":"官方维护多条评测轨道，当前提供统一 mini-SWE-agent 的 Bash Only 视图；2026 年发布了同家族 ProgramBench。","overview":"面向真实软件维护的常用参考。给定 issue 和修复前的仓库，agent 需要定位问题、修改代码，并通过测试。Verified 是经人工筛选的 500 个任务子集。","methodology":"在隔离环境中应用 agent 生成的补丁，再运行问题相关测试与回归测试。% Resolved 表示满足判定条件的任务比例。比较时须确认使用同一数据集与 agent 设置。","example":"示意任务：某 Python 库在边界输入下抛出异常。agent 阅读 issue、定位相关函数、补充处理逻辑，并运行回归测试。此示例为编辑概括，并非原题。","strengths":["真实仓库与已有测试，使结果比孤立函数题更贴近维护工作。","统一运行环境的榜单视图有助于控制 agent 差异。"],"limitations":["Verified 主要围绕 Python 项目，不能直接代表其他语言。","测试通过不等于代码质量、可维护性或产品交付质量。","公开题集存在被训练数据或专门优化影响的可能。"],"bestFor":"选择处理仓库 issue 与修复任务的 coding agent。","notFor":"判断从零构建产品、视觉设计或所有语言的综合能力。","homepage":"https://www.swebench.com/","repository":"https://github.com/SWE-bench/SWE-bench","sources":[{"label":"官方榜单与数据集说明","url":"https://www.swebench.com/"},{"label":"评测实现","url":"https://github.com/SWE-bench/SWE-bench"}]},{"status":"published","verifiedAt":"2026-10-24","featured":true,"slug":"terminal-bench","name":"Terminal-Bench","organization":"Stanford · Harbor · Laude","category":"terminal","description":"在真实终端环境中完成多步骤任务，观察 agent 从计划到执行的完整能力。","version":"4.0","taskCount":"以版本清单为准","metric":"Resolution rate","environment":"隔离终端环境 · Harbor","tags":["长程任务","Shell","Agent harness"],"maintenance":"active","maintenanceEvidence":"官方于 2026-08-28 发布 4.0，校准任务资源、修复 19 个任务并移除 8 个任务。","overview":"适合关注终端型 coding agent 的读者。任务需要 agent 自主操作工具并检查结果；官方同时展示 model 与 agent，提醒读者评估的是一个系统。","methodology":"在预设环境中运行 agent，通过任务 verifier 判断完成情况。4.0 校准时间、CPU 与内存配置，降低基础设施噪声。结果应连同运行成本、agent 和版本一起阅读。","example":"示意任务：接手一个无法构建的项目，排查环境依赖、修改配置，最终产出能通过验证的构建结果。此示例为编辑概括，并非原题。","strengths":["覆盖连续工具调用与环境操作，贴近终端 agent 的使用方式。","官方更新日志明确说明任务与基础设施修订。"],"limitations":["不同版本删除或修复任务，跨版本分数不能直接相减。","模型、harness、工具配置和预算都会影响结果。","长任务的运行费用与环境资源需要单独考虑。"],"bestFor":"理解 Codex、Claude Code、pi 等终端工作流需要的能力。","notFor":"仅用一个成功率断定基础模型的纯推理能力。","homepage":"https://www.tbench.ai/","repository":"https://github.com/laude-institute/terminal-bench","sources":[{"label":"官方榜单","url":"https://www.tbench.ai/"},{"label":"4.0 版本说明","url":"https://www.tbench.ai/news/terminal-bench-4-0"}]},{"status":"published","verifiedAt":"2026-10-24","featured":true,"slug":"livecodebench","name":"LiveCodeBench","organization":"UC Berkeley · MIT · Cornell","category":"coding","description":"用带发布时间的竞赛题观察代码能力，减少旧题记忆对结果的干扰。","version":"按 release 与时间窗口","taskCount":"随版本变化","metric":"pass@1 / pass@5","environment":"代码生成与测试执行","tags":["动态题集","算法编程","污染控制"],"maintenance":"periodic","maintenanceEvidence":"官方仓库提供多个 release、按日期筛选的评测脚本及勘误文档。核验到的 release 列表包含 v6；未据此推断更新频率。","overview":"从 LeetCode、AtCoder 和 Codeforces 收集竞赛题，并记录发布时间。时间窗口是阅读结果时必须保留的条件，而非可省略的备注。","methodology":"生成候选程序并执行测试，以 pass@k 等指标统计结果。官方同时研究代码生成、自我修复、代码执行和测试输出预测等场景。","example":"示意任务：将一道有明确输入、输出和复杂度约束的算法题转换为程序，并通过隐藏测试。此示例为编辑概括。","strengths":["题目时间戳有助于选择训练截止日期之后的任务。","明确 release 与时间窗口便于检查可比性。"],"limitations":["竞赛编程并不等价于维护大型仓库。","时间划分可降低污染风险，但不能证明完全没有污染。","采样次数、解码设置与测试时限会影响分数。"],"bestFor":"比较算法实现与代码推理的一个能力侧面。","notFor":"直接预测 agent 的产品交付效率。","homepage":"https://livecodebench.github.io/","repository":"https://github.com/LiveCodeBench/LiveCodeBench","sources":[{"label":"项目主页","url":"https://livecodebench.github.io/"},{"label":"版本与运行说明","url":"https://github.com/LiveCodeBench/LiveCodeBench"}]},{"status":"published","verifiedAt":"2026-10-24","featured":false,"slug":"swe-bench-multilingual","name":"SWE-bench Multilingual","organization":"SWE-bench Team","category":"engineering","description":"把真实仓库修复扩展到 9 种语言，补上仅看 Python 评测的盲区。","version":"Multilingual","taskCount":"300","metric":"% Resolved","environment":"42 个代码仓库","tags":["多语言","真实仓库","问题修复"],"maintenance":"active","maintenanceEvidence":"官方主站保留独立榜单轨道，评测代码与 SWE-bench 仓库整合。","overview":"来自 42 个仓库的 300 个任务，覆盖 C、C++、Go、Java、JavaScript、TypeScript、PHP、Ruby 和 Rust。","methodology":"沿用 SWE-bench 的任务形式：issue、修复前代码与可执行测试。更广的语言分布有助于观察 Python 之外的工程能力。","example":"示意任务：在 TypeScript 仓库中修复一个类型或运行时行为缺陷，同时保持既有测试通过。此示例为编辑概括。","strengths":["补充 Python 评测无法覆盖的语言生态。","与 SWE-bench 评测工具兼容。"],"limitations":["不同语言的样本分布需要查看数据集，不能假设平均。","总分会掩盖某一语言或仓库上的短板。"],"bestFor":"为多语言项目选择 agent 时增加一个观察维度。","notFor":"认为 300 个任务完整覆盖所有语言的生产场景。","homepage":"https://www.swebench.com/multilingual.html","repository":"https://github.com/SWE-bench/SWE-bench","sources":[{"label":"官方方法与语言分布","url":"https://www.swebench.com/multilingual.html"}]},{"status":"published","verifiedAt":"2026-10-24","featured":false,"slug":"swe-bench-pro","name":"SWE-Bench Pro","organization":"Scale AI","category":"engineering","description":"关注更复杂、跨文件的工程任务，把长程软件维护放进评测。","version":"Public / Held-out / Commercial","taskCount":"1,865（全体）","metric":"Pass@1","environment":"企业级软件仓库","tags":["长程任务","跨文件修改","企业软件"],"maintenance":"periodic","maintenanceEvidence":"官方研究页提供论文、数据、仓库和榜单入口；不同公开程度的子集独立解释。","overview":"原始研究覆盖 41 个仓库、1,865 个问题，并区分公开、保留和商业子集。设计目标是观察更复杂的工程维护任务。","methodology":"agent 在仓库中完成可能涉及多个文件的修复，使用测试检查结果。理解总分之前，先确认报告的是哪个子集和哪一种 scaffold。","example":"示意任务：追踪业务应用中跨模块的问题，修改多个文件并验证兼容性。此示例为编辑概括。","strengths":["强调较长时间跨度和更复杂的代码修改。","数据分层有助于讨论公开题集与评测污染的关系。"],"limitations":["1,865 是研究全体规模，并非公开集规模。","非公开子集的独立复现受访问条件限制。","不能与 SWE-bench Verified 的百分比直接横比。"],"bestFor":"补充观察复杂软件维护和长程推理表现。","notFor":"把不同子集与不同基准的分数合成统一总分。","homepage":"https://labs.scale.com/papers/swe_bench_pro","repository":"https://github.com/scaleapi/SWE-bench_Pro-os","sources":[{"label":"Scale 官方研究说明","url":"https://labs.scale.com/papers/swe_bench_pro"}]},{"status":"published","verifiedAt":"2026-10-24","featured":false,"slug":"programbench","name":"ProgramBench","organization":"Meta · Stanford · Harvard","category":"engineering","description":"只给二进制与文档，让 agent 从零重建完整程序的行为。","version":"官方 200-task 评测","taskCount":"200","metric":"Resolved / Almost","environment":"完整程序重建","tags":["从零构建","行为验证","长程任务"],"maintenance":"active","maintenanceEvidence":"官方页面标注 200 个任务，并显示更新于 2026-10-22。","overview":"给定编译后的程序与文档，agent 构建一个行为一致的实现。它把关注点从补丁修复扩展到程序级实现。","methodology":"使用隐藏行为测试检查重建程序。官方区分完全解决和通过至少 95% 行为测试的 Almost，并提供成本视角。","example":"示意任务：通过文档与可执行程序观察输入输出，设计并实现具有相同行为的替代程序。此示例为编辑概括。","strengths":["关注完整软件制品与系统性实现能力。","同时区分完全通过、接近通过和运行成本。"],"limitations":["行为测试无法覆盖所有可能输入。","程序重建与真实产品需求发现、设计协作仍有差别。"],"bestFor":"观察从零实现与长时间任务推进能力。","notFor":"把行为一致性视为安全性或生产质量的证明。","homepage":"https://programbench.com/","repository":"","sources":[{"label":"官方任务、指标与榜单","url":"https://programbench.com/"}]},{"status":"published","verifiedAt":"2026-10-24","featured":false,"slug":"bigcodebench","name":"BigCodeBench","organization":"BigCodeBench Team","category":"coding","description":"通过多库组合的实际编程任务，考察代码生成与指令理解。","version":"Full / Hard · Complete / Instruct","taskCount":"1,140（Full）","metric":"Pass@1","environment":"Python 库与测试环境","tags":["库调用","指令理解","执行验证"],"maintenance":"periodic","maintenanceEvidence":"官方榜单提供 Full/Hard、Complete/Instruct 视图及运行设置说明；不将榜单可访问等同于持续更新。","overview":"Full 集包含 1,140 个编程任务；Hard 子集更小。Complete 和 Instruct 使用不同任务表达方式，分别观察补全与自然语言指令下的实现。","methodology":"在任务环境中执行生成的代码并检查测试。阅读分数时需固定题集、任务表达方式、解码策略与聊天模板。","example":"示意任务：组合文件处理、数据分析与标准库函数，完成一个有测试约束的 Python 程序。此示例为编辑概括。","strengths":["多库调用比孤立的小函数更接近部分日常编程工作。","区分明确文档补全与简短自然语言指令。"],"limitations":["Full 与 Hard 的分数不可直接对比。","受控程序任务不能替代长期仓库工程评测。"],"bestFor":"判断 Python 库使用和需求到代码的转化能力。","notFor":"用单个总分推断所有语言的 agent 表现。","homepage":"https://bigcode-bench.github.io/","repository":"https://github.com/bigcode-project/bigcodebench","sources":[{"label":"官方榜单与设置说明","url":"https://bigcode-bench.github.io/"}]},{"status":"published","verifiedAt":"2026-10-24","featured":false,"slug":"tau2-bench","name":"τ²-bench","organization":"Sierra Research","category":"tools","description":"在用户与 agent 共同改变环境的任务中，检验工具调用与协作可靠性。","version":"τ² · Retail / Airline / Telecom","taskCount":"按领域划分","metric":"pass^k reliability","environment":"用户模拟器与业务工具","tags":["工具调用","用户协作","可靠性"],"maintenance":"active","maintenanceEvidence":"τ 家族持续演进：官方 2026 年更新了任务审计并增加 τ³ 分支。本条目仍明确指向 τ² 的双控任务。","overview":"评估 agent 与用户对话、调用工具并遵守业务规则的能力。τ² 引入双控环境：用户也可以执行动作，agent 必须协调双方。","methodology":"与模拟用户进行多轮交互，调用领域工具，以可验证的最终状态检查任务。可靠性指标关注重复尝试能否一致成功。","example":"示意任务：agent 根据业务规则排查服务问题，并引导用户完成只有用户才能执行的操作。此示例为编辑概括。","strengths":["关注真实工具副作用和多轮协作。","重复成功的可靠性视角补充一次成功率。"],"limitations":["模拟用户与真实用户存在差别。","不同领域、用户模型与任务修订会改变结果。","不是直接的编程能力评测；τ² 与 τ³ 需分开阅读。"],"bestFor":"理解带工具、用户交互与规则约束的 agent。","notFor":"仅依据业务工具任务选择最强代码生成模型。","homepage":"https://taubench.com/","repository":"https://github.com/sierra-research/tau2-bench","sources":[{"label":"官方版本演进","url":"https://taubench.com/"},{"label":"任务与评测代码","url":"https://github.com/sierra-research/tau2-bench"}]},{"status":"published","verifiedAt":"2026-10-24","featured":false,"slug":"osworld","name":"OSWorld","organization":"XLang Lab & collaborators","category":"computer","description":"让 agent 操作真实桌面应用，观察跨应用任务能否执行到位。","version":"2.0","taskCount":"以 2.0 任务集为准","metric":"Task success rate","environment":"真实桌面与应用环境","tags":["Computer use","跨应用","长程任务"],"maintenance":"active","maintenanceEvidence":"旧版官方主页明确引导至 2026-06-26 发布的 OSWorld 2.0。","overview":"面向计算机操作 agent 的真实环境评测。新版强调长程现实任务；访问旧榜单时，应检查是否仍在引用早期的 369 任务集合。","methodology":"agent 观察界面并操作应用，执行评测器检查任务最终状态。截图、动作空间、步数预算、应用版本均属于结果条件。","example":"示意任务：在桌面中编辑文件并跨应用整理信息，最后检查保存产物是否符合要求。此示例为编辑概括。","strengths":["覆盖 GUI 交互与跨应用流程。","以实际执行结果而非描述性答案评估完成情况。"],"limitations":["桌面环境与应用版本会影响可复现性。","旧版、Verified 与 2.0 的结果不能混用。"],"bestFor":"观察 coding agent 延伸到浏览器与桌面操作时的能力。","notFor":"将界面操作分数当作纯代码能力。","homepage":"https://osworld-v2.xlang.ai/","repository":"https://github.com/xlang-ai/OSWorld","sources":[{"label":"OSWorld 2.0 官方主页","url":"https://osworld-v2.xlang.ai/"},{"label":"旧版与迁移说明","url":"https://osworld-v1.xlang.ai/"}]},{"status":"published","verifiedAt":"2026-10-24","featured":false,"slug":"evalplus","name":"EvalPlus","organization":"EvalPlus Team","category":"coding","description":"用更充分的测试重新审视 HumanEval 与 MBPP，理解“通过测试”的边界。","version":"HumanEval+ / MBPP+","taskCount":"按子集划分","metric":"Pass@k","environment":"函数级代码执行","tags":["测试充分性","函数生成","经典参考"],"maintenance":"reference","maintenanceEvidence":"作为经典评测方法参考保留。官方维护 HumanEval+、MBPP+、EvalPerf 与 RepoQA 入口，未据此承诺固定更新节奏。","overview":"在 HumanEval 与 MBPP 的基础上扩充测试，提醒读者：测试覆盖变化可能改变对代码正确性的判断。","methodology":"对同一候选代码运行更充分的测试集合，观察原始测试漏检的错误。应明确使用的是原始题集还是增强测试。","example":"示意任务：一个函数能通过常规输入，却在空列表或边界值时失败；增强测试将暴露这个问题。此示例为编辑概括。","strengths":["直观呈现测试充分性对分数的影响。","为理解函数级生成评测提供历史与方法背景。"],"limitations":["经典公开题集的污染与饱和风险较高。","不建议单独用于选择当前的 coding agent。"],"bestFor":"理解为什么更充分的测试会改变代码正确性判断。","notFor":"单独作为当前 agent 综合能力的排名依据。","homepage":"https://evalplus.github.io/","repository":"https://github.com/evalplus/evalplus","sources":[{"label":"官方评测方法入口","url":"https://evalplus.github.io/"}]}],"total":10,"limit":30,"offset":0}