Quick Screen results

Local Benchmark Framework v3.0.1 — corrected adapter, frozen before scored trials
19 October 2026

ADAPTER REVISION
An initial Qwen pilot added Pydantic argument validation, rejecting object-valued
API bodies that the upstream tool explicitly accepts. The pilot is invalid
harness evidence, preserved separately, and excluded from scored results. v3.0.1
dispatches tool arguments directly to the unmodified upstream functions, with
only signature binding and the upstream empty-object sentinel normalization.
All affected candidate tasks restart from fresh worlds; no selection changes.

PURPOSE
Evaluate a complete model/quant/engine/agent configuration for local business
automation and coding. HumanEval+ v2.1 remains historical context, not the v3
coding metric. Public benchmarks can be present in training; no contamination-
free claim is made for a model with an unknown training cutoff.

TIERS (time targets are PER CONFIGURATION, not for the whole model queue)
1. Quick Screen: 8 AutomationBench tasks = 2 simple tool-use checks + 1 task
   from each of sales, marketing, operations, support, finance, HR. Four
   LiveCodeBench problems: 1 easy, 2 medium, 1 hard. Short inference test.
   Target 20–45 minutes excluding download/build and cold model provisioning.
   One attempt per task; AutomationBench 16 response steps, 180 seconds/task,
   2048 output tokens/response. Coding 180 seconds/problem, 4096 output tokens.
   No generated-code execution in inference or agent containers.
2. Comparison Suite: 30 AutomationBench business tasks (5/domain), the same
   2 simple checks reported separately, 8 LiveCodeBench (2 easy/4 medium/2 hard),
   3 Terminal-Bench 2.0 tasks, 2 SWE-bench Pro public issues. Target 4–12 hours.
   Automation 50 steps/360 seconds/task/4096 tokens; coding 300 seconds/8192
   tokens; terminal 30 minutes/60 steps; SWE 60 minutes/100 steps. One attempt.
   These caps are local suite policy and prevent direct leaderboard equivalence.
3. Full Validation: all 600 AutomationBench public business tasks, all 89
   Terminal-Bench 2.0 tasks, the pinned SWE-bench Pro public release, and a
   frozen complete LiveCodeBench release. Target 7–30+ days; actual hardware,
   release size and official task budgets determine runtime. Run only after a
   separate user decision. Also report simple-domain diagnostics separately.
   Use upstream task timeouts and locked agent protocol; repeat complete runs
   where feasible and disclose repeat counts and uncertainty.

VERSION POLICY
Pin source Git commits, dataset revisions, task IDs, task contract hashes,
container image digests, prompts, tool schemas and settings before any run.
Quick tasks nest inside Comparison tasks. Deterministic SHA256 ordering uses
seed [seed fingerprint in task manifest; exact selected task IDs published]; no selection based on candidate output or pass rates.
For the current LiveCodeBench screen use the official v6 increment (test6.jsonl,
Jan–Apr 2025 problems), with a pinned dataset revision and full file checksum.
Select by difficulty, SHA256(seed + platform + question_id); retain all tests.
Full Validation uses release_v6 (all six files), not only the v6 increment.
Changing a task or grading bug creates a protocol revision and a complete
affected-profile rerun; preserve the original attempt and reason. Do not
silently retry model failures or substitute easier tasks.
SWE-bench Pro currently offers validated V2 (642 public tasks). Pin that version
for future work rather than silently using the older 731-task v1 set.
Terminal-Bench 2.0 is intentionally the user's 89-task target; newer editions
are separate versions and must not silently replace it.

MEASUREMENT AND SCORING
AutomationBench: use upstream task prompts, simulated tools and assertion rubric.
Our bounded Chat Completions loop is a local adapter, not an official leaderboard
submission. Use the upstream api toolset (api_search/api_fetch/base64_encode),
unchanged services gating, initial-state normalization and free-assertion rules.
Never send initial state, hidden assertions or expected outcomes to the model.
Report basic-tool strict passes /2, business strict passes /6 or /30, macro
domain pass rate, partial assertion credit, tool errors, steps, wall time and
token usage. Simple tasks never inflate the business score. Strict task success
comes from upstream task_completed_correctly; deadline/step exhaustion is also
reported and does not erase the observed final-state rubric outcome.
LiveCodeBench: official generic Python code-generation prompt, official code
extraction and upstream test runner against all public and private test cases.
One generated answer per problem, pass@1; no repair using test feedback. Report
easy/medium/hard separately, count /4 or /8, all-test pass rate, generation
timeouts, code extraction failures, test runtime failures and grader failures.
Offline scoring after inference stops: 6 seconds per upstream test and an
additional 180-second container ceiling per problem (screen policy).
Terminal/SWE: report task resolution, agent steps, timeouts, test/infrastructure
failures and costs separately. Freeze the agent and tool implementation, not
only the model. Never expose solutions, hidden tests or Docker socket to agents.
Where upstream tasks omit the world's clock, freeze it at 2026-10-19T00:00:00Z;
preserve all task-provided clock values. This initialization is in the manifest.
Infrastructure errors are visible and not passed; show scheduled denominator
and valid-evaluation denominator side by side when a grader error occurs.
Never remove model transport failures, empty finals, malformed tool calls or
timeouts from the scheduled denominator.
No opaque combined score across business, code and speed. Show a scorecard.
Wilson 95% intervals accompany small binary samples. A 6-task or 4-problem
screen is a triage signal, not a statistically reliable model ranking.

SPEED, RESOURCES AND PROMOTION
One warm-up excluded from summary; three fixed 256-token generation requests
and three fixed ~2K-token prefill requests, serial concurrency=1. Use streamed
timestamps for time to first nonempty content token. Report median end-to-end latency, actual prompt/completion tokens and
end-to-end completion tok/s.
Speed requests explicitly use reasoning_effort=none for every model; quality
requests retain each tested profile's reasoning mode. Completion-token counts
may include reasoning tokens when a backend does not honor the speed setting.
When engine timings exist report prompt/decode separately with their source;
never label completion tokens / total wall time as pure decode throughput.
Keep cold startup distinct. Five-second GPU/memory sampling can miss peaks;
record memory reserve, peak sampled VRAM, minimum sampled free VRAM, guest OOM,
GPU faults and temperature. Context configured is not context quality tested.
Use the existing 64K/1024 MiB reserve configuration for this six-model screen.
Predeclared advisory promotion: business candidate if simple 2/2 and business
>=2/6; coding candidate if coding >=2/4. Runtime-health requirement: no GPU
fault, OOM, or transport failure. A failed tool API is a compatibility result,
not evidence that the weights cannot reason. Hold for diagnosis when runtime
or harness failures prevent a valid capability measurement. Report raw scores
even when advisory gates fail. No automatic Tier 2/3 start.

CURRENT MODEL MATRIX
Qwen3.6 35B-A3B UD-Q3_K_M; Ornith 1.5 APEX-I-COMPACT;
Tiel Coder 35B-A3B UD-Q3_K_XL; Gemma4 26B-A4B UD-Q4_K_M thinking disabled;
MiniCPM-V 4.7 mix_q3gud_im (text only); LFM2.5 24B-A2B UD-Q4_K_XL.
Keep exact existing weight hashes/runtime commit/draft configuration. Gemma
requires reasoning_effort=none and enable_thinking=false on every request.
Other modes preserve the previous tested engine settings. Request temperature
0 and seed 12345; log what the engine accepts. Do not add the speed-only Qwen
Q4 control as a seventh quality profile or repeat the known broken Gemma mode.

ISOLATION AND EVIDENCE
Reuse only the authorized benchmark LXC and run one GPU worker at a time. Other
LXCs remain stopped. Preserve all earlier frozen results and application config.
Inference: offline, nonroot, read-only base/model/source mounts, dropped caps,
no-new-privileges, resource limits. Automation tools mutate only an in-memory
fictional world; no real SaaS accounts, network requests or messages. Simulated
tool URLs are routing identifiers, not permission to contact those services.
Code scoring: separate offline nonroot disposable container, no GPU/host socket,
read-only input, writable bounded tmpfs and output only; inference stopped first.
Record every request/response, terminal condition, final-world assertions and
grading output privately. Public reports contain aggregate results, safe task
IDs, pinned versions and hashes; omit personal information, host/IP paths,
credentials, raw simulated contacts and operational logs. Preserve the prior
official-only PR policy: v3 custom reports are not added to that engine PR.
Deliver protocol + frozen task manifest + machine-readable scorecard + HTML
report + private evidence manifest and handoff notes. Report setup and inference
time separately; local electricity, depreciation and labor remain unmeasured.

PRIMARY SOURCES
https://github.com/zapier/AutomationBench
https://github.com/LiveCodeBench/LiveCodeBench
https://huggingface.co/datasets/livecodebench/code_generation_lite
https://www.tbench.ai/news/terminal-bench-2
https://github.com/scaleapi/SWE-bench_Pro-os
No API keys are supplied: upstream ChatGPT tools use their documented simulated
fallback; unsupported cloud-dependent queries remain visible tool errors.