V3.0.1 · FIXED PUBLIC SUBSET · RX 6700 XT 12 GBQuick Screen:
business automation + coding
Six local model configurations. Eight workflow tasks, four coding problems, and a short inference test per configuration.
This is a small, bounded triage screen. These are local subset results, not official leaderboard scores. Business quality, coding and speed remain separate; the previous HumanEval+ results are historical context.
Limited business separation. Completion was only 0–1 of 6 business tasks under these limits. A one-task lead is not a reliable general ranking. Inspect failed checks, agent behavior and budget constraints before expanding the comparison.
Quick Screen scorecard
Basic tools: 2 simple tasks, excluded from the business score. Business: 1 task per domain. Coding: 1 easy, 2 medium, 1 hard, with all public and private tests. Partial credit averages upstream assertion scores across the six business tasks.
Runtime health checks GPU faults, guest OOM and API/transport failures; it does not indicate workflow quality or production readiness.
Business outcomes by domain
Failures and configured limits
A context rejection means the accumulated conversation exceeded the configured window; it does not establish the model’s maximum supported context or a GPU memory fault. This local agent did not compact its tool history.
Coding outcomes by problem
Measured inference latency
Three requests per measurement type, serial concurrency 1, 256-token output cap, one excluded warm-up. TTFT measures first nonempty streamed content; dashes mean unavailable. Completion throughput includes prefill and request overhead and may include reasoning tokens. These numbers are not directly equivalent to the earlier official speed suite.
Resources and startup
Profiles with a GPU fault flagged in engine logs: 0/6. Profiles with sampled guest OOM or OOM-kill increments: 0/6.
Five-second point samples may miss brief peaks. Temperature includes the hottest exposed GPU sensor, including hotspot. Total model time includes startup and offline scoring; the protocol time target excludes initial downloads and provisioning.
Uncertainty and advisory promotion
Qwen3.6 35B-A3B: business 95% interval 0.0–39.0%; coding 4.6–69.9%. Advisory: hold/review.
Ornith 1.5: business 95% interval 0.0–39.0%; coding 4.6–69.9%. Advisory: hold/review.
Tiel Coder 35B-A3B: business 95% interval 0.0–39.0%; coding 15.0–85.0%. Advisory: coding candidate; Tier 2 requires a separate decision.
Gemma4 26B-A4B · thinking off: business 95% interval 0.0–39.0%; coding 15.0–85.0%. Advisory: coding candidate; Tier 2 requires a separate decision.
MiniCPM-V 4.7 · text only: business 95% interval 0.0–39.0%; coding 15.0–85.0%. Advisory: coding candidate; Tier 2 requires a separate decision.
LFM2.5 24B-A2B: business 95% interval 3.0–56.4%; coding 4.6–69.9%. Advisory: hold/review.
Predeclared advisory gates: business requires basic tools 2/2 and business ≥2/6; coding requires ≥2/4. Both require valid grading and a healthy runtime. These gates nominate further testing, not production readiness. One business task changes the pass rate by 16.7 points; one coding problem changes it by 25 points.
Frozen v3 protocol
Hardware: AMD Radeon RX 6700 XT 12 GB, Ryzen 5 3600, 32 GB system RAM. These configurations use GPU and system RAM; the complete weights are not all resident in VRAM. PolyStrata 0.1.45, native HIP gfx1031, pinned runtime commit in the structured data.
- Small fixed public subsets; not official leaderboard scores or a reliable fine ranking.
- AutomationBench final-state completion rates are not directly comparable to the earlier v2.1 practical rubric or HumanEval+ percentages.
- Wilson intervals describe small binary counts; the fixed, stratified tasks are not a random sample of all real business work.
- No guarantee against training contamination; model training cutoffs unknown.
- The four frozen Quick Screen coding problems are all AtCoder; selection was stratified by difficulty, not platform.
- Simple tasks excluded from business denominator. All-test coding pass@1, one attempt.
- Speed is completion tokens divided by total request wall time, not pure decode throughput.
- Fixed repeated speed prompts may reuse engine KV/prefix state. Long-prompt latency is a warm-session measurement, not isolated cold prefill.
- Automation toolset api; 16 response steps, 180 seconds per task, 2048 output tokens.
- Quick Screen limits are stricter than upstream defaults; failures do not establish inability with a longer budget or a different agent.
- Coding 180 seconds generation, 4096 output tokens; offline grading 6 seconds/test and 180 seconds/problem.
- No external API credentials: upstream simulated tool fallbacks apply.
- Gemma thinking disabled on every request; other quality profiles retain prior settings.
- Results measure complete model/quant/runtime/agent configurations, including native draft and tail-skip settings; they are not weights-only comparisons.
- An invalid pilot with additional argument validation was preserved and excluded; all scored runs use corrected v3.0.1 dispatch.
- Accepted requests and responses are archived. Failed HTTP responses have error summaries; their response bodies were not captured.
- Tier 2 and Full Validation have not been run.
64K configured context, explicit 1024 MiB VRAM reserve. Offline nonroot inference with read-only source and models. Simulated workflow tools only; generated code was scored in separate offline disposable containers after inference stopped. No cloud inference/scoring was purchased. Electricity, depreciation and labor are unmeasured.
Tier 2: 30 business tasks, 8 coding problems, 3 Terminal-Bench tasks and 2 SWE-bench Pro issues. Full Validation: complete pinned public suites. Neither tier was started.
Full three-tier v3 framework · Structured results · Scorecard CSV · Frozen task manifest