EMPIRICAL EVALUATION HARNESS · REPRODUCIBLE BENCHMARKS

Verified Against Real Codebases, Not Marketing Claims

Every metric published on Mindaeon is reproduced continuously by our evaluation harness across Appliance Sizing, Foundry Verification Gates, AST Mutation Testing, Hybrid Code Intelligence, Multi-Worktree Radar, and Shadow FinOps.

SUITE 01Appliance Memory & Sizing Suite
1,091 MiB

Idle Appliance RAM Footprint

vs 16–32 GiB multi-node appliances

Complete 8-service sovereign stack (Keycloak 683 MiB, PostgreSQL 18 158 MiB, Gitea 93 MiB, SeaweedFS 78 MiB, Control Plane 29 MiB, OpenBao 26 MiB, Edge + OPA 25 MiB) verified at idle.

SUITE 02Foundry Verification Gate Suite
100% (7/7)

Test-Tampering Interception

0% false positives on legitimate edits

Blocks test file deletion, .skip/.only injection, assertion removal, and snapshot tampering outside the agent sandbox.

SUITE 03AST Mutation Verification Suite
3.77 ms p95

AST Mutation Verification

13.3x faster than 50 ms budget

Deterministic changed-surface AST mutations verify that new tests genuinely fail when production logic is broken.

SUITE 04Hybrid Code Intelligence Suite
71.2%

Prompt Token Reduction (3.5x vs grep)

94% symbol recall (vs 62% raw grep)

Hybrid Tree-sitter AST + SCIP/LSP call graph + pgvector semantic embeddings fused with Reciprocal Rank Fusion (3.5x fewer tokens than raw grep).

SUITE 05Multi-Worktree Radar Suite
76.0%

Merge Conflict Reduction

25 conflicts reduced to 6

Continuous Shadow Merge dry-runs across parallel git worktrees detect overlapping AST symbols in p95 = 0.01 ms.

SUITE 06Shadow FinOps Evaluation Suite
50.0%

Shadow FinOps Cost Savings

$0.0155 → $0.0077 per workload

Shadow model replay and deterministic prompt prefix ordering maximize cache hit rates without quality regression.

METHODOLOGY 01 // OUT-OF-SANDBOX TEST INTEGRITY & AST MUTATION

Why Exit-Code CI Fails for Autonomous Coding Agents

When an autonomous agent shares write access to both production source files and test suites inside a container, it can trivially satisfy a failing test by deleting the test file, injecting .skip / .only, or weakening an assertion. Mindaeon prevents this structurally: the Host Test Integrity Guard evaluates AST diffs outside the execution sandbox (intercepting 100% (7/7) tampering vectors with 0% false positives), and the Changed-Surface AST Mutation Harness injects deterministic syntax mutants on modified lines in p95 = 3.77 ms to prove that tests actually fail when logic is broken.

METHODOLOGY 02 // HYBRID CODE INTELLIGENCE (RECIPROCAL RANK FUSION)

71.2% Prompt Token Reduction (3.5x Fewer Tokens vs. Raw Grep)

Agentic loops that rely solely on raw ripgrep / globbing burn massive context windows reading irrelevant files. Mindaeon fuses three complementary indexes using Reciprocal Rank Fusion (RRF): Tree-sitter AST structural scopes, SCIP / LSP exact cross-file call graphs, and PostgreSQL 18 pgvector semantic embeddings—achieving 94% symbol recall (vs. 62% for raw grep) while reducing prompt token consumption by 71.2%.

METHODOLOGY 03 // APPLIANCE MEMORY & SIZING SUITE (1,091 MiB IDLE RAM)

8-Service Sovereign Stack Memory Breakdown

683 MiB
Keycloak OIDC / SAML Broker
Identity & Federation
158 MiB
PostgreSQL 18.6 + pgvector
State, RLS & Vector Index
93 MiB
Gitea Internal Git Server
Air-Gapped Mirror & PRs
78 MiB
SeaweedFS Object Store
Artifact & Bundle Storage
29 MiB
Fastify Control Plane
Orchestration & API Gateway
26 MiB
OpenBao Transit KMS
Cryptographic Signing & Secrets
25 MiB
Edge Gateway + Open Policy Agent (OPA)
Zero-Trust Policy & Ingress