Skip to content
Return to Mission Control

Selected system / personal and professional work

Experiment in progress

Model evaluation and evidence-led optimization

I compare local and hosted models using structured evaluation evidence. Results guide routing and workflow improvements; model fine-tuning remains exploratory.

The problem

General-purpose frontier models can be an expensive default for repeatable tasks. Choosing a smaller model requires evidence about task quality, context size, latency, quantization, runtime, and the cost of correcting failures.

Scope

On a single AMD Strix Halo machine with 128 GB unified memory, I compare local and hosted models across context, quantization, runtime, quality, and routing. Server-hosted Supabase supports structured AI interaction records and vector retrieval. I use interaction records, task outcomes, evaluator and judge feedback, and other sources to assess model performance; those results guide routing and workflow improvements now and inform future fine-tuning and specialist-model experiments. LoRA, CPU-friendly specialists, and a game-companion model remain exploratory. Faster inference and reasoning, lower input, output, and reasoning-token use, and more efficient agent-to-agent communication are design goals, not measured results.

My contribution

I use model evaluation results to guide model selection and workflow decisions, while fine-tuning and specialist-model training remain exploratory.

Architecture and technical decisions

  1. Route by task difficulty, privacy, cost, latency, and demonstrated quality instead of sending every task to the largest available model.

  2. Benchmark small and mid-sized models against the same task evidence and validation criteria used for frontier-model comparisons.

  3. Use AI interaction records, evaluator and judge feedback, task outcomes, and other sources to compare model performance.

  4. Apply evaluation results to routing and workflow improvements; treat fine-tuning, LoRA, and compact CPU-friendly specialists as experiments, not routine model updates.

Constraints and qualifications

  • Model quality must be measured against the same tasks and validation criteria before routing changes.
  • Context size, quantization, runtime, latency, privacy, and correction cost affect the useful choice.
  • Local benchmarks run on one AMD Strix Halo machine with 128 GB unified memory; evaluation evidence informs routing and workflow decisions.
  • Optimization goals include lower input, output, and reasoning-token use and faster inference; these are design goals, not measured outcome claims.
  • Fine-tuning, LoRA, and CPU-friendly specialists for classification, bounded decisions, completion checks, and narrow reviewer or evaluator tasks remain experiments, not production deployments.

Guiding principles

  • Use the least expensive intelligence that reliably solves a task.
  • Retain validated execution history as an evaluation and dataset asset, not disposable logs.
  • Route only when evidence shows a model meets the task quality and operating constraints.

System views

Choose intelligence by the work and its constraints.

I benchmark local and hosted models against task quality, context, latency, and cost. One AMD Strix Halo machine with 128 GB unified memory handles my primary local-AI workload. Specialist models remain experimental.

Models — conceptual system topologyAn architectural illustration, not live telemetry or model availability. Prefer deterministic software or a small capable model; use frontier reasoning when the task warrants it, based on quality and constraints. Execution mode: once. Components: Engineering task (Difficulty · failure cost); Assess constraints (Privacy · context · latency); Compare evidence (Quality · cost · reliability); Deterministic software (Known structure · rules); Small local model (Private · fast inference); Specialist model (Future · after evaluation); Frontier model (Complex reasoning); Human escalation (High risk · uncertainty); Validated outcome (Route from evidence). Ordered stages: Engineering task; Assess constraints; Compare evidence; Deterministic software and Small local model and Frontier model; Validated outcome. Connections: Engineering task to Assess constraints; Assess constraints to Compare evidence; Compare evidence to Deterministic software; Compare evidence to Small local model; Compare evidence to Specialist model (conditional); Compare evidence to Frontier model; Compare evidence to Human escalation (when risk requires) (conditional); Deterministic software to Validated outcome; Small local model to Validated outcome; Specialist model to Validated outcome (conditional); Frontier model to Validated outcome; Human escalation to Validated outcome (conditional). Optional conditional paths are shown but do not run in the illustrated sequence.Engineering taskDifficulty · failure costAssess constraintsPrivacy · context ·latencyCompare evidenceQuality · cost ·reliabilityDeterministicsoftwareKnown structure · rulesSmall local modelPrivate · fast inferenceSpecialist modelFuture · after evaluationFrontier modelComplex reasoningHuman escalationHigh risk · uncertaintyValidated outcomeRoute from evidence
  • idle
  • incoming
  • active
  • outgoing
  • settled

Conceptual sequence · not live telemetry.

Execution sequence

  1. 01
    Engineering taskEngineering task → Assess constraints
  2. 02
    Assess constraintsAssess constraints → Compare evidence
  3. 03
    Compare evidenceCompare evidence → Deterministic software · Compare evidence → Small local model · Compare evidence → Frontier model
  4. 04
    Deterministic software + Small local model + Frontier modelDeterministic software → Validated outcome · Small local model → Validated outcome · Frontier model → Validated outcome
  5. 05
    Validated outcome

Conditional paths

  • Compare evidence → Specialist model · Future · after evaluation
  • Compare evidence → Human escalation · High risk · uncertainty · when risk requires
  • Specialist model → Validated outcome
  • Human escalation → Validated outcome
Conceptual execution · explicit dependencies · not live telemetry

Every run can improve the next system decision.

I use repeatable model evaluations to compare performance and preserve outcomes for analysis.

Evidence and learning — conceptual system topologyAn architectural illustration, not live telemetry. Model comparisons inform workflow decisions now and may guide future fine-tuning experiments. Execution mode: continuous. Components: Task definition (Goal · constraints); Context used (Sources · provenance); Tool activity (Calls · outputs); Candidate result (Proposed change); Validation (Tests · deterministic checks); Independent review (Findings · confidence); Human decision (Escalation · acceptance); Final outcome (Verified · stopped · revised); Durable evidence (Structured execution history); Workflow evaluation (Reliability · failure analysis); Model routing (Quality · cost · latency); Future specialization (Future training · optional). Ordered stages: Task definition; Context used; Tool activity; Candidate result; Validation; Independent review; Final outcome; Durable evidence. Feedback loop: Durable evidence; Workflow evaluation; Model routing; the next continuous pass explicitly revisits Context used; Tool activity; Candidate result; Validation; Independent review; Final outcome; Durable evidence. Connections: Task definition to Context used; Context used to Tool activity; Tool activity to Candidate result; Candidate result to Validation; Validation to Independent review; Independent review to Human decision (when review escalates) (conditional); Human decision to Final outcome (conditional); Independent review to Final outcome; Final outcome to Durable evidence; Durable evidence to Workflow evaluation (feedback); Workflow evaluation to Model routing (feedback); Durable evidence to Future specialization (conditional); Model routing to Context used (feedback). Optional conditional paths are shown but do not run in the illustrated sequence.Task definitionGoal · constraintsContext usedSources · provenanceTool activityCalls · outputsCandidate resultProposed changeValidationTests · deterministicchecksIndependent reviewFindings · confidenceHuman decisionEscalation · acceptanceFinal outcomeVerified · stopped ·revisedDurable evidenceStructured executionhistoryWorkflow evaluationReliability · failureanalysisModel routingQuality · cost · latencyFuture specializationFuture training · optional
  • idle
  • incoming
  • active
  • outgoing
  • settled

Conceptual sequence · not live telemetry.

Execution sequence

  1. 01
    Task definitionTask definition → Context used
  2. 02
    Context usedContext used → Tool activity
  3. 03
    Tool activityTool activity → Candidate result
  4. 04
    Candidate resultCandidate result → Validation
  5. 05
    ValidationValidation → Independent review
  6. 06
    Independent reviewIndependent review → Final outcome
  7. 07
    Final outcomeFinal outcome → Durable evidence
  8. 08
    Durable evidence

Ongoing learning loop

  1. 01
    Durable evidenceDurable evidence → Workflow evaluation
  2. 02
    Workflow evaluationWorkflow evaluation → Model routing
  3. 03
    Model routingModel routing → Context used

The next pass explicitly revisits Context used → Tool activity → Candidate result → Validation → Independent review → Final outcome → Durable evidence for another pass.

Conditional paths

  • Independent review → Human decision · Escalation · acceptance · when review escalates
  • Human decision → Final outcome
  • Durable evidence → Future specialization · Future training · optional
Conceptual execution · explicit dependencies · not live telemetry

Intended effect

Repeatable evaluation supports model comparison and evidence-led routing and workflow improvements. Applying that evidence to model weights remains exploratory; speed and token efficiency are goals, not measured results.

Technologies and components

  • PostgreSQL
  • Supabase
  • Vector retrieval
  • Local inference
  • Quantization
  • LoRA experiments