Selected system / personal and professional work
Experiment in progressModel evaluation and evidence-led optimization
I compare local and hosted models using structured evaluation evidence. Results guide routing and workflow improvements; model fine-tuning remains exploratory.
The problem
General-purpose frontier models can be an expensive default for repeatable tasks. Choosing a smaller model requires evidence about task quality, context size, latency, quantization, runtime, and the cost of correcting failures.
Scope
On a single AMD Strix Halo machine with 128 GB unified memory, I compare local and hosted models across context, quantization, runtime, quality, and routing. Server-hosted Supabase supports structured AI interaction records and vector retrieval. I use interaction records, task outcomes, evaluator and judge feedback, and other sources to assess model performance; those results guide routing and workflow improvements now and inform future fine-tuning and specialist-model experiments. LoRA, CPU-friendly specialists, and a game-companion model remain exploratory. Faster inference and reasoning, lower input, output, and reasoning-token use, and more efficient agent-to-agent communication are design goals, not measured results.
My role
My contribution
I use model evaluation results to guide model selection and workflow decisions, while fine-tuning and specialist-model training remain exploratory.
System design
Architecture and technical decisions
Route by task difficulty, privacy, cost, latency, and demonstrated quality instead of sending every task to the largest available model.
Benchmark small and mid-sized models against the same task evidence and validation criteria used for frontier-model comparisons.
Use AI interaction records, evaluator and judge feedback, task outcomes, and other sources to compare model performance.
Apply evaluation results to routing and workflow improvements; treat fine-tuning, LoRA, and compact CPU-friendly specialists as experiments, not routine model updates.
System boundaries
Constraints and qualifications
- Model quality must be measured against the same tasks and validation criteria before routing changes.
- Context size, quantization, runtime, latency, privacy, and correction cost affect the useful choice.
- Local benchmarks run on one AMD Strix Halo machine with 128 GB unified memory; evaluation evidence informs routing and workflow decisions.
- Optimization goals include lower input, output, and reasoning-token use and faster inference; these are design goals, not measured outcome claims.
- Fine-tuning, LoRA, and CPU-friendly specialists for classification, bounded decisions, completion checks, and narrow reviewer or evaluator tasks remain experiments, not production deployments.
Design approach
Guiding principles
- Use the least expensive intelligence that reliably solves a task.
- Retain validated execution history as an evaluation and dataset asset, not disposable logs.
- Route only when evidence shows a model meets the task quality and operating constraints.
Conceptual architecture
System views
Choose intelligence by the work and its constraints.
I benchmark local and hosted models against task quality, context, latency, and cost. One AMD Strix Halo machine with 128 GB unified memory handles my primary local-AI workload. Specialist models remain experimental.
- idle
- incoming
- active
- outgoing
- settled
Conceptual sequence · not live telemetry.
Execution sequence
- 01Engineering taskEngineering task → Assess constraints
- 02Assess constraintsAssess constraints → Compare evidence
- 03Compare evidenceCompare evidence → Deterministic software · Compare evidence → Small local model · Compare evidence → Frontier model
- 04Deterministic software + Small local model + Frontier modelDeterministic software → Validated outcome · Small local model → Validated outcome · Frontier model → Validated outcome
- 05Validated outcome
Conditional paths
- Compare evidence → Specialist model · Future · after evaluation
- Compare evidence → Human escalation · High risk · uncertainty · when risk requires
- Specialist model → Validated outcome
- Human escalation → Validated outcome
Every run can improve the next system decision.
I use repeatable model evaluations to compare performance and preserve outcomes for analysis.
- idle
- incoming
- active
- outgoing
- settled
Conceptual sequence · not live telemetry.
Execution sequence
- 01Task definitionTask definition → Context used
- 02Context usedContext used → Tool activity
- 03Tool activityTool activity → Candidate result
- 04Candidate resultCandidate result → Validation
- 05ValidationValidation → Independent review
- 06Independent reviewIndependent review → Final outcome
- 07Final outcomeFinal outcome → Durable evidence
- 08Durable evidence
Ongoing learning loop
- 01Durable evidenceDurable evidence → Workflow evaluation
- 02Workflow evaluationWorkflow evaluation → Model routing
- 03Model routingModel routing → Context used
The next pass explicitly revisits Context used → Tool activity → Candidate result → Validation → Independent review → Final outcome → Durable evidence for another pass.
Conditional paths
- Independent review → Human decision · Escalation · acceptance · when review escalates
- Human decision → Final outcome
- Durable evidence → Future specialization · Future training · optional
Intended effect
Intended effect
Repeatable evaluation supports model comparison and evidence-led routing and workflow improvements. Applying that evidence to model weights remains exploratory; speed and token efficiency are goals, not measured results.
Technologies and components
- PostgreSQL
- Supabase
- Vector retrieval
- Local inference
- Quantization
- LoRA experiments