Infrastructure · cybersecurity · AI inference and agent operations · Michigan · remote

I direct AI agents to build and run production infrastructure, and I publish where they fail.

The audit · August 2–3, 2026

13 claims, made by three AI systems and me, checked against live state.

One agent reported critical CVEs down 80%. The rescan said 28%.

View as a table
Audited claims
OutcomeClaims
Held up5
Material error, another system5
Material error, my own claim3
Total13
Critical CVE count
MeasurementCountChange
Baseline210—
Agent reported41−80%
Rescan, live state151−28%

Read the audit

I've spent more than twenty years in automotive manufacturing, much of it leading production teams, and one year as the on-site IT person at the company's second building. That is where I learned process discipline: how to run something the same way every time, and how to find the real cause when it breaks. When I got serious about computers I brought that with me. I started with cybersecurity, then built infrastructure to practice on, and rebuilt it three times, each version more secure and easier to recover than the last.

I'm looking for a team where I can do this work at a larger scale.

Receipts hover or tap a number

19 monitored hosts, source: Prometheus targets, baseline 2026-08-086-node RKE2, source: kubectl get nodes, baseline 2026-08-08~120 pods, source: kubectl get pods -A, counted 117, 2026-10-293-node HA Vault, source: vault status, baseline 2026-08-0823 GitOps apps, source: Argo CD, all Synced and Healthy, 2026-10-29GPU inference, source: llama.cpp and ExLlamaV3 behind LiteLLM, lane roster, 2026-10-29

Selected work

Product · daily driver

Workbench

A coding-agent app: a Go server with its own agent loop, a web app, and a desktop app. Every thread works in its own git worktree, anything that reaches outside it waits for my approval, and plugin harnesses run confined inside a Linux Landlock sandbox. The boundary is in the harness code, not in a prompt asking the model to behave.

~52,200 lines of Go 535 Go tests 40 recorded decisions Private repository · demo on request

Product · close to release

Groundly

A self-hosted research notebook. Every claim in an answer cites the exact passage it came from, and when the sources don't support an answer, it says so. A citation eval gates CI. There is an answer pack for control plans, PFMEAs, and work instructions, the documents automotive manufacturing runs on and ones I've worked with myself.

Apache-2.0 · public release pending

Platform · production

KWS

The private cloud itself. Bare metal through Kubernetes, GitOps, SSO, HA Vault, an internal PKI, and a GPU inference tier. I built it so my own agent platform would have somewhere to run that it could not break.

Incident records and decision logs

Platform · production

Hirdforge

The multi-agent platform Workbench's harness came from, for running coding agents under supervision. Routing is deterministic, work runs in isolated sandboxes, and every change lands as a pull request that only merges when tests and CI pass. It runs continuously on production Kubernetes.

67,115 lines of Go 225 files 382 of 808 commits authored by agents Public repository

Published

How agents fail

I audited thirteen claims that three AI systems made about this estate over two days in August. Eight contained a material error, three of them were mine, and seven of the eight were a correct measurement taken against the wrong thing.

The taxonomy

How I work

I build and run the physical and platform layers, the nine-layer security architecture, and the multi-GPU inference tier myself. I do not write or read application source code (Go, Python, TypeScript), including the Go that Workbench and Hirdforge are written in. For application software and routine operations, I set the architecture and constraints, own the spec and its definition of done, and direct agents to do the work. What I judge is whether the tests pass, what CI reports, and what the cluster actually does after a change lands. I learned that habit over twenty years in automotive manufacturing, nine of them leading production teams of up to 35 people, where a fix counts when the line is running again and not when someone says it is fixed.

The runbooks, incident records, and decision logs live in the repository, so an agent orients itself from the history of the project instead of from a prompt someone retypes each morning. Work happens in isolated branches and sandboxes with git as the rollback boundary, and completion is decided by tests and CI rather than by a model grading its own work.

KWS runs on local agents. Open-weight models on my own hardware (cards up to 32GB, plus Qwen Flash, a 180B model split across CPU and GPU on several machines) handle the normal operations, the smaller projects and campaigns, and the specs, plans and documentation that go with them, roughly 90% of the operations and 80% of the coding. The estate can run on them alone, airgapped if required. I bring in frontier models, currently Claude 5.5 (Opus, Sonnet, and Haiku) and Gemini 4 Argon, for the bigger problems: large project decisions, steering a project that has drifted off track, and root-causing deeper technical issues.

I've been handed wrong reports often enough that the evidence a change worked has to come from the running system. Vault's transit auto-unseal was silently dead for 27 hours while every dashboard showed green, and it only surfaced when someone queried Vault directly. Agents run checks like that one now, and I judge what comes back rather than accepting a summary.

Harnesses ·
t3 code · Claude Code · Codex · opencodecloud + local · Grok · Antigravity · Hermes · Workbenchbuilt it, daily driver · Hirdforgebuilt it

These are the harnesses I drive, on the local open-weight fleet above for the routine work and on a frontier model when a bigger problem calls for one.

What I operate

KWS, the private cloud everything here runs on. Work reaches it through a pull request, models are reached through one gateway, and nothing in the cluster talks to the internet directly.

KWS architecture KWS architecture. Change path: the operator writes the spec, agents open a pull request, Gitea Actions CI runs gitleaks and Trivy and pushes images to an internal mirror of 87 digest-verified images, because the cluster has no internet. Merging the pull request is the deploy, through ArgoCD with 23 self-healing applications. The RKE2 Kubernetes cluster has 6 nodes: 3 control-plane masters with etcd on NVMe, one per mini PC, and 3 workers on two ProLiant DL380 Gen9 servers. In the cluster: 3-node HA Vault, cert-manager with an internal CA, External Secrets, Cilium with deny-all egress, Falco and Talon for runtime detection and automatic quarantine, and a sandbox namespace per pull request. Workloads: Workbench and Groundly. Model calls go through a LiteLLM gateway that routes by role to llama.cpp lanes on 3 GPU hosts, including Qwen Flash 180B split across CPU and GPU. Two cloud roles are opt-in, never a default or a fallback. Beside the cluster, as VMs: an HA front door pair (HAProxy and keepalived) for the API and ingress, Keycloak SSO, the Vault transit VM that holds the unseal key, and monitoring with Prometheus and Loki across 19 hosts. Substrate: OpenNebula/KVM on 5 hypervisors, LINSTOR/DRBD replicated block, restic-encrypted backups to a 7 TB local tier and an append-only offsite archive, and OPNsense with Suricata and a Cisco Nexus 10G switch at the edge. Change path Inference RKE2 Kubernetes · 6 nodes · ~120 pods Platform Workloads Workers: 3 VMs on 2× ProLiant DL380 Gen9 Beside the cluster · VMs Substrate · 19 monitored hosts spec pull request image model calls local: about 90% of the work Operator (me) architecture, spec, merge call Agents Workbench · opencode · t3 coderead the cluster over MCP Gitea + Actions CI gitleaks and Trivy gatesbuild, scan, push Image mirror 87 images, digest-verifiedthe cluster has no internet ArgoCD 23 apps · self-heal on allmerging the PR is the deploy Control plane 3 masters, one per mini PCetcd on NVMe Vault HA ×3 raft storageauto-unseal cert-manager internal CATLS to the pod External Secrets Vault into podsno secrets in git Cilium eBPF network policydeny-all egress Falco + Talon runtime detectionauto-quarantine PR sandboxes namespace per PRdefault-deny Workbench coding-agent appmy daily tool Groundly research notebookcites the passage or abstains LiteLLM gateway routes by role, not modelquotas, cost attribution llama.cpp lanes 3 GPU hosts · role-routedcards up to 32GB Qwen Flash · 180B split across CPU and GPU Cloud roles · opt-in never a default or fallback Front door, HA pair HAProxy + keepalived, one VIPfor the API and ingress Keycloak one SSO for every servicewith its own Postgres Vault transit holds the unseal keyShamir keys kept offline Monitoring Prometheus + Loki, 19 hostsoutside what it watches OpenNebula / KVM 5 hypervisors: 3 mini PCs,2× ProLiant DL380 Gen9 LINSTOR / DRBD replicated block forVM disks and volumes Backups restic, encrypted → 7 TB localtier + append-only offsite Edge OPNsense: firewall, DNS,Suricata IDS · Nexus 10G
change path: merging the pull request is the deploy model calls opt-in cloud roles, never a default or fallback Every box is a running component. Addresses and the IP plan are left out on purpose.

Defense in depth

The same system from the security side. Each layer is built on the assumption that the one in front of it can fail, and each one says how it was tested.

Layer In place How it was tested
Edge OPNsense firewall with Suricata IDS on the WAN. Two front-door gateways on different servers share one virtual IP. Tested: Failover drilled, and fixed until it completes in 13 seconds or less. Jul 2026
Network The cluster has no route to the internet, blocked at the gateway. Apps built on the standard chart run default-deny under Cilium. Tested: A probe from inside an app pod reached its own backend and nothing else: not Vault, not another app, not the Kubernetes API. Sep 2026
Identity and TLS Keycloak single sign-on for every service, per-namespace RBAC, and an internal CA issuing TLS all the way to the pod. Tested: RBAC smoke tests. A hostile tenant probe found the real tenancy boundary is the Argo CD project, not the namespace: alone, a namespace stops nothing, because a tenant pull request could name any destination and the platform would deploy it there. Fixed, proven, and torn down. Jul–Oct 2026
Secrets HA Vault, with the unseal key on a separate transit VM and the Shamir keys kept offline. External Secrets delivers secrets to pods so none live in git, and gitleaks fails any build that tries. Tested: Vault raft snapshots are in the encrypted nightly backups and have been restored in a test. After the total power loss, Vault was unsealed by hand with the offline keys, as designed. Jul–Aug 2026
Supply chain Every image is mirrored internally and pinned by digest. Trivy gates CI and scans the registry into Prometheus on a schedule. Tested: An agent reported an 80% drop in critical CVEs. The rescan said 28%, and 28% is the number on record. The purple-team exercise found an expired registry certificate blocking every image pull, renewed from the internal CA the same night. Aug 2026 · public record →
Workloads Falco watches syscalls on the nodes, and Talon quarantines a pod automatically when a rule fires. Pod Security labels report baseline violations. Tested: Purple-team exercise: an agent dropped and executed a binary in a test pod, Falco caught it, and the pod was isolated. Running the full chain exposed that the response path had been silently dead since a policy port mismatch, fixed and re-verified that night. The detection battery that followed found five Talon rules referencing Falco rule names absent from the deployed 0.44.1 ruleset: four restored from upstream and committed, one excluded on a recorded reason. Oct 2026
Hosts Kernel patching by draining each hypervisor and live-migrating its VMs back. Journals from eight hosts go to Loki, and auditd, fail2ban and AIDE run on most of them. Tested: Patch campaign across all five hypervisors with 22 of 22 VMs and 6 of 6 Kubernetes nodes up throughout. Jul 2026
Agents and automation Infrastructure credentials live on one host. Terraform runs as an identity limited to templates and networks, agents get one Vault subtree on ten-minute tokens, every change is a pull request, and Workbench holds anything that reaches outside a thread’s worktree. Tested: A sweep of the estate found no infrastructure credentials off that host apart from one documented read-only identity. Building the Terraform identity turned up a path to administrator in the first design, which was closed and probed. Aug 2026
Recovery Backups encrypted with restic before they leave the network, kept locally and in an append-only offsite archive, and restore-tested. A cold-start runbook brings everything up in dependency order. Tested: A total power loss came back in about 1h50m with no data loss. Jul 2026 · public record →

Running KWS

KWS runs everything on this page. This is how it is operated day to day, and the record behind each part.

Every change is a pull request

Agents work in branches and open pull requests. Merging is the deploy: ArgoCD applies it within seconds and reverts anything changed by hand. A pull request can get its own sandbox namespace, and rolling back is a git revert.

Decision records, public →

Recovery is written down, and it has been used

A cold-start runbook brings the estate up in dependency order, from the firewall through to Vault. In July a total power loss came back in about 1h50m with no data loss. A deliberate chaos test killed two of the three control-plane members, and the placement gap it exposed was fixed.

Incident records, public →

Backups are encrypted, offsite, and checked

The databases, Vault and every git repository are encrypted with restic before they leave the network, kept on a local tier and in an append-only offsite archive, and restore-tested. When one crontab rewrite silently stopped the Vault and Gitea backups for twelve days, the fix was a watchdog on a systemd timer, not cron, that asks restic itself for the last success of each restic-backed job, because a fresh log line is not evidence of a backup.

Backup runbook and report, August 2026

The front door fails over

Two gateway VMs on different servers share one virtual IP through keepalived and HAProxy, in front of both the Kubernetes API and the ingress. Failover was drilled, investigated for speed, and fixed until it completed in 13 seconds or less.

Failover drill reports, July 2026

Patching without taking anything down

In July every hypervisor was drained, rebooted onto a new kernel and refilled by live migration: 22 of 22 VMs running and all six Kubernetes nodes Ready throughout. Separately, an RKE2 and Cilium upgrade closed 24 critical CVEs with no workload outage.

Patching campaign report, July 2026

Models earn their place

Local coding sessions run concurrently across three GPU machines, routed by role so no app names a model. New models are benchmarked before they get a lane, and one was taken back out the same day because the coding tools could not yet parse its tool calls.

Inference lane reports, Aug–Sep 2026

Records marked public are in kitporath/kws. The others are dated reports in the private infrastructure repository, and I'm happy to walk through any of them.

The hardware

Decommissioned enterprise servers and consumer GPUs in my workshop. The rack holds the two servers that carry the cluster's workers and platform VMs, the 10G switch, and two of the three inference machines. The control plane sits on three mini PCs beside it.

The rack, front elevation A 20U rack. From the top: a blank panel, a CAT6 patch panel, a Cisco Nexus 9372TX-E switch, a blank, baldar (ProLiant DL380 Gen9, 2U), a blank, thor (ProLiant DL380 Gen9, 2U), a blank, frey (Lenovo ThinkStation, 4U on a shelf), a blank, and vidar (a custom 4U build on a shelf) at the bottom. ASGARD 2019181716151413121110987654321 CAT6 patch panel Cisco Nexus 9372TX-E baldar · ProLiant DL380 Gen9 thor · ProLiant DL380 Gen9 frey · Lenovo ThinkStation vidar · custom 4U build
Cisco Nexus 9372TX-E
The 10G fabric for the estate, with 40G uplinks. vidar connects at 40GbE.
baldar · ProLiant DL380 Gen9
Xeon E5-2695 v4, 72 threads, 125 GB. Hypervisor for Kubernetes workers, platform VMs, and one half of the front-door pair.
thor · ProLiant DL380 Gen9
Xeon E5-2690 v3, 48 threads, 125 GB, 4 TB SATA drives. Hypervisor, the other half of the front-door pair, and the 7 TB local backup tier.
frey · Lenovo ThinkStation
RTX 3090 24 GB. One inference lane.
vidar · custom 4U build
Ryzen 9 5900X, 94 GB, an RTX PRO 4000 Blackwell 24 GB and an RTX 5060 Ti 16 GB on 40GbE. One inference lane, and home to the agent layer and the nightly backup and CVE-scan jobs.

Beside the rack

Three mini PCs
Two Ryzen 3 PRO 2200GE and one i5-6500T. One Kubernetes master each, with etcd on NVMe, so losing either big server cannot take quorum with it. One of them also runs the OpenNebula front-end and the LINSTOR controller.
OPNsense
Router, firewall, DNS and Suricata IDS. The first thing up in a cold start.
loki · admin workstation
Radeon AI PRO R9700 32 GB and RTX 3060 12 GB. The estate's largest inference lane, a narrow trial seat, and the console the cold-start runbook is worked from.

The rack runs on a UPS. Photo of the rack

Four systems

Newest first. The first three are the infrastructure generations, and Hirdforge is the platform they carry.

PlatformHow it was builtWhere it is now
KWSbare metal to Kubernetes, GitOps, SSO, Vault, GPU inference agent-built under my direction production · survived a power loss and a chaos test
Hirdforgemulti-agent engineering platform agent-built under my direction production Kubernetes · running continuously
AsgardProxmox, Talos, full rack, 10G built with AI collaboration completed · runs Hirdforge in production
Valhallak3s on mini PCs by hand retired · those machines now run the KWS control plane

Contact

I'm looking for AI platform, infrastructure, and forward-deployed roles: teams running AI infrastructure, teams bringing coding agents into real engineering work, and companies that need models self-hosted. Remote preferred. US citizen, no sponsorship needed, based in Michigan on Eastern Time.

Resume · Evidence index · llms.txt · resume.json

This site is written for people and readable by agents. Every figure on it has a dated source.