Writing
14 posts
- Re-load-testing the AI tutor - the speed prefix cache buys, and measuring quality under load
Against the deployed prod vLLM, with the real hint payload. After swapping to Qwen3.6 I load-tested again, isolated how much of TTFT prefix cache buys by turning it on and off, and scored format compliance automatically.
- Only login was broken - a container that resolved DNS but could not get out
Only Google login failed with a server-configuration error. The cause was not callback registration but a container that could not reach the internet - it resolved DNS but could not connect. Narrowing through env, prisma, ufw, MTU, and docker0 vs the compose bridge to find a lost NAT rule, then clearing tangled iptables state with a clean reboot.
- I broke working code by fixing it - self-closing cells in a regex xlsx parser
The dashboard target lines vanished. The culprit was a commit I had made days earlier, "fixing" an off-by-one that was not one. Our regex xlsx reader absorbed a neighbor cell value through a self-closing empty tag, and the original code had been correct for exactly that reason.
- The PSU died and the server went dark - scoping data loss before recovery
The production server PC lost its PSU and the site went down. The urgent thing was not recovery but data. Comparing the freshness of the live volume, the daily backup, and a stale cloud snapshot, and why replacing just the power supply was zero-loss and least hassle - plus the belated realization that every backup lived on the same disk.
- From Vercel/Supabase to on-prem - migrating student data to a campus server
Moved a production CMS off the cloud (Vercel + Supabase) onto an on-campus server. The @ in the DB password, why the TLS cert would not issue (the campus firewall, not the server), self-hosted Auth.js UntrustedHost, and an OAuth client living in the wrong Google project.
- When CI commits to its own repo - a race fixed with concurrency serialization
Merging PRs back-to-back made the image-tag auto-commit job fail on rebase conflicts. Fixed with concurrency serialization and idempotent reapply instead of merge.
- Why a production pod restarted 398 times - a liveness probe vs image release skew
Restarting 398 times while the app booted fine. Narrowing from kubectl logs → describe → replicaset → git → exec to find a health-check path deployed before the image that served it.
- A two-way Markdown WYSIWYG - data-source-line and postMessage
To let non-developers write without learning Markdown, the left editor and the right preview had to edit each other both ways. The walls I hit building a bidirectional WYSIWYG, and how I got past them.
- A serverless distributed edit lock built from git commits
The GitHub Contents API has no file locking. Here is how I stopped concurrent editors from clobbering each other - no backend, no DB, just .locks commits and a heartbeat.
- A WAL runaway filled the disk - recovering a CNPG/Longhorn cluster
WAL filled the disk and PostgreSQL would not start. A dead standby on a low-capacity node had pinned WAL through its replication slot. A layered recovery across CNPG, Longhorn, and node devices - and why "HA" without backups was never HA.
- From Ollama to vLLM - scaling an AI tutor to 30 concurrent users
From codellama to Qwen3.6, from Ollama to vLLM. Measuring every step with Locust to make a local-LLM AI tutor answer within 3 seconds - and the model decisions along the way.
- The link itself was the key - sharing a 7-day journal summary safely
I had to hand a struggling user’s 7-day journal to a counselor who has no app account - so the link itself became the credential. Storing only a token hash and freezing the data into a snapshot.
- The fish sank into the garden - layer rendering in the mind garden
A garden where flowers sit on soil and fish swim in a pond. Flat coordinates could not decide what covers what. How I assigned each item an integer layer, sorted, and stacked them.
- The LLM will fail eventually - why I put a fallback under emotion analysis
Emotion analysis with GPT-4o mini was the core of the app. But the client failed to init, and asking for JSON did not guarantee JSON. How I stopped treating an LLM response as reliable infrastructure.