Training a harness for model-agnostic and task-environment-agnostic capability improvements with PyTorch-like framework [P](reddit.com)

I worked on this project (https://github.com/workofart/harness-training) for the past few months to reframe "Agent-driven Self-improving Harness" to "Harness Training". The idea is simple, the harness is trained once with a frozen task LLM against a given task environment. Then you can then swap out the task LLM to any model and evaluate the "frozen trained harness" with any task LLM on any new task environment. Since this was a general problem, I took the chance to create a general PyTorch-like training framework. Right now, you can train with any OpenAI-compatible API for interfacing with the task LLM and train against Terminal-Bench or SWE-Bench tasks, but you can easily extend it to support any task environments. criterion = StrictPareto() optimizer = GreedyMonotonic() trainer = Trainer( config_path="config/train_harness.yaml", estimator=AgenticEstimator( backend=CodexAgentBackend(...) ), criterion=criterion, optimizer=optimizer, ) for loss in trainer.epochs(30): # Records the baseline-vs-candidate verdict loss.backward() # Optimizer either fast-forwards the candidate change (git commit) as a new baseline or rejects it (preserved as git ref) optimizer.step() I wrote a blog post (https://www.henrypan.com/blog/2026-07-18-harness-training) on this journey, including (but not limited to): results from using this harness training framework to improve general capabilities across many task LLMs to beat Terminal Bench 2.0 (Terminus Harness) and also transfer learnings towards better task-solving abilities in unseen task environments (e.g. harness trained on SWE-Bench tasks solving Terminal Bench tasks). how this framework is built learnings on what was missing in my initial version of the project (hint: determinism) Any feedback is appreciated. Thanks! submitted by /u/Megadragon9 [link] [Kommentare]