Glad to see this coming with gVisor support to help secure the Kernel- IMHO we should expect frontier models to find Kernel exploits.
I am working on a project similar in spirit that uses microsandbox (libkrun) to run inside a tiny and fast VM. It includes other security properties that are needed for some workloads.
You definitely want to do that. I have a Github App that I use for my AI agents, and that has its own associated restricted credential.
There are going to be cases where you want to white list an org or a repo for read access that is not under your control and Github filtering will be a simple way to do that.
Github can be a source of hostile code, prompt injections, and exfiltration- you may want to lock down using repos that aren't yours.
The tool inherited this feature from the prior implementation and its something I am still exploring.
AI tells me that code using the verus! macro everywhere would double in build time but if only used ocassionally the verus! macro would only increase build time by a few percent. The attribute annotation would basically be free, but there is a downside that loop invariants would be rejected by stable rustc.
I use a workflow that has different named subagents. [1] Agent profiles can be pinned to models. So you set the model you want on your main thread as the orchestrator. Create an agent for the "planner", "implementer", and "reviewer" and set the model you want for each. Right now I am orchestrating and implementing with Deepseek, planning with Astra, and reviewing with Opus.
I am doing this with the Pi harness right now. To use a Claude monthly plan you need to use the pi-claude-bridge plugin.
If you are using just Claude for example you can use Sonnet as the implementer and Fable/Opus as the planner.
They state Luna is good enough, but its accuracy of findings is 74% whereas Astra is 96%. Dealing with false positives is expensive.
I am finding AI doing its own reviews as part of the process to be the key to productivity. I do subagent (fresh context reviews) at multiple stages with well-specified review criteria. It is really expensive to do with OpenAI or Claude API billing. Deepseek or the discounted monthly plans from OpenAI or Claude can be discounted similar to the 28x they state for Luna compared to Astra and you maintain much higher quality.
He likes JetKVM. However, they appear to be sold out. The Mini seems great but I have yet to receive a preordered item on the advertised timeline.
That article pointed to ArkKVM which is a hardware clone of JetKVM but they have now released their own software stack as open source which has Tailscale support
I am already a zellij user and not sure if I want to drive everything from an IDE instead. I definitely see the appeal though.
I like the discoverability of the text prompt commands.
I like that the terminal is more of a first class citizen.
I like that I can run this with just `go run`.
The themes are pretty bad right now IMHO. I use solarized/gruvbox themes- both light and dark.
Amazing work! But deeply frustrating on the lack of reproducibility and how far off this is from a proper SDLC.
How can I inspect exactly how agents were prompted? How can I reproduce this setup and try out my own AI setup? Am I missing a link to a repo somewhere?
These evals probably match how many people are using AI, but its not the full package of how we know software development needs to be done, and not how I do it with AI. The closest would be "Audit" which scored the highest when using xhigh- that actually incorporates a review cycle- something we know is the most important part of the software development process for code correctness (design/specification is not as much of an issue in this problem since the task is to write code against an existing spec). However, we don't know what instructions they have in their "Audit".
I would love to benchmark my own flow [1] if I can be given their exact problem. What it does is (assuming there is already a solid spec)
* plan with expensive model. Review the plan.
* implement with cheap model. Review for spec compliance and code quality.
* Reviews are done adversarially from the expensive model with a fresh context.
* ensure that verifications (automated or manual) are performed.
For non-trivial changes, the review and verification process almost always catch significant issues.
The workflow does use TDD. I do find useless tests being written and I need to dig into this part of the workflow a lot more, so its great to see that aspect of this article. My experience writing software has taught me that code must be written to be easy to test, but not necessarily done TDD style.
As someone who also spent a bunch of time benchmarking various techniques, this is as good as you can do without publishing a formal versioned benchmark suite that you want to maintain and run forever at immense cost to yourself. If you actually go to benchmark your own flow as you mentioned, you will quickly run into like a dozen problems that discourage you from publishing.
- Are you sure that temperature and other nondeterminism isn't affecting your output?
- Are you sure you're not being routed through an A/B test at this moment?
- Are you sure there's not a bug affecting the model at this moment?
- Are you sure that you picked the right model and effort level?
- Are you sure that your result generalizes across providers?
- Are you sure that you set up the correct level of sandboxing and the agent can't e.g. look at a sister directory or git history in the current directory for answers?
- Are you sure that the agent isn't leaking answers in memory or its conversation history?
- Are you sure that tool calls aren't somehow affecting results?
- Are you sure that your results are robust, i.e. you see the same results with mild tweaks to the prompt?
- Are you comfortable keeping your blog post live when your results are invalidated next week with the next model launch?
And that's just a quick list off the top of my head.
I personally decided that it wasn't worth it, I'm glad that Dan decided to publish his. Frankly I think we could use a lot more of these "I ran these 2 techniques side by side and here's what I saw" anecdata, because most people who promote prompt techniques can't produce a single prompt they ran twice because they never actually tested it per se.
I am really grateful that he’s publishing as well.
I am just asking for the bare minimum to actually understand what has been tested and for reproducibility of methods- the publication of the prompts. It would take a lot less time than all the guess work analysis write up and be a lot more useful.
Without the prompts the rest of your questions about reproducibility are moot.
This post boils down to, "I tried a bunch of things that didn't work very well". But we don't actually know what he tried. Everything hinges on Dan's prompting skills.
Get them under version control.
I have a git repo with my skills for software development [1]. There is an installer script that symlinks to the skills. Updating the skills on a machine is then just a matter of advancing the git repo. One of the skills comes with some bash scripts, but the rest are effectively just prompts.
Putting project specific skills in projects works well.
I make sure they work by understanding every skill, reviewing pull requests, and testing the end product. The result is rarely perfect, so I am constantly tweaking the skills and how I use AI.
The way to control LLM memory with existing tools now is to ask it to write out a file with all relevant information (this could include explicit retraction instructions). Then clear out the context. Basically /compact.
I am working on a project similar in spirit that uses microsandbox (libkrun) to run inside a tiny and fast VM. It includes other security properties that are needed for some workloads.
https://github.com/gregwebs/agent-vm/#agent-vmreply