We’re slowly learning that agents are like us: they’re much more efficient at solving problems if the problems are formulated perfectly.

At least this is what we see with tasks that perform extraordinary well with agents: if you give them a pass/don’t pass test, and a durable session, they will work for hours, days, weeks, months, to find you the best possible algorithm/implementation.

For example: “implement the best sorting algorithm” and let the agent run the benchmark itself.

The problem is that the pass/don’t pass test can be broken, and agents love to cheat (it appears so).

So your test needs to include not only that your sorting algorithm is fast, but that it is correct on ALL inputs.

For such a scenario, verifying that the list output is sorted might appear to be enough, but it’s not: the agent could produce something that is correct only some of the times.

So the better test is something more like a quickcheck/proptest: you generate many inputs, and check that all outputs are correctly sorted. This is usually fine enough.

For example, see how Yukon had people optimize a zero-knowledge proof (ZKP) system (called Flock), and how it verified submissions by generating random inputs and checking if the optimized candidates would produce valid ZK proofs.

In other scenarios, when you’re dealing with adversarial inputs, you can’t risk that an algorithm is X% correct where X is “not close enough” to 100.

In these scenarios, formal verification is the savior. This is what my coworkers did with zk.golf where people battle to optimize ZK circuits and a service verifies that submitted circuits are sound and complete (basically secure) with a mathematical proof written in lean (based on our framework clean).

In the last year I’ve been working on zkao, an AI security engineer that people can “hire” to continuously look for bugs in their codebases. The bugs that zkao attempts to find are contained in the hardest of software: cryptographic software.

I’ve learned the hard way that finding bugs in cryptography software is hard because of two reasons:

  1. it’s not always clear what the threat model is
  2. it’s not always clear what security properties the system requires

To illustrate the first problem, let me give you an example. We sometimes look at libraries that expose public-key cryptography algorithms and accepts elliptic curve points. zkao used to (and probably still do sometimes) report lack of subgroup membership verifications as bugs. If that doesn’t make sense to you, I’ll reformulate it as: things don’t work too well if the inputs are not trusted, as they are not sanitized in the implementations.

In many of my past audits, I’ve reported these kind of bugs as low-severity findings depending on the context, and especially when I didn’t find better bugs. When you don’t find good bugs as part of an audit, you’re always a bit ashamed, and so you try to pad with OK-bugs. But these days, people hate it. Perhaps they always did, but it’s especially worse now because their own vanilla agents will report those as well. And what your SOTA security experience does by default is what I call vibe auditing.

So anyway, I don’t want zkao to report such bugs most of the time, but sometimes maybe it should. It really depends on the threat model of the library, and how it’s used in the overall app, and so on. So before we can figure out if these are bugs, and exploitable in practice, we need to understand the overall context of the application, and this is a whole different problem. It’s a problem of integration and context acquisition, and you need more than a prompt to fix that.

That’s perhaps the main reason why vanilla models are often unusable due to the wall of false positives or low findings they report.

To illustrate the second problem, “it’s not always clear what security properties the system requires”, think about the fact that a lot of cryptographic protocols are poorly specified in the first place and it is not clear what security properties one should look for in them. For example, many confidential transfer protocols have had a hard time formulating what properties they need for their protocols to be secure. People used to list what seemed like random security properties, without realizing if one encapsulated others. For example, what it took was some brainstorming and out-of-the-box thinking when Zooko thought about and found the Faerie gold attack.

Screenshot 2026-11-17 at 20.33.46.png

Zcash Protocol Specification Version v2026.9.0-105-gf217a3

A way to prove protocols secure in cryptography is to model their ideal functionality, and then compare that to the real-world spec, and see if someone could observe a trace and distinguish in which world it happened (in the real world or in this ideal world). But even then, the goal of finding the right security properties has shifted to finding the right ideal functionalities (and there are often many).

Even in “simple” cryptography, it took some time for people to figure out the right way of modeling core primitives. For example, Neal Koblitz and Alfred Menezes wrote an entire paper called ANOTHER LOOK AT SECURITY DEFINITIONS (in titlecase).

Finding the “right” definition of security is often elusive. Concepts such as symmetric and public-key encryption, hash functions, and message authentication have been around for decades, but the choice of security models is still fraught with unsettled issues.

For example, it was thought that Duplicate-Signature Key Selection (DSKS) attacks were not really an interesting property of signature schemes:

It must be emphasized that possession of the duplicate-signature key selection property does not constitute a weakness of the signature scheme — the goal of a signature scheme is to be existentially unforgeable against an adaptive chosen-message attack.

yet it turned out that a beautiful DSKS attack completely broke the Let’s Encrypt protocol. For the anecdote, I missed this DSKS attack as part of my very first audit (of Let’s Encrypt) in an internship at NCC Group in 2015.

Another example is DJB famously arguing against the need to check the cofactor of its X25519 curve, which also led to the break of the end-to-end encryption protocol of Matrix (which I found (I can’t miss them all), and talk about in more detail in my book Real-World Cryptography).

Anyway, it seems like if you lay out the threat model correctly, and you figure out all the security properties you want to prove, agents can do miracles. Figuring out exactly what these pass/don’t pass test look like is not easy, but it really gets you 90% of the way.

Since I haven’t seen much ink being spilled on the subject, I’ll take this opportunity to name this test after myself: I call it the Wong test. I’m ashamed at my own self-importance, but you gotta try to name something after you at one point in your life, and I’m getting older.

Part of figuring these Wong tests is that even once you have them, it might not be enough… Because you need to convince yourself that you found them all!

There is one final “advantage” of precise definitions that is rarely mentioned in discussions of the subject. They are helpful for the attackers. Like everyone else, the adversaries have limited resources and a desire to concentrate them where they’ll do the most good (or rather, the most bad). When publicity about a protocol specifies the security model that served as a guide for its design and a basis for the security guarantee, adversaries can also use that model as a guide, focusing their efforts on devising attacks that are outside the model.

suggested reads:

It seems like as a codebase changes, or as models get better, some prompts are good to rerun. You know, like a periodic “find all the bugs” prompt, although I wouldn’t recommend that one (security is hard, and there are better tools like www.zkao.io out there to find bugs using AI).

Anyway, I call that concept agent gardening, and it’s a mix of prompts that help maintain a codebase in order by constantly refactoring or cleaning up code.

I’m curious as to what people think are good prompts to run periodically. Mine live in a file called GARDENING.md where each prompt is accompanied with a date/commit next to it to understand when the last time we ran it was, and what was the outcome of that run.

To give you an idea, here are some of the prompts that live in that file:

  • should we update any of our dependencies?
  • is there dead code we should remove?
  • do we duplicate code and make some things overly complicated, which could benefit from some abstraction (an interface, a constructor, a common library/package, etc.)
  • are we lacking e2e testing in some area?
  • review past agent transcripts produced by agentic code in this codebase and figure out if there are things that we do that consume way too many tokens in the context of our agents, maybe commands, or prompts, or something that adds tons of tokens for no good reason
  • is there anything in your memory that could be useful to hardcode into AGENTS.md for other coworkers to benefit?
  • are the docs up-to-date?
suggested reads:

I’ve realized that not many people are using this kind of techniques for “it’s 2027 soon, agentic codebases need a different kind of testing”, and I’ve been told I should document it somewhere at some point, so here it is.

A lot of tests in my codebases are not written as code, they’re written as scenarios in a SCENARIOS.md file.

Agents know that they should maintain a manageable list of such scenarios that describe end-to-end tasks and expected behavior from different point of views.

Agents can decide to run through all of them, or some of them, after code changes in order to verify that some expected high-level specification is still respected.

To do that, agents have access to two things:

  1. A codebase CLI that has commands to speed up running through these scenarios. Things like adding the right data in the database, running well-known db queries with the right settings, spinning up redis, etc. No end-to-end tests, just atomic things.
  2. A way to orchestrate any LLM responses, so that I pay sub cost instead of actually calling the token API. Usually I do this by having my own LLM provider in pi, and a codebase CLI command that creates the server side of things and let the LLM interact with it to see requests and produce responses.

That’s it. I call it ADT for Agent-Driven Testing.

suggested reads:

As a small team, we always have a million things to focus on. In my mind, I sometimes think that we could expand our team in order to be able to brainstorm all these issues we’re trying to optimize: how can we have better oracles to query our codebases, how can we get our workers to start faster, how can we spend our budget more intelligently across tasks that we need to run, how can we cache artifact builds more efficiently across workers, etc.

A large company like Meta would have people focusing on these different parts of the stack, and coming up with clever algorithms to optimize things ad-infinitum. On the other hand, a big team has trouble moving fast, they’re constantly fighting against an aging codebase, and coworkers are fighting against other coworkers, and also nobody dares making changes too large as often nobody understands the whole codebase and is scared of breaking things.

Making important changes is fast and easy with a small team, we already knew that. Today, coding agents make this even more insane.

Wait, we have coding agents, why don’t we use these to solve all of these problems? Well, with coding agents we have another problem: the nursing problem.

Agents have to be nursed, we don’t trust them fully yet, they don’t seem to understand our tastes, where we want to go, and they’ll gladly do everything at the same time without caring too much about our metrics. Sure we can try to hardcode all of the stuff we want to optimize, and all of the stuff we don’t care about, but they still seem to come up with weird decisions that we don’t like. So for most of us, we don’t want to let them roam around and change code without supervising them.

The solution I’ve been toying with the idea of infra agents. Agents that have long, durable sessions, and live within the infrastructure, in place of code that would have attempted at deterministically figuring out what to do.

meat-proxy.png

We’ve all sort of coming down to the conclusion that flows and orchestrations with a few edgecases should be hardcoded, and that the ones with a million edgecases should be given to the agents, so I guess I’m talking about the ones in the middle that should be hardcoded as well but we don’t have time to figure out how to do it and we don’t want to let crazy agents hardcode these rules forever.

So the solution is to just have an agent move and live in that part of the codebase, see the stuff we’re seeing, and make decisions based on that. When you see everything, the decision is much easier to make on the spot / just-in-time. Maybe with time they’ll come up with scripts to get there faster, and with more time these scripts will become so reliable that they will encode the full algorithm that we would have come up with in the first place had we had the time to do that.

I call this point crystallization. I’m not sure it ever happens, because our codebases are constantly shifting nowadays, but maybe we will see instances of crystallization happen around us, maybe you will witness some of that.

suggested reads:

Stochastic agents shifting the boundaries between deterministic infrastructure and CLI/MCP tools.

We’ve all been amazed at how many LLMs and decisional models we’ve managed to integrate, in all sorts of places, in our codebases.

What was once a complicated deterministic process combining many heuristics that had to be managed carefully is now just an agent with a high-level prompt written in English that the whole team can easily maintain.

And even saying that, it’s not clear we’re still editing these prompts manually, or even reading them anymore.

In my case, after integrating my first agent loop in the codebase, I quickly reached for a configuration language in order to orchestrate these agents. Small configuration languages are better, and so I set out for a minimal syntax in yml files.

The configuration language basically looked like a GitHub workflow, simple enough that you’d think it would be really useful and that should be the obvious boundary between stochastic agents and deterministic code.

Deterministically automate what you can deterministically automate.

Underneath that, things were more flexible, constantly changing, and so subagents seemed like the right orchestration approach. One root agent following some high-level script, and getting better and better at orchestrating an army of subagents as models improved.

A friend then asked me, wait why use a configuration language to orchestrate your root agents? Why not just use prompts as well?

This is when the idea of a border between deterministic code and stochastic agents constantly shifting appeared in my head.

The idea of deterministic code is that it is fast, cheap, and predictable. But the predictability has a cost: it is not flexible, and bugs in the code can leave your application in unrecoverable states that will get your application stuck until you fix the code and ship an update.

Agents are slow, expensive, and often non-local. Open-weights are changing the game, but they are still slow and expensive. That hasn’t changed much. At least if you want output tokens of quality.

But we can all predict a future where this ceases to be the case. A future where every MacBook Pro ships with a special AI chip, an NPU, and SteveJobs 5.5 running on it, so that producing code on the fly is no longer slow, expensive, or non-local.

In such a world, perhaps the deterministic code orchestrating our agents will vanish. How much will it vanish? It’s hard to tell. Maybe we won’t use apps anymore, or UIs, maybe everything will be requested from agents on the spot.

Maybe agents will be the only UI humans will interact with going forward, maybe natural language will be the only API.

But beneath these agents, I want to argue, deterministic code is alive, and thriving.

Perhaps, deterministic code is not shifting, it’s getting teleported underneath agents’ feet.

We’re less and less preoccupied by how to run some infrastructure, and getting busier with what tools and custom CLIs and MCPs we’re giving our agents access to.

Tomorrow’s money will be in what your agents can interact with, the building blocks they can use to do useful things, and the code they can generate on the fly for the future requests they will have to handle.

If you’re building for tomorrow’s world, my advice would be to focus on that: one building block that would be useful for many people’s agents.

I sometimes wonder what the future of collaborative work looks like in this agentic world. We’ve set up and philosophized about all of these engineering processes to death. From dependency injection to scrum masters, from best practices to the minimum number of reviewers required for your PR to land, none of this matters anymore, all of this went to trash.

Most of our important tests are now shifting to SCENARIOS.md files, describing high-level flows in English, that agents can test using their pstack-inspired verification commands.

I still talk to people who work at companies where people manually write code, and manually review code, and everything sounds so slow, and so archaic.

suggested reads:

I get the same email often enough (“hey, we’re shipping some crypto, who do we talk to?”) that I figured I’d just write the answer down once.

So here’s the answer: if you’re building anything that touches cryptography and you want someone to look at it before it goes live, reach out to zkSecurity.

For the newer readers: a few years ago I cofounded zkSecurity, and what started as a “let’s audit ZK circuits” shop has grown into something much broader. We audit advanced cryptography in general now (ZK, sure, but also MPC, FHE, TEEs, threshold signatures, consensus protocols, post-quantum, and the boring primitives everyone gets wrong), we do formal verification for when “we reviewed it carefully” isn’t good enough, and we do development and design work too. The team is world-class, a mix of researchers, hardcore devs, and CTF people. Most of them are more qualified than me, which is exactly the situation you want when you’re handing off code you care about =)

There’s a second answer I’ve started giving lately too: zkAO, an AI tool we built that finds bugs in cryptographic codebases.

I want to be careful here, because everyone is slapping “AI” on a landing page right now and most of it is noise. But I’ve been doing this for over a decade, and I’ll just say it: I think zkAO is the best tool out there right now for finding cryptography bugs in real codebases.

Crypto bugs aren’t your average off-by-one. They’re things like a missing range check, a reused nonce, an unvalidated subgroup, a constraint that’s underconstrained. Off-the-shelf tools just miss these, because they don’t understand the cryptography. zkAO does, and that’s the whole point of it.

It’s not a replacement for a real audit. But it finds a lot of bugs cheaply and early, and it’s good enough that we use it ourselves on our own audits.

So if you’re shipping crypto: zksecurity.xyz for the audits and the formal verification, zkao.io to throw an AI bug-finder at your codebase and see what falls out.

Anyway, that’s the pitch.

suggested reads:

self-willed NFT

In 2022, Vitalik floated the idea of “soulbound NFTs”, NFTs that would be linked to your identity forever.

Today, with zkML (which I’ll explain in a bit), I think we can go further: NFTs that will live a life of their own, and decide by themselves when to change owner. I call these Self-willed NFTs.

The simple explanation here is that an NFT could represent an agent with its own personality (potentially secret), that people could try to seduce into updating their ownership.

Of course, you could still allow monetary-based transfer of ownership, but these concepts are taboo in countries with a complicated history like the US.

Anyway, self-willed NFTs can be truly realized through two technologies:

  • LLMs, this should be obvious
  • zkML, which allows someone to prove that the agent took a deterministic step according to their personality, values, and information available to them

Time to introduce zkML in a few sentences. LLM inference can be seen as a function of two inputs: the model and the prompt, and one generated output. Adding “zk” to it allows you to hide the model, or (part of) the prompt, or the output, or a combination of all three. While maintaining the integrity of the computation of course.

A self-willed NFT could thus have a secret personality committed on-chain (a hash of a pre-prompt), and zkML could be used to prove that some public inference is the correct result of a secret pre-prompt and prompt, as well as a configured and agreed model.

Then the agent could also control tools like a “change of ownership” tool in order to update this. Of course you would have to set a temperature of 0 to prevent abuse from the operator, and also assume that an operator has access to the pre-prompt (or you could have a trusted setup and then an MPC to hide the pre-prompt forever…)

You can imagine that conversations with the LLM could be rate-limited and gated (and ordered) on-chain, via a token, and potentially even private between the self-willed and the NFT.

zkML has seen a lot of advances, notably ICME’s Jolt Atlas, Zkonduit’s EZKL, the Mina zkML library, PSE’s circom-ml library, the work of Modulus Labs, and the work of Daniel Kang. Check it out, it’s a really interesting subfield of ZK! (or a subfield of AI, whichever you’re from.)

It was six years ago, when I joined the security team of Libra/Diem at Facebook. Two years later I was security lead and I had learned a huge number of lessons, some even life lessons.

I’ve been trying to motivate myself to record a vlog of some sort with some of the lessons learned during that time. It was a hectic two years during which I got my first gray hair, attended a house party with Mark Zuckeberg, grilled some s’mores on the Facebook campus while stalling all night to help announce the Libra project to the world at 6 AM, met a kidnapper as part as wanting to extend my crypto security circle, and wrote a book while traveling the US during covid.

But anyway, while I find the time and the will to record that one, I thought I would talk about a book that I rarely talk about, the security engineer handbook.

It’s a tiny read, with no bullshit and fluff, that contains a huge amount of learnings and gotchas that one should know about when joining a security team, and trying to make it, as a first timer. A huge amount of lessons I learned from are in that book, and I would recommend it anyone who’s in school and learning about security, or getting started in a security team, or working at a security team and wondering about the meta.

Anyway, it’s here.

suggested reads:

elsewhere

book Real-World Cryptography Applied cryptography, from Manning. podcast Two And A Half Coins Crypto, banking & distributed systems. youtube Cryptography videos Concepts & security, explained.
page info:
page 1 of 63
629 posts total