All posts
/5 min read

Six questions to ask any AI agent security vendor, including us

A short evaluation checklist for anyone buying AI agent governance. Every question is one we answer in writing, and two of them end most demos.

Anthony Levy
Anthony LevyFounder, damn.dev

Buying in this category is hard right now because every product demos the same. Discovery finds agents nobody knew about. A dashboard goes red. Something gets allowed or denied. Everyone claims visibility and control.

The differences are architectural, and they only surface under specific questions. Here are the six I would ask. They apply to us as much as to anyone else, and we answer all of them in writing before you sign anything.

1. What do you have to read in order to work?#

Some products cannot function without ingesting prompts, model responses, and each reasoning step, because their decision is a model judging whether the agent's intent looks acceptable.

That can be a genuinely good product. But understand what it makes the vendor. If your agent reads a patient record, a salary table, or an unreleased financial statement, and the security product must read the agent's context to do its job, that content is now flowing to a third party. In most European regulated contexts that vendor has just become a data processor, with everything that follows: a DPA, a subprocessor entry, a transfer assessment, and a conversation with your DPO.

Ask specifically: what is the minimum you must read for the product to work at all? Not what you can be configured to redact. What is required.

Our answer: nothing. Everything we ship decides on the shape of an action, not on its content. Which credential, which system, which operation, on whose behalf. We scope one optional inspection tier that is off by default and runs entirely on your own infrastructure with your own certificate authority, and even then the content does not reach us. We enable inspection. We do not perform it.

2. Where does the record live, and who can change it?#

The audit trail is the artifact you will eventually hand to an auditor, a regulator, or your own board. Three follow-ups:

Does it live in your infrastructure or the vendor's? If the vendor's, what happens to two years of evidence the day you leave, or the day they are acquired?

Can it be altered, and can you prove it was not? Append-only in a database means "we do not offer a delete button." A hash chain means each entry commits to the one before it, so an edit anywhere breaks verification everywhere after. Those are very different guarantees.

Is it one record, or one per vendor silo? Three tools each keeping their own log is not an audit trail. It is three narratives you get to reconcile under time pressure.

3. What happens when your service is unreachable?#

Every enforcement product has a failure mode. There are only two, and both are legitimate.

Fail open means the action proceeds when the check cannot complete. The upside is that your engineers keep working during an outage. The cost is a window where nothing is enforced.

Fail closed means the action is refused. The upside is that the guarantee holds. The cost is that a vendor's bad afternoon becomes your outage.

The wrong answer is not knowing, or not having decided per surface. Ours differs deliberately by surface: a cooperative hook inside a developer's coding tool fails open, because a governance layer that freezes an engineer's session gets uninstalled within a week. Enforcement that happens at a credential or a resource you own does not fail open, because there is no separate check to fail. The credential either works or it does not.

4. Which of my agents does this govern that I did not deploy?#

This is the question that separates the market into two shapes.

Platforms govern the agents you build on them. Gateways govern the traffic you route through them. Both are real, and both stop at their own boundary. Neither covers the agent your CRM vendor turned on in a product update last month, the tool a contractor installed on day one, or the MCP server an engineer added this morning without knowing there was a gateway to route through.

Ask for the coverage story on an agent that will never be rebuilt, never be migrated, and was never approved. If the answer requires the agent to move, you are buying a migration project with a governance feature attached.

5. What does it cost me to remove you?#

Nobody asks this, and it tells you more than the feature matrix.

A product that took a week to install and cannot be uninstalled without breaking production has quietly become infrastructure. That is not automatically bad, but it should be a decision rather than a discovery.

Our honest answer today: at the beginning, very little. We land as a config line in tools you already run, and a config line can be deleted. That is deliberate, because it is the only shape a team without a mandate will actually try. It changes as you route more through us. The day your compliance evidence is our chain and your agents' credentials are issued by us, removing us is a real project. You should know which side of that line you are on, and you should be told before you cross it rather than after.

6. Show me one thing your product does not stop.#

If a vendor cannot answer this in under ten seconds, one of two things is true. Either they have not thought hard about their own boundaries, or they have and would prefer you did not.

Ours, without hedging: a cooperative hook inside a coding agent is best effort. It fails open by design and a determined agent can route around it. What it cannot do is act quietly. The record shows what happened whether or not the gate held.

And a broader one: we cannot contain an agent running on a vendor's cloud. Nobody can. We govern what it reaches by owning the credentials and the systems it has to arrive at, which is a different and more limited claim than the word "contain" implies.


Two of these end demos: question one, and question six.

We would rather you ask both, early, and to everyone.