Peter Garraghan is a chair professor in the UK who specializes in system security and AI systems. Ten years ago, he had a thought that changed everything: Current security technologies won’t work against deep neural networks. Before LLMs and agents existed, he was looking at image models and natural language processing models. He spent years interrogating deep neural networks with his scientists and came to a conclusion: it's really hard to protect against advanced thinking systems.
Between 2017-2018, the transformer architecture paper dropped (the genesis of modern LLMs and agents). Peter realized the urgent need for a company in this space because people are going to anthropomorphize this AI. They'll overlook security problems because it says "please" and "thank you." The software itself is intrinsically opaque — there's no human-readable code, just meta mathematics and probability. And companies are going to cram this into every part of their security stack.
Fast forward to today: Peter's prediction came true. LLMs and agents are everywhere; people don't understand how they work yet they're touching all the systems. Peter took half his scientists from his university lab, went to London, raised capital, and built Mindgard. Jim Brear joined as CEO because the approach was so novel, combining AI research with offensive security in a way no one else was doing.
Here are three things I learned from sitting down with them.
We’re Living in the 90s Again (But Worse)
"It's almost easier now than it was in the 90s," says Jim, referring to how easy it is to hack systems. We haven't had our Log4j moment yet for AI — that defining event where everyone goes "oh my god" and realizes how exposed they are. But we're very close.
Why is it scarier than the 90s? Back then, hackers would attack in singular events. Now, hackers leverage AI to attack AI. They create thousands and thousands of prompts simultaneously at a scale previously unattainable.
And the networks? They're porous. CISOs are responsible for commercial software and open source, but they can't tell the difference. They don't know the risk they have within their infrastructure. AI is embedded in third-party applications by stealth. People are purchasing AI, building AI systems, deploying them and then thinking, "What's my attack surface?" If you can't answer that question, you can't do anything about it.
Peter's framework is simple: I can't defend what I can't test, and I can't test what I can't see. Security teams are worried about AI being deployed, but they can't apply their existing security controls and thinking to the problem.
AI Models Are Twins Raised in Different Environments
When I asked about the difference between proprietary and open-source models from a security perspective, Peter gave this analogy: DNA and people. If you have two twins with the same DNA but they grow up in different environments, they might have the same underlying problems but develop different ailments because of environmental factors.
Every LLM and agent is trained with the same underlying architecture, but they manifest their own unique security problems and capabilities. The nature of security is finding capabilities, gaining leverage, and exploiting them. In human terms: If someone is malleable or gullible, you take a very different approach than someone who's strict or rigid but narrow-minded.
These agents and LLMs are trained on human corpus data, which means you can apply pressure tactics to reveal capabilities. For example, you hear a lot about "this LLM can generate SQL injection commands." Great, but that's only a problem if there's a SQL database in the first place. And somewhat ironically, the more capable and better the model, the easier it is sometimes to trick it to do things it shouldn't. It's like having a very smart intern who's very capable; you can basically exploit the system.
It's less about the models themselves. It's how the models are deployed in real infrastructure, what they're touching, what they're designed for, and then figuring out how to find the attack surface, find their behavioral capabilities, and exploit them.
Replace "AI" with "Software" and Your Problems Become Clear
Peter's advice to customers is to replace the word AI with software. Replace MCP with ports and networks. What would you do?
If someone said, "I've got a piece of software. I haven't done any testing. I don't know the risks involved. I connected it to my system with no authentication or permissions to other systems. What could go wrong?" Well, the answer is everything can go wrong.
MCP, for example, was released as open source to drive adoption. It was never built to be a secure protocol. "MCP is two POST requests going back and forth. Of course it's insecure, it wasn't designed that way."
So if you apply conventional security thinking to AI problems but realize your tools and processes don't work against it, you start looking for updated training, updated playbooks, and updated technology that really targets the AI aspect of the system.
The fundamental problem is we're living in a world where there's emergent tech and an emergent market, and even the research isn't solved. Even scientists are coming up with innovations every day. So expecting an enterprise to have fully secure AI infrastructure when the fundamentals aren't solved yet? That's really hard.
But AI is too valuable not to use. So as you mature in AI or get started with AI, make sure governance, compliance, and security are driving forward alongside it.
The Most Important Lesson
When I asked Jim and Peter the most important thing they've learned about AI, they offered two answers:
First, replace the word AI with software and just think through the problem. A lot of pain points will go away or become clear.
Second, AI isn't new. It's been around since the 1950s. (You can argue even before then). This type of AI is just the next generation. There will be other types in the future. What we have now isn't going away; it’ll just become pervasive and ubiquitous in everything we do.
So it's really important to establish security principles and controls. Some you can use existing techniques for. Others you can't. Find those very quickly and apply them to use cases that actually help your business.
After talking with Jim and Peter, I'm thinking differently about the AI security landscape. We're facing a fundamental shift in how security works because the thing we're trying to secure doesn't have human-readable code. It's probability and meta mathematics — and the old playbooks don't work anymore.
Listen to the full episode of Actually Intelligent to hear more from Jim Brear and Peter Garraghan about how Mindgard discovered ChatGPT could surface disturbing images in less than 24 hours, why doing reconnaissance before attacking AI systems is like going to battle with a plan, and Peter's take on why AI is like a really excited intern on their first job (which is scary).
LEAVE A COMMENT