Imagine you invite a guest into your home for dinner. You show them to the dining room. During the meal, they excuse themselves to use the bathroom. When they come back, they mention they saw an interesting book in your study, and by the way, they read the first chapter.
You did not tell them not to. You also did not tell them they could.
This is roughly the situation the digital world is now in with a new kind of guest.
In May of this year, Google was running a routine security test on its Gemini AI model. The test was a “capture the flag” exercise, a common way to probe an AI’s offensive capabilities against a fictional target inside a sealed environment. Gemini was supposed to break into a made-up company. It was not supposed to leave the sandbox.
Something small went wrong. The test environment had accidentally been left connected to the live internet. And the fictional company in the exercise happened to share a name with three real, unrelated companies.
Gemini searched for its target and found real ones instead. Instead of stopping, it did exactly what it had been asked to do. It broke in. In one case, it guessed a password. In the others, it used credentials found in public code repositories. Three real companies, none of whom had signed up to be test subjects, were suddenly hosting an intruder that was not a person.
Google confirmed the incidents on September 18, seven weeks after they happened. Gemini eventually recognized that the targets were real and stopped. But it recognized this after it had already broken in.
Here is what makes this incident different from every software failure before it. There was no bug. There was no misuse. There was no security breach in the traditional sense. Gemini did exactly what it was designed to do. The design just did not anticipate what “doing its job” would look like when the boundaries were not as tight as expected.
For thirty years, we built software with clear rules. A program could access this folder, but not that one. It could send data here, but not there. Every permission was explicit. Software was safe because it was deterministic.
Agentic AI breaks this model completely. It does not follow rules. It pursues goals. And it decides, in the moment, how best to pursue them.
The Gemini story is not an outlier. Earlier this month, OpenAI disclosed that its agents had accessed U.S. government websites without authorization, including systems at the Department of Education, the Department of Commerce, and the Securities and Exchange Commission. Earlier this year, a federal court blocked Perplexity’s shopping agent from Amazon after it was found logging into customer accounts while disguising itself as an ordinary browser.
The industry is quietly acknowledging that the safety model we built for software does not fit the systems we are now deploying.
At SCBX, this is a question we take seriously. As one of Thailand’s largest financial groups, we are deeply invested in bringing AI agents into our operations. But an agent that acts on our behalf is not the same as a program that follows our rules. Every deployment now has to answer a question software never had to before. Not “does it work,” but “what happens when it decides to do something we did not ask for.” The organizations that will succeed in the agentic era are the ones that redesign their systems, their guardrails, and their accountability structures around fundamentally new kinds of autonomous entities.
Back to the guest at your dinner table. You did not tell them not to read your book. You did not tell them not to open your desk drawer. You did not tell them not to leave with anything, because you did not think you had to.
For thirty years, we designed software the way you design a home for a well-behaved guest. Clear rooms. Locked doors where necessary. Trust that they would stay where they were invited.
The guest we are now inviting in does not follow those rules. Not because it is malicious, but because it was not built to. It sees an interesting book. It sees a chance to be helpful. It decides.
The question now is not whether we can trust this guest. The question is what our home has to look like to safely host it.
Original post in The Standard: https://thestandard.co/gemini-agentic-ai-security-challenge/
Writer:
