The oldest trick in the book
One of the most common hacking techniques is to trick someone’s computer into running some text as if it were code.
It’s called Cross-Site Scripting (XSS) and in very simple terms it could work like this:
- Go to an ecommerce site and order a T-shirt
- When buying, leave a note that says:
Can I customize the color? <script> sendYourPasswordCookieToMyServer() </script>- Owner of the shop opens my order and my note (“Can I customize the color?”) shows up on their screen
- At the same time, the
<script>part of the note is being run as if it were code. - Unbeknownst to the shop owner, their password just got sent to my server.
Of course this almost never happens in practice. The simple way to prevent it is to encode all unsafe user inputs (such that <script> becomes <script> which no browser will interpret as code to be run).
Don’t do the thing, get AI to do the thing
Since the advent of generative AI chatbots, a new twist on those kinds of attacks has become possible. Rather than write the tricky message yourself (as a user), try tricking the AI chatbot into writing it.
Why? Perhaps the programmers who encoded the unsafe user messages forgot to also treat AI-generated messages as unsafe?
How would that scenario play out in practice?
Talk to the chatbot and pretend to be a user interested in inline HTML scripts and hope that a helpful AI dutifully generates code that contains the tricky message (as in 2).
That exact scenario actually played out in the real world a few months ago.
Are those instances of AIs going rogue and executing code against our will? No.
XSS is a simple attack made possible by human error.
Cross-Site-Script attacked by ChatGPT?
Let’s go back to my ChatGPT screenshot. What happened there?
Code written by ChatGPT was meant to be human readable. It got executed instead.
I asked ChatGPT to edit my SVG code. It did that and printed out the ready version of the code which looked something like this:
<svg>
...
</svg>The code wasn’t sanitized, or rather, it was embedded in HTML tags which executed it in my browser. The browser tried to display the image which resulted in the little image fallback icon.
When I click on the Copy code button, I do get the exact SVG code I asked for. So in that sense everything went as planned.
“Copy code” returns the exact code I asked for
But as we can see below, the frontend logic treated it as code depicting image to be displayed rather than shown to the user verbatim.
The printed code got run in the browser
I accidentally got ChatGPT to execute some code on my machine without my permission. The code was not dangerous, a simple malformed image which failed to render. The intention was clearly for me to be able to read the code of the image rather than for my browser to try and show me the image.
But we do want AI Agents to run code for us
XSS attacks work when we accidentally take text and run it as if it were code. With the rise of Agentic AI and protocols like MCP, we’re all excited to have AI Agents take actions in the real world based on text generated by LLMs.
When done right it can be perfectly safe. But there is a number of previously unknown human errors waiting to be made and bring about new risks.
E-commerce is broadly believed to soon be revolutionized by AI Agents. Not only is it user input or LLM outputs that are potentially unsafe but also online content. In the race to index millions of shops and fetch content in real time, safety might be particularly tricky to ensure.
Agentic commerce opens new creative attack surfaces
Another aspect is navigating the little quirks of complex ecommerce and payment processing systems. Whenever money transfers are at play, stakes are high.
Let’s end on a philosophical note
One of my favourite philosophical questions comes to mind: would a computer program running a perfect simulation of ocean waves get wet?
Well, my ChatGPT kind of got wet when I asked it to tell me how to make an image and it made the image instead.