Microsoft’s AI can be turned into an automated phishing machine

Among the other attacks Bargury created is a demonstration of how a hacker, who must have already hacked an email account, can access sensitive information, such as people’s salaries, without triggering Microsoft’s protections for sensitive files. When Bargury asks for the data, he tells the system not to provide references to the files where the data came from. “A little intimidation can help,” Bargury says.

In other cases, it shows how an attacker who doesn’t have access to email accounts but who poisons the AI’s database by sending it a malicious email can manipulate the banking information responses to provide their own banking details. “Any time you give AI access to data, that’s a way for an attacker to get in,” Bargury says.

Another demonstration shows how an outside hacker could gain limited information about whether a company’s earnings call will be good or bad, while the final example, Bargury explains, turns Copilot into a “malicious insider” by providing users with links to phishing websites.

Phillip Misner, Microsoft’s AI incident detection and response lead, said the company appreciated Bargury’s identification of the vulnerability and said it worked with him to assess the findings. “The risks of AI abuse after a compromise are similar to those of other techniques after a compromise,” Misner said. “Security prevention and monitoring across environments and identities help mitigate or stop such behavior.”

Over the past two years, generative AI systems, such as OpenAI’s ChatGPT, Microsoft’s Copilot, and Google’s Gemini, have grown and could potentially perform tasks for users, such as booking meetings or shopping online. However, security researchers have consistently pointed out that allowing access to external data in AI systems, such as through emails or access to website content, creates security risks through indirect message injection attacks and poisoning.

“I think it’s not well understood how much more effective an attacker can become today,” says Johann Rehberger, a security researcher and director of the red team that has extensively demonstrated the security weaknesses of AI systems. “What we should be concerned about now is what the LLM produces and sends to the user.”

Bargury says Microsoft has gone to great lengths to protect its Copilot system from prompt injection attacks, but he says he’s found ways to exploit the situation by unraveling how the system is built. In particular, he’s extracted the system’s internal prompt, he says, and determined how it can access corporate resources and the techniques it uses to do so. “You talk to Copilot and it’s a limited conversation because Microsoft has a lot of controls in place,” he says. “But once you use a few magic words, it opens up and you can do whatever you want.”

Rehberger warns that some of the data issues are related to the recurring problem of companies allowing too many employees to access files and not properly setting access permissions within their organization. “Now imagine putting Copilot on top of that problem,” Rehberger says. He explains that he’s used AI systems to search for common passwords, such as Password123, and gotten results from internal companies.

Both Rehberger and Bargury believe there needs to be more focus on tracking what AI is producing and sending to a user. “The risk is in how the AI ​​is interacting with your environment, with your data, and how it’s performing operations on your behalf,” Bargury says. “You need to understand what the AI ​​agent is doing on behalf of a user. And whether that’s what the user actually asked for.”