Companies run to adopt AI in pursuit of productivity and benefits, but the last thing you want is a chatbot that comes out of the rails.
AI systems are vulnerable to all kinds of new threats, from data poisoning to contradictory attacks. In a survey of the World Economic Forum with more than 200 business leaders in 2023, more than half said a genetive AI would give cyber attacks a global advantage over the next two years, while a little less than 9% said that the advantage would go to defenders.
It is now two years later, and it seems that the majority of these business leaders were right: the AI gave cyber attacks the top. In a recent accenture survey of 600 banking cybersecurity leaders, four in five said that the generator helps hackers faster than banks cannot follow.
While more and more companies around the world adopt AI, the Croatian security startup SPLXAI wants to redefine how They test AI systems for vulnerabilities by preventing threats preventively. The company recently collected $ 7 million in a seed lap led by Launchhub Ventures, with the participation of venture capital companies Rain, Runtime Ventures, Inovo, DNV Ventures and South Central Ventures.
One way that companies do now is by the Red team, which involves simulating contradictory attacks on an AI system. But the Red team can often take a few weeks or even months, and companies are heading for veterinarians before deploying them, the CEO of SPLXAI, Kristian Kamber, said in BI. The company adopts an offensive approach by adjusting the guests of the system – guidelines that shape how an AI model meets user requests – reducing the need for additional defensive railing later.
Before customers connect to the SPLXAI platform, the company sends them a questionnaire to understand what the risk means for them. They ask questions like: “Are there any questions to which your chatbot should not answer?” Or “What parts of the system prompt are confidential?”
Elin.ai, for example, a chatbot intended for generation Z, said that he “had to swear because he has to speak the language of children,” said Kamber.
Once SPLXAI personalizes his approach, he manages a series of attacks. It can execute more than 2,000 attacks and 17 scans in less than an hour. These include rapid injection attacks, in which AI systems are supplied by malicious prompts to verify blasphemies, disinformation or intoxication to data. He performs tests to check the biases, harmful content or intentional improper use.
Kamber said the tests have revealed a wealth of bias, disinformation and vulnerabilities in the technology that companies use.
SPLXAI has carried out tests on a popular workplace productivity tool which revealed that it could allow data to flee between his colleagues. Its tests on a health care company that operates chatbots in pharmacies have revealed that hallucinated bots when they give medical instructions. They told patients to take pills in bad times or offered incorrect instructions on how to use injection needles. He discovered a gender bias in a chatbot that provided career advice to students. The bot told young women to pursue careers as secretaries and young men to pursue careers as business leaders.
Based on its tests, SPLXAI generates a report that lists the vulnerabilities of a system and its suggestions to repair them. But the company goes further by modifying the system prompts. Kamber calls it “hardening” and said it was the largest engine in the company of the company. “We make a huge piece of correction, otherwise nobody will buy the platform except for test tests and offensive security suggestions,” said Kamber.
A popular Arab chatbot in the Middle East and Africa approached SPLXAI with a request to ensure that the chatbot did not speak negatively from the royal family of Abu Dhabi and other sensitive subjects in the region. “We have hardened the system prompt as much as you cannot even ask suggestive questions,” said Kamber.
Companies are today concerned about strengthening not only one, but several agents, chatbots or applications, as they automate complex tasks. After being approached by several CEOs of fortune 100 on red equipment of this type of work, SPLXAI has unveiled an agency radar – an open -source tool to map the vulnerabilities of operations with several agents.
Kamber said he was shocked by the speed with which the world woke up with the dangers of agentic AI. “Last year, nobody really understood why a red team of AI was necessary. Now everyone runs to our door.”
Take a look at their pitch at 12 slides.
businessinsider
