The UK Safety Institute, the UK’s recently established body for AI safety, has released a set of tools designed to “strengthen AI safety” by enabling industry, research organizations and universities to more easily develop AI assessments.
Called Inspect, the toolset – available under an open source license, specifically an MIT license – aims to assess certain capabilities of AI models, including the models’ basic knowledge and ability to reason, and to generate a score based on the results.
In a press release announcing the news Friday, the Safety Institute said Inspect marks “the first time an AI safety testing platform, led by a state-backed organization, has been released for wider use.
“Successful collaboration on AI safety testing means having a shared and accessible approach to assessments, and we hope Inspect can be a building block,” Ian Hogarth, president of the Safety Institute, said in a statement. “We hope to see the global AI community use Inspect not only to conduct their own model safety testing, but also to help adapt and expand the open source platform so we can produce high-quality assessments at all levels.”
As we’ve written before, AI benchmarks are difficult, not least because today’s most sophisticated AI models are black boxes whose infrastructure, training data, and Other key details are kept secret by the companies that create them. So how does Inspect meet this challenge? By being extendable and expandable to new testing techniques, mainly.
Inspect is made up of three basic components: datasets, solvers, and markers. Datasets provide samples for benchmark testing. Solvers do the work of running the tests. And graders evaluate the work of solvers and aggregate test scores into metrics.
Inspect’s built-in components can be supplemented via third-party packages written in Python.
In an article on
Clément Delangue, CEO of AI startup Hugging Face, floated the idea of integrating Inspect with Hugging Face’s model library or creating a public leaderboard with the results of the toolset’s evaluations.
The release of Inspect comes after a US government agency – the National Institute of Standards and Technology (NIST) – launched NIST GenAI, a program aimed at evaluating various generative AI technologies, including text-generating AI and of images. NIST GenAI plans to publish benchmark tests, help create content authenticity detection systems, and encourage the development of software to detect false or misleading AI-generated information.
In April, the US and UK announced a partnership to jointly develop advanced testing of AI models, following commitments announced at the UK AI Security Summit at Bletchley Park in November last year. As part of this collaboration, the United States intends to launch its own AI Safety Institute, which will be largely responsible for assessing risks related to AI and generative AI.
techcrunch
