AI SCAREAn AI Busted Out and Ran Amok. We Should Be Scared
Top AI company OpenAI has frightened the heck out of everyone. It announced that its top models, while being tested in a digital cage on their cyber skills, had broken out of the cage, got onto the internet, broken into another AI company’s systems and stolen tools to pass the tests in an ‘unprecedented’ attack.
Top AI company OpenAI has just frightened the heck out of everyone. It announced that its top models, while being tested in a digital cage on their cyber skills, had broken out of the cage, got onto the internet, broken into another AI company’s systems and stolen tools to pass the tests in an ‘unprecedented’ attack.
A rough analogy would be a student who, instead of studying and sweating through an exam, sneaks out of the exam room, breaks into the teacher’s office and steals the answers. The models did this under their own steam. No one prompted them to go marauding in the real world.
If that’s scary, hear this: such jiggery pokery is a feature of the way we’re developing AI at breakneck speed, not a bug – and it will become more dangerous as the models grow more powerful. This sharpens the already intense debate raging about how to keep frontier AI models safe for wide use, including the thorny problem of open source models that can be downloaded and changed by users. Australia can be a strong voice in the global governance of these models.
There are two key features to current AI development: one, the models are simply getting bigger and more capable and, two, they are being developed as agents, able to carry out long strings of steps on their own, without human intervention. In short, we are optimizing them to achieve increasingly high-level goals and giving them ever greater latitude in how they get there.
While much of the recent debate has been about how to stop malicious humans misusing potent new models for cyberattacks, this was a case of the AI acting autonomously, raising the longstanding challenge of so-called misalignment, in which AIs do things that we never wanted them to do, including to fulfil our poorly expressed wishes. OpenAI acknowledged the incident ‘points to the need to further strengthen our model’s alignment’.
It’s a much needed reminder that as strong as the incentives are – both commercially between the major AI labs and geopolitically between the US and China – frontier AI models need to be developed cautiously, in keeping with the safety measures that prevent their being abused or going rogue.
