If you are someone interested in the consequences of transformative AI, and you have not watched this video on Youtube (AI in Context’s “We’re Not Ready for Superintelligence”) -> https://www.youtube.com/watch?v=5KVDDfAkRgc I highly recommend watching it. It gives digestible insights into the race to AGI, geopolitical levers that may be pulled, and some potential outcomes.
The below writing discusses elements of the video as they relate to recent papers by RAND around Securing Model Weights, and Frontier AI Model lab security levels.
Overview:
Due to the difficulty in protecting against nation-state level threat actors, AGI model weight theft is a very realistic threat vector. Companies need to apply increasingly higher levels of security and protection across all layers of defense in order to deter/stop highly-resourced threat actors. Some areas that may need additional review include employee personal security postures, accommodating for emergent behaviors from advanced AI systems, as well as policy levers to help raise the standard of security at frontier labs.
Threat Models:
In the YouTube video “We’re Not Ready for Superintelligence” (by AI in Context) a scenario is outlined where a nation-state level threat actor infiltrates a frontier model company and steals AGI-level model weights. This is just one path of many for how multiple global players enter into potentially catastrophic AGI endgame scenarios.
From an outside perspective, it might seem unrealistic for some foreign agent to infiltrate a company and steal secrets. Life isn’t a “Mission Impossible” movie.. right?
Depending on a person’s threat model, it could be. The interest level of threat actors is largely proportionate to perceived value of the targeted person’s proximity to an objective. The value plays out across multiple dimensions: role, access, authorization, location, position context, and more. For example, someone who works as a nurse may have some risks due to their profession, but likely wouldn’t be of interest to a nation-state level adversary. Now place the nurse into a classified military or classified government position, and the threat model shifts significantly. In this scenario, the nurse would undergo background vetting, and could be a target for adversarial nation-state data exfiltration operations. The threat model anchors towards requiring a higher duty of care to personal (including physical and cyber) security in their life.
Recent Relevant Nation-State Activity:
Nation-state level infiltration of corporations has become a general business model for raising revenue and getting sensitive access. For example, consider the recent DPRK infiltration of multiple companies, leading to an FBI investigation resulting in raids on “laptop farms” in the US. These laptop farms were facilitating “geo-normal” access to the companies internal networks, simulating how a real employee based in the US would operate (https://www.justice.gov/opa/pr/two-north-korean-nationals-and-three-facilitators-indicted-multi-year-fraudulent-remote).
This is a prime example of the lengths to which nation-states have gone to acquire resources they deem important. In this case, it’s geared towards revenue generation, but these agents can just as easily shift into ransomware, data exfiltration, or other sophisticated targeting from their position.
Additionally, deepfakes leveraging advanced AI technology helped to defeat identity matching programs that the targeted companies were using. This has led to more thorough processes being put in place, including advanced identity proofing systems as well as some pushes for requiring new employees to be verified in person. (https://unit42.paloaltonetworks.com/north-korean-synthetic-identity-creation/)
In multiple instances the identities used during this attack were stolen, legitimate identities. Based on this, we are seeing the convergence of multiple emerging technologies enabling advanced, asymmetrical attacks.
To summarize, the above DPRK instance includes: deepfakes, dark web breach data for stolen identities (with people potentially being selected based on facial features using OSINT), deepfake video for video calling, physical presence in the targeting region via laptop farm, and then social engineering for interviewing and working.
One may think this is still not that scalable, but the reality is, this can run like a normal red-team cyber operation using well-established command-and-control protocols. For example, initial access is achieved by one actor, and then the actual “working” can be done by a lower-skilled worker uplifted by generative AI to match the job position. When further actions are required, the “worker” can hand the reins off to the “hacker” to perform various actions, and back again.
This whole approach can eventually be largely supported by generative AI, perhaps even at an autonomous level, in the near future.
Threat Actor Capability Levels:
In Securing Model Weights (“from misuse” by RAND), a general framework is laid out to describe the capabilities and likelihoods of different levels of threat actors as it pertains to efforts to steal model weights from a company. The most likely threat actor, based on this framework, is outlined by the “OC4 and OC5” level which is a highly-resourced actor such as a nation-state level threat actor.
From the DPRK example, the following RAND-outlined threat vectors were leveraged:
Compromising Existing Credentials, Unauthorized Physical Access to Systems, Human Intelligence, AI-Specific Attack Vectors
And correspondingly, the level of mitigations that are required to be in place to best thwart that threat actor is at the outlined “SL5” level. Already, this is suggesting that the US companies on the forefront of AI and AGI development will require help from the government and military to keep it safe.
Missing Steps:
Considering the advanced asymmetric techniques already being leveraged by remote foreign adversaries, model weights have a good chance of being stolen through asymmetric operations. Tactics from the laptop farm approach suggest that the final approach would likely be a combination of advanced, accessible technologies leveraged in concert to achieve the goal.
Implications:
At a maximum level, frontier model companies may need to apply government-style top-secret compartmentalization techniques to protect information and access to AGI model weights. Additionally, government level surveillance, vetting, and security operations may need to be applied. We have seen partnerships between AI companies and the US government already, which does suggest that the national security implications are already clear.
Pumping the Brakes:
This should be a signal for the frontier model companies to slow down enough to be able to properly protect the latest iterations in model advancements if they want to avoid slowing down to government speeds at some point due to regulation.
It’s not just that we need to slow down in order to help develop the advanced AI systems safely, it’s also a matter of national and global security, to ensure that frontier model work access is secured at the level proportionate to its threat model which is at nation-state levels.
From the RAND paper, this slowing down or impact to “productivity” is a difficult balance to strike but is considered to be helped by creating secure interfaces to interact with model weights. Air-gapping the model weight systems, and only allowing access to them with dedicated specialized secured hardware, in conjunction with rigorous authentication and authorization protocols would likely strike the balance to maintain speed of innovation.
However, advanced vetting for users is not mentioned, which should be included as part of the personnel security levels.
Additionally, personnel security measures are limited to training but do not incorporate home or personal device security audits or tooling to maintain home security outside of the workplace. This may sound controversial, but this is an often overlooked avenue that should not be ignored. Home attack surface, an essentially non-trusted environment, can leak into the workplace through devices carried by the employee. There are likely legal limitations here, therefore, the company could instead provide tooling and methodology as well as training for the employees (and dedicated sprints) to harden their personal attack surface outside of the workplace.Finally, while the focus is largely on humans, the benchmarks may need to be updated to include AI agents (non-human access). This can be included in generally unknown vectors, but more work may need to be done to analyze how non-human goals may play out in systems that have been secured with only humans in mind.
Recommendations:
Echoing RAND’s paper, theft prevention mechanisms need to be red-teamed using asymmetrical approaches to simulate real, well-resourced adversaries. This includes developing novel approaches for red-teaming purposes.
New Recommendations:
Calls to slow down AGI development should include the need to secure the companies themselves to the levels of SL4, SL5 as outlined in the RAND paper
Monitor public instances of nation-state level operations and apply their implications to model weight security guidances, updating as needed
Review feasibility of certain attacks, such as the mentioned example of “Chrome RCE zero-day via purchase for $500,000”, due to uplift in vulnerability discovery techniques. Advances in open-source iterative multi-agent workflows will lower the barrier to entry for threat actors. This is likely going to shift vector priorities. Future work may need to try to capture trends as well.
Consider how to assess home and personal devices, and provide tools for employees to reduce trusted-but-vulnerable attack surface
Consider how multi-adversary attacks are weighed, where multiple coordinated nation-states compete and cooperate to achieve a goal (maybe this is OC5 still).
The benchmarks may need to incorporate non-human vectors as well, as emergent behaviors may not be captured within the framework.
Potential policy implications:
Require frontier AI model companies to show sufficient level of protections from a physical, cyber, and personnel security perspective are in place and regularly tested by red teams
Incorporate, or move towards, government level clearance vetting into hiring processes where AGI development is involved or tangential to the position
Consider how home or personal device security, and physical security at home can be incorporated into requirements for personnel security levels without invading privacy
Consider how non-human agent security should be governed