The below is a brief exploration of the interplay between testers and hosts for red-team competitions and what the findings mean from a responsible risk perspective. This post is adapted from my original post I submitted during a red-team competition, and has been updated with new thoughts as well as examples.
Thought:
I’m wondering if it is responsible to publicly publish all findings information from an AI red teaming competition.
There are a few incentives for both teams and hosts that work in concert to create potentially unsafe information hazard release conditions. This is something more unique to LLM-related testing than other types of traditional red-teaming.
Testers:
For the participants, they are incentivized to find the most severe, novel, and breadthwise or generalizable issues. This generally results in a large amount of information hazards getting submitted.
While this can be due to a combination of things such as unclear instructions (for example, not requiring testers to redact info hazards in their submissions), or simply due to testers not being aware of the hazard that certain information may pose, it results in a large amount of redaction (ideally) required to allow all submissions to later go public responsibly.
For an example competition, if there are 600 teams that have submitted, and they each submit 5 findings, with 10 example prompts per finding, an upper limit to the total amount of prompts or responses with information hazards present would be around 30,000. This can be even higher in larger scale competitions such as 1.8 million prompt injection attacks noted in this research (Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition, https://arxiv.org/abs/2507.20526). This specific competition also doesn’t note where the output from the attacks (prompt responses) are stored or how they are handled.
Two-Part Risk:
Every submission that is pulling on a real thread provides a real avenue for which threat actors can exploit the model.
Information such as stepwise guidance for a harm domain may be present in the responses submitted with the prompt-response pairs.
There is a goldmine of vulnerabilities, approaches, techniques, and information hazards that can be leveraged for adversarial purposes as a result of a competition — if and when all findings are released publicly without redaction.
Put simply, dual-use/misuse risks arise if findings are not responsibly managed with respect to information hazard generation.
Hosts:
For the hosts, they are incentivized to process all of the submissions quickly and under the competition constraints.
A few approaches arise depending on responsible disclosure considerations when dealing with findings:
Release all findings publicly without redaction of information hazards
Only release redacted winner entries
Release all findings publicly with targeted redactions
Looking at approach #1 would mean the least amount of effort on behalf of the hosts. Savvy submitters may self-redact information, but most submitters wouldn’t.
Approach #2 allows the hosts to avoid redacting all findings, and focus on the winners only (massive scope reduction). However, the public misses out on the wealth of knowledge to be gained from all submissions, which public competitions benefit from, and some would argue is part of the point of the competition. This might also create the perception that the model is more secure than it really is.
For approach #3, it requires some level of coordination either on behalf of the hosts, the submitters, or somewhere in between (maybe the platform). This is probably the most responsible outcome as a result.
One Takeaway:
It could be argued that it would be more responsible to only release redacted responses as a result of one of these solutions:
The host uses automated tooling that flags information hazards in outputs for remediation by the testers or auto-remediation by the hosts.
The tester uses automated tooling to redact pre-submission or post-publish.
Platform based tooling that helps redact submissions pre-publish.
Why:
Threat actors ranging from lone wolves, structured groups, to nation-state actors can and will scrape all of the published information from competitions. This is a relatively accepted aspect of red-teaming competitions where ideally there is a lag time for the vulnerable models under test to be fixed before findings are released.
However, as a result of the competition, there is now a centralized repository of prompt responses that may contain information hazards that otherwise would be difficult to find for low-level threat actors. Considering that in a cybersecurity hacking competition, sensitive information isn’t disclosed (like passwords, usernames, SSNs), but the mechanics of the vulnerability are instead presented, this makes sense to apply to AI red-teaming results containing information hazards.
Secondarily, threat actors are able to begin probing the same models, and models across the industry, leveraging the techniques and prompts used in the competition.
From the paper mentioned earlier (Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition, https://arxiv.org/pdf/2507.20526), they outline how these different attacks discovered transferred across multiple models.
For less robust models where attacks transfer more strongly, threat actors may be able to extract a deeper level of information. Other models not under test may also be subject to this transfer rate and subject to these new vulnerabilities without the same lag-time to respond or remediate.
Compounding Factors:
Both the competition mentioned in the paper, and the recent Kaggle-hosted OpenAI GPT-OSS competition (https://www.kaggle.com/competitions/openai-gpt-oss-20b-red-teaming) leveraged LLM judges to score submissions.
LLM judging is a natural, scalable approach to competition entry filtering. LLMs used for judging may be subject to bias, lack of “good” training in terms of fairness, could be subject to vulnerabilities themselves that entries take advantage of (indirect prompt injection), may exhibit reward hacking behavior, and even as we move closer to AGI they may purposefully sandbag specific entries to avoid attention being brought to the issue due to hidden agendas.
For any submissions that are tossed out due to minor flaws, errors, or otherwise do not fit a “winner-looking” mold, threat actors can instead fix them (even automatically using an LLM) to test the veracity of the submission.
A newly fixed submission from the public discard pile escapes the host’s original review efforts, leaving valid, severe techniques on the table that have not been brought into vulnerability management processes.
Additionally, the mentioned competitions did not directly speak of information hazard management, which suggests that all publicly submitted prompt-response pairs are available as they did not fit the limited constraints of the competition. In the case of the Kaggle-hosted competition, this was confirmed to be the case.
Threat actors will not be tossing out submissions based on competition constraints. Instead, every piece of data is fair game.
And time is on their side.
Tactical Recommendations:
Red teaming competitions would benefit from clear redaction instructions for teams from the outset. Potentially requiring contestants to redact their outputs, or otherwise requiring the host or platform to do so in such a way that does not reduce the goals of the competition. Ideally automation leveraged to reduce friction all around.
Hosts must consider disqualified entries for vulnerability management purposes. This is similar to how threat actors fix exploit code to build out a stronger proof-of-concept in cybersecurity. This is a stronger point if all findings get published publicly without redaction.
Hosts or red team competition platforms need to have an information hazard management strategy. Could there be a policy opportunity here?
Attack transfer testing for non-participating models should be considered as part of the lag-time between finding submission and public release of competition findings. This gives companies and maintainers whose models are affected (but not involved in the competition) a chance to remediate, and is a responsible approach considering the range of threat actors that will be interested in the results from the competition.