Anthropic partnered with HackerOne to run an AI red teaming challenge on Claude 3.5 Sonnet, aiming to test and improve its Constitutional Classifiers against harmful content, especially CBRN-related queries. From Feb 3–10, 339 researchers engaged in over 300,000 interactions, using various jailbreak methods. Four teams earned $55,000 in bounties, with one successfully applying a universal jailbreak. The exercise highlighted the value of external researchers in identifying vulnerabilities and reinforcing AI safety, demonstrating that collaborative red teaming is vital for building secure and resilient AI systems.

Join for free to read