Talk Python to Me: #521: Red Teaming LLMs and GenAI with PyRIT
English is now an API. Our apps read untrusted text; they follow instructions hidden in plain sight, and sometimes they turn that text into action. If you connect a model to tools or let it read documents from the wild, you have created a brand new attack surface. In this episode, we will make that concrete. We will talk about the attacks teams are seeing in 2025, the defenses that actually work, and how to test those defenses the same way we test code. Our guides are Tori Westerhoff and Roman Lutz from Microsoft. They help lead AI red teaming and build PyRIT, a Python framework the Microsoft AI Red Team uses to pressure test real products. By the end of this hour you will know where the biggest risks live, what you can ship this quarter to reduce them, and how PyRIT can turn security from a one time audit into an everyday engineering practice.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/sentryagents'>Sentry AI Monitoring, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/agntcy'>Agntcy</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <h2 class="links-heading mb-4">Links from the show</h2> <div><strong>Tori Westerhoff</strong>: <a href="https://www.linkedin.com/in/victoriawesterhoff/?featured_on=talkpython" target="_blank" >linkedin.com</a><br/> <strong>Roman Lutz</strong>: <a href="https://www.linkedin.com/in/romanlutz/?featured_on=talkpython" target="_blank" >linkedin.com</a><br/> <br/> <strong>PyRIT</strong>: <a href="https://aka.ms/pyrit?featured_on=talkpython" target="_blank" >aka.ms/pyrit</a><br/> <strong>Microsoft AI Red Team page</strong>: <a href="https://learn.microsoft.com/en-us/security/ai-red-team/?featured_on=talkpython" target="_blank" >learn.microsoft.com</a><br/> <strong>2025 Top 10 Risk & Mitigations for LLMs and Gen AI Apps</strong>: <a href="https://genai.owasp.org/llm-top-10/?featured_on=talkpython" target="_blank" >genai.owasp.org</a><br/> <strong>AI Red Teaming Agent</strong>: <a href="https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/ai-red-teaming-agent?featured_on=talkpython" target="_blank" >learn.microsoft.com</a><br/> <strong>3 takeaways from red teaming 100 generative AI products</strong>: <a href="https://www.microsoft.com/en-us/security/blog/2025/01/13/3-takeaways-from-red-teaming-100-generative-ai-products/?featured_on=talkpython" target="_blank" >microsoft.com</a><br/> <strong>MIT report: 95% of generative AI pilots at companies are failing</strong>: <a href="https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/?featured_on=talkpython" target="_blank" >fortune.com</a><br/> <br/> <strong>A couple of "Little Bobby AI" cartoons</strong><br/> <strong>Give me candy</strong>: <a href="https://blobs.talkpython.fm/little-bobby-ai-1.png" target="_blank" >talkpython.fm</a><br/> <strong>Tell me a joke</strong>: <a href="https://blobs.talkpython.fm/little-bobby-ai-2.png" target="_blank" >talkpython.fm</a><br/> <br/> <strong>Watch this episode on YouTube</strong>: <a href="https://www.youtube.com/watch?v=N681L4BXTUw" target="_blank" >youtube.com</a><br/> <strong>Episode #521 deep-dive</strong>: <a href="https://talkpython.fm/episodes/show/521/red-teaming-llms-and-genai-with-pyrit#takeaways-anchor" target="_blank" >talkpython.fm/521</a><br/> <strong>Episode transcripts</strong>: <a href="https://talkpython.fm/episodes/transcript/521/red-teaming-llms-and-genai-with-pyrit" target="_blank" >talkpython.fm</a><br/> <br/> <strong>Theme Song: Developer Rap</strong><br/> <strong>Served in a Flask</strong>: <a href="https://talkpython.fm/flasksong" target="_blank" >talkpython.fm/flasksong</a><br/> <br/> <strong>---== Don't be a stranger ==---</strong><br/> <strong>YouTube</strong>: <a href="https://talkpython.fm/youtube" target="_blank" ><i class="fa-brands fa-youtube"></i>youtube.com</a><br/> <br/> <strong>Bluesky</strong>: <a href="https://bsky.app/profile/talkpython.fm" target="_blank" >@talkpython.fm</a><br/> <strong>Mastodon</strong>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <strong>X.com</strong>: <a href="https://x.com/talkpython" target="_blank" ><i class="fa-brands fa-twitter"></i>talkpython</a><br/> <br/> <strong>Michael on Bluesky</strong>: <a href="https://bsky.app/profile/mkennedy.codes?featured_on=talkpython" target="_blank" >@mkennedy.codes</a><br/> <strong>Michael on Mastodon</strong>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/> <strong>Michael on X.com</strong>: <a href="https://x.com/mkennedy?featured_on=talkpython" target="_blank" ><i class="fa-brands fa-twitter"></i>mkennedy</a><br/></div>
https://talkpython.fm/episodes/show/521/red-teaming-llms-and-genai-with-pyrit
