Galileo

Most teams running eval pipelines on multimodal agents are silently missing two failure modes. The first: bad inputs that look fine to the eval layer but break the agent. A blurry product photo. A customer call with three seconds of dropouts every minute. A PDF that was scanned poorly. The agent produces a confident, completely wrong response. The eval pipeline sees a clean transcript and a clean output. Everything passes. Meanwhile, the user leaves unhappy. The second: using text-based evals on a non-text input. Did the agent identify the safety vest in the photo? Did it correctly infer customer frustration from tone, not words? Did it count the items in the shelf image accurately? Text-only evals can’t answer these questions. We just shipped Multimodal evals to fix both. If you're building agents for PDF extraction, image description, visual compliance, or support-call analysis, give it a run on your own traffic. Read the docs: https://lnkd.in/gM9qjSm3

Most teams running eval pipelines on multimodal agents are silently missing two failure modes. The first: bad inputs that look fine to the eval layer but break the agent. A blurry product photo. A customer call with three seconds of dropouts every minute. A PDF that was scanned poorly. The agent produces a confident, completely wrong response. The eval pipeline sees a clean transcript and a clean output. Everything passes. Meanwhile, the user leaves unhappy. The second: using text-based evals on a non-text input. Did the agent identify the safety vest in the photo? Did it correctly infer customer frustration from tone, not words? Did it count the items in the shelf image accurately? Text-only evals can’t answer these questions. We just shipped Multimodal evals to fix both. If you're building agents for PDF extraction, image description, visual compliance, or support-call analysis, give it a run on your own traffic. Read the docs: https://lnkd.in/gM9qjSm3

𝗧𝗵𝗲 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻 𝗶𝘀 𝗻𝗼𝘁 𝗷𝘂𝘀𝘁 𝘄𝗵𝗲𝘁𝗵𝗲𝗿 𝘆𝗼𝘂 𝗵𝗮𝘃𝗲 𝘁𝗵𝗲𝗺. 𝗜𝘁 𝗶𝘀 𝘄𝗵𝗲𝗿𝗲 𝘁𝗵𝗲𝘆 𝗹𝗶𝘃𝗲. One of the strongest examples from our new blog: an agent team thought its prompt injection guardrail was working. The dashboard looked clean. The model said risk was low. But the system was only catching 2 of the 10 OWASP scenarios. The rest, indirect injection, zero-shot attacks, multi-turn manipulation, cross-agent propagation, were effectively invisible. That is the trap with agent security: coverage gaps can look exactly like safety. That story is one of several in our new blog, written with Bianca, Michael, and Obine A. 𝗟𝗮𝗿𝗴𝗲 𝗲𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲𝘀, 𝗲𝘀𝗽𝗲𝗰𝗶𝗮𝗹𝗹𝘆 𝗶𝗻 𝗳𝗶𝗻𝗮𝗻𝗰𝗶𝗮𝗹 𝘀𝗲𝗿𝘃𝗶𝗰𝗲𝘀, 𝗮𝗿𝗲 𝗺𝗼𝘃𝗶𝗻𝗴 𝗳𝗿𝗼𝗺 “𝘄𝗲 𝗸𝗻𝗼𝘄 𝗢𝗪𝗔𝗦𝗣 𝗺𝗮𝘁𝘁𝗲𝗿𝘀” 𝘁𝗼 “𝘄𝗲 𝗰𝗮𝗻 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗲𝗻𝗳𝗼𝗿𝗰𝗲 𝗶𝘁.” And the pattern keeps showing up across teams: security controls cannot live inside every individual agent. They need to be centrally owned, centrally updated, and enforced consistently across every production use case. 𝗪𝗵𝗮𝘁 𝘁𝗲𝗮𝗺𝘀 𝗰𝗮𝗿𝗲 𝘁𝗵𝗲 𝗺𝗼𝘀𝘁 𝗮𝗯𝗼𝘂𝘁: → Prompt injection is much broader than most teams assume. Direct attacks are only one slice of the problem. Indirect retrieval-based injection, multi-turn steering, and cross-agent contamination all need coverage. → PII leakage keeps coming up as a hard gating requirement, especially in banking. One quote from the piece stayed with me: “We don’t need to prove that PII doesn’t leak 99% of the time. We need to prove it doesn’t leak, period.” → Heuristic controls hit a wall fast. Regex, keyword filters, and custom rules help early, but they create maintenance burden, leave coverage gaps, and do not scale as agent use cases multiply. → Policy updates need to propagate immediately. When a new threat vector appears or requirements change, security teams need one policy definition that every agent picks up within seconds, across ADK, LangGraph, CrewAI, or custom stacks. 𝗧𝗵𝗲 𝗲𝗻𝗱𝗴𝗮𝗺𝗲 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻: 𝗖𝗮𝗻 𝘆𝗼𝘂 𝗼𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝗮𝗹𝗶𝘇𝗲 𝗢𝗪𝗔𝗦𝗣 𝗮𝗰𝗿𝗼𝘀𝘀 𝘁𝗵𝗲 𝗲𝗻𝘁𝗲𝗿𝗽𝗿𝗶𝘀𝗲 𝗳𝗼𝗿 𝗮𝗹𝗹 𝘁𝗵𝗲 𝗮𝗴𝗲𝗻𝘁𝘀? Learn more...

Building a crew of agents is the easy part. Knowing what they're doing and stopping them when they're off course is where most teams get stuck. We're co-hosting a live session with CrewAI to cover exactly that. Join Galileo co-founder and CTO Yash Sheth and CrewAI founder and CEO João (Joe) Moura as they walk through how to govern multi-agent systems at scale, covering behavior, cost, and compliance. In this session, you'll learn: → How to enforce safety and security policies in CrewAI agents. → How to steer agents to the best models and fallback tools at runtime to improve accuracy and control token costs → How to govern all your agents, whether CrewAI, internal or third-party, with one centralized set of policies → How to include non-technical stakeholders (such as risk and compliance) in writing or maintaining policies – no coding required Last chance to register here: https://lnkd.in/gRjQNnwM

EU AI Act audits begin in August. The theoretical conversation about AI governance just became a procurement requirement with deadlines attached. Large banks now require security sign-off before any agentic use case reaches production. Risk teams are blocking deployments until observability and governance are in place. Many enterprises guard against only 2-3 of the 10 OWASP threat categories for agentic AI. Prompt injection guardrails cover approximately 2 of 10 defined injection variants. Entire attack categories, tool misuse, identity abuse, privilege escalation, and inter-agent communication risks, remain invisible to existing controls. Traditional application security rests on one foundational property: the system under protection is a constrained actor with fixed logic. Agentic AI is an adaptive actor with open-ended behavior, and is fundamentally different to secure. We just published Operationalizing the OWASP Top 10 for Agentic AI; a security whitepaper that shows how to turn the OWASP framework into enforceable, auditable controls using a central control plane architecture. Read our whitepaper to: – Understand why agents break traditional application security models – Map every OWASP ASI01–ASI10 threat to concrete detection controls – Architect a central control plane that enforces policy across every agent – Separate platform-level and per-agent controls without duplicating effort – Close the gap between prompt injection guardrails and full OWASP coverage – Build an immutable audit trail regulators and CISOs will accept –  Apply the same infrastructure to GDPR, EU AI Act, and internal requirements  –  Validate OWASP threat coverage with aligned test suites, not generic benchmarks The enterprises that treat OWASP as a checkbox will fall behind. The ones that treat it as the architectural blueprint for agentic AI governance will lead. Download the whitepaper here: https://lnkd.in/ge-nDAvJ Written by:  Pratik Bhavsar, AI Engineer, Galileo Michael Branconier, FDE, Galileo Bianca DePriest, Enterprise Sales, Galileo Obine Adoh, Security, Galileo

ICYMI: Another week, another set of workflow improvements across our platform! Here's what just went live in this week's release notes 👇 Playgrounds: Playgrounds can now dynamically detect datasets’ variables, making it easier to add variables to playground prompts. Logs filtering: Logs can now automatically display available columns to filter. Custom model integration: Galileo’s custom model integrations now support model properties for users who wish to further customize LLM integration parameters. OpenAI models: GPT 5.4 Mini and Nano now available across Playground, Prompt store, Synthetic Data Generation, and Metrics Hub Annotation Queues (Enterprise Beta): Keyboard shortcuts and auto-advance to speed up annotator workflows Read more here: https://lnkd.in/gv5zwJKV

Governance is the part nobody talks about. Until something breaks. Join us for a live webinar on April 21st to learn how to keep your agentic systems safe, reliable, and in control — at scale. Galileo co-founder & CTO Yash Sheth and CrewAI founder & CEO João (Joe) Moura will cover: → Enforcing safety and security policies in CrewAI agents → Steering agents to the best models and fallback tools at runtime → Governing all your agents with one centralized set of policies → Bringing non-technical stakeholders into the loop — no coding required 🗓 April 21 | 10am PT 🎟 Register: https://lnkd.in/gRjQNnwM #AgenticAI #CrewAI #AI