Picture this: it’s an tiresome afternoon, you’re at your desk wondering why that XYZ vendor invoice hasn’t been paid yet. You whisper to your phone: “Hey, check the XYZ vendor’s status, trigger payment if all’s clear, and send the email confirmation to finance.” And that’s it … done.

The response is not just a simple chat answer, but multiple connected steps: lookup, analysis, decision, execution, feedback. That’s the Agentic AI we know these days. But, do you really trust it?

In the new paper Agentifying Agentic AI by Virginia Dignum and Frank Dignum, we get a roadmap for why so many of today’s autonomous-AI dreams fall short — and what we (as founders, builders, leaders) can do about it.

Let me walk you through what I learned — mixing the theory with what I’ve seen, lived, built — and what this means for you and me as founder-builders in 2025.

The Problem: Autonomy without Agency

In our startup world, we constantly hear about autonomous agents, AI assistants, chain-of-thought, multi-step auto, etc. But as it happens, nearly all of those are behaviorally autonomous, not agentically grounded.

Let’s unpack:

• While modern large-language-model (LLM)-based systems can act and generate, they often lack explicit internal states, commitments, reasoning architectures, and social/institutional context.
• Example in the paper: ask an AI-agent system to book a flight. It may do so OK one day, but on another day shift parameters and book a return on the wrong date just because the model saw a “sale”. Because there is no guarantee of goal-directed behavior or verifiable reasoning.
• The consequence? When you deploy a multi-agent workflow — many “agents” acting in concert, or interacting with humans/institutions — you get the black box problem: you don’t know why an action was taken, whether it aligns with your governance or business logic, or how to trace/rationalize failures.

I’ve seen this first-hand. In one of my ventures, we built a “smart pipelines manager” using LLMs. At first it looked magical. But then it started “deciding” to skip a crucial QA check because the logic had drifted — and we had no clear way to audit what went wrong. The behavior was unpredictable; we ended up delaying the rollout.

To make true “agency” possible, we should borrow from the 30 years of research in the autonomous agents and multi-agent systems (AAMAS) community.

From Autonomy to Agency: What’s Really Missing

In the AAMAS world, agents are built with belief-desire-intention (BDI) models: beliefs capture the state of the world, desires the goals, intentions the chosen commitments. The structure allows you to ask: Why did the agent do this? What goal was it pursuing? How did it decide this plan? Most modern LLM-based “agents” don’t have that explicit internal state — they operate on statistical patterns. That leaves you building something that acts but you can’t verify or interpret.

When multiple agents (or humans + agents) interact, conventional agent frameworks use structured communication protocols: requests, commitments, promises — all understood unambiguously. The paper points out that many contemporary agentic systems rely on free-form natural language, which lacks rigorous semantics. This means agents talking to each other (or to us) can misinterpret, fail to coordinate, or produce brittle behavior.

When it comes to Incentives, Coordination, Social context, that’s where it gets more interesting for startups and founders. The idea that agents aren’t just reactive tools, they are entities in an ecosystem with aligned incentives, norms, roles, and governance. So if you build an ecosystem of agents (tool-agents, human-agents, institutional agents), you must think about why they cooperate, how they negotiate, what they value.

If you deploy agents into the wild — against humans or any other systems — you should embed mechanisms for reputation, trust, and strategic decision-making. The authors highlight that current agentic systems often lack memory of past behavior, lack trust metrics, and thus cannot reliably participate in socially-aware systems.

Why This Matters Now?

The timing couldn’t be better. According to recent industry surveys, a large majority of organizations report using AI in at least one business function. Scaling agent-style systems is a dominant trend.

But beware, one major analytics firm estimates that over 40 % of agentic-AI projects will be cancelled by end-2027, due to governance, value, and coordination failures.

What this means: The “low-hanging fruit” of AI (e.g., features-based automation) is being eaten. The next frontier is agents that interact, decide, act over time, coordinate with other agents — and that’s where business value is huge, but so is complexity and risk.

If you’re a founder considering building or buying agentic AI, I’d advise you to shift your mindset: from “We’ll build the smartest assistant” to “We’ll build the capable actor that participates in a system of actors”. The difference is subtle but massive.

My Founder Playbook: What I’d Do Differently

When I reflect on our failures and pivots, this is how I’d approach agentic AI today. First, I’d define the agentic job more explicitly. Not just automate vendor payments but “Agent X will monitor vendor status, evaluate risk, execute payment if criteria Y are met, and alert/involve humans if exceptions occur.” I’d define the world state it must inspect, the goal it must commit to, the fallback sequences.

Then, I’d build a modular architecture: one module for state-beliefs (what does the world look like?), one for intentions (what goal is committed?), one for actions (what will you do?), one for audit/logging (how do we trace what happened?). Even if the heavy lifting is done via an LLM, wrap it in structured scaffolding for introspection and accountability.

Next, I’d think about the ecosystem: What other agents exist? Humans? Tools? Systems? How will they communicate? I’d define protocols: “Agent A completes a task, sends message type X to Agent B, which then takes action Y.” I’d define roles, interfaces, fail-safe hand-offs.

Importantly, I’d build in governance from day one. Who owns the agent’s decisions? What metrics define success/failure? What reputation or trust mechanisms track the agent over time? If the system misbehaves, how do we intervene? How do we revert?

Finally, I’d launch narrow. Start with a specific use-case where the interaction surface is limited, breakpoints are observable, and human oversight remains in place. Prove value, refine intentions and coordination, then edge into broader scopes (multi-agent flows, cross-domain coordination).

Looking Ahead: The Big Picture

Agentic AI is a step forward from passive prediction to active, interactive, context-aware behavior. But without structure, coordination and social embedding, the word “agentic” simply becomes “autonomous but opaque”.

For us founders, the opportunity is real: build systems where agents not only do but reason, coordinate, negotiate, trust, and adapt in human-socio-technical contexts. But success will go to the teams who invest in the scaffolding behind the “smart assistant” — not just in the flashy demo.

We’ll see major shifts: more agents interacting, more humans delegating decisions, more institutional/organizational embedment of agents. But with those shifts come demands: transparency, auditability, governance, value alignment. The winners will ask: What’s our agent’s role, in what system, under whose oversight?

If I could go back and talk to “founder-me” two years ago, I’d say: “Don’t just build something smart; build something meaningful. Don’t just automate; orchestrate.” Because in this new agentic era, your biggest challenge won’t be will the AI understand? but will the ecosystem (agent + agent + human + institution) function reliably, ethically, predictably, and with value.

If you’re thinking of diving into agentic AI, lean on frameworks that emphasize intention, communication, coordination, governance. Build your agent not as a lone star, but as part of a constellation.