We have arrived at the Hard Question

There is a moment in every major technology transition when the question shifts from “Should we?” to “How fast can we?”, and then, almost immediately, to “Why isn’t this working the way we expected?”

Agentic AI has arrived at that third question faster than any enterprise technology in recent memory.

This isn’t a piece about whether agentic AI matters. That question is settled. It’s about what it actually takes to make it work inside a real enterprise, and what separates organizations building genuine competitive advantage from those accumulating expensive technical debt dressed up as innovation.

But here is what those numbers don’t tell you.

35% of enterprises had already adopted AI Agents by 2023, yet the majority of those deployments never reached production

The adoption curve is steep. The success curve is not.

The gap between those two lines is where most organizations are right now, and understanding why it exists is the most important strategic work a CTO can do in 2026.

6 Trends Every CTO Needs to Understand Right Now

Agentic AI is not one shift. It is six happening simultaneously across architecture, workforce, infrastructure, security, cost, and control. Miss one and the others will surprise you.

Trend 1: Multi-Agent Systems Are the New Default Architecture

The question is no longer whether to deploy an agent. It is how to coordinate ten of them. Networks of specialized agents collaborating, delegating, and handing off context are becoming the standard deployment model.

The orchestration layer is now a core architectural decision, not a detail to figure out later.

Trend 2. The “Silicon Workforce” Is Being Managed

Agents are increasingly treated as collaborators, not tools. That changes everything: how teams are structured, who owns accountability, and how performance gets measured.

The organizations moving fastest are not just deploying agents. They are redesigning the work itself: identifying which decisions require human judgement, which can be delegated to agents, and how handoffs between the two are governed.

Trend 3. Purpose-Built Infrastructure Is Replacing General Stacks

Your stack was built for humans making decisions at human speed. Agents operate in real time, across systems, continuously. That gap, between what agents need and what most enterprise architectures provide, is where the majority of pilots quietly stall. Not because the model failed. Because the infrastructure was never ready.

Agentic workloads demand infrastructure that can operate in real time, across systems, and at machine speed, including:

• Real-time data access across enterprise platforms
• Modular, API-first architectures that agents can interact with dynamically
• Secure identity and access management for autonomous systems
• Agent-compatible data pipelines capable of continuous decision loops

Trend 4. The Board Is Now Asking About ROI, And CTOs Need Answers

AI budgets are under scrutiny. Deployment counts are not a business outcome. Organizations without clear cost governance frameworks for agentic workloads are accumulating spend without visibility, and boards are starting to ask hard questions that require better answers than “we’re building capability.”

CTOs who can articulate cost-per-outcome, error rates, and human oversight ratios are winning budget conversations. Those who cannot are losing them, or worse, discovering their AI programs are being quietly deprioritized.

Trend 5. Vertical Industry Adoption Is Accelerating

Agentic AI is rapidly moving beyond generic enterprise IT experimentation and into industry-specific operational deployments. The relevant benchmark is no longer the general enterprise AI market; it is how organizations within your industry are operationalizing agentic systems:

• Healthcare: reducing administrative burden and improving care coordination
• Financial services: real-time compliance monitoring and fraud detection
• Legal operations: document review, case analysis, and contract intelligence
• Supply chains: logistics planning, inventory optimization, and demand forecasting

Organizations in these verticals are embedding AI agents into domain-specific workflows, not running isolated experiments. The gap between early adopters is already measurable in operational efficiency and cost structure.

Trend 6. Agent Identity Is the Security Gap Nobody Is Talking About

95% of the enterprises have no identity protections for their autonomous agents - exposing transactions, not just content.

Agents do not generate content; they execute transactions. Without verifiable identity, scoped permissions, and audit trails, you have autonomous systems with unchecked access running inside your enterprise. That is not a future risk. It is a current one.

The security frameworks most organizations rely on were designed for human users and static applications. Neither model accounts for autonomous agents that act continuously across multiple systems, without direct human oversight at the point of execution

Trend 7. Governance Is Finally Catching Up But Not Fast Enough

Organizations are beginning to establish frameworks designed to address risks unique to autonomous systems, including:

• Autonomous decision errors that directly affect business operations
• Prompt injection attacks that manipulate agent behavior
• Credential misuse or unauthorized system access
• Shadow AI deployments operating outside official governance structures

The risks are operational, not theoretical, and regulatory scrutiny is accelerating.

Governance is no longer a compliance checkbox. It is the mechanism that keeps you in control of systems capable of acting independently, at machine speed, across your most sensitive infrastructure.

Conclusion

The organizations that will define the next decade of enterprise technology are not the ones that deployed the most agents. They are the ones who deployed them thoughtfully, with the infrastructure to support them, the governance to control them, and the clarity to measure what they actually deliver.

Agentic AI is not a pilot project anymore. It is an operational reality, and the gap between organizations that treat it that way and those still running experiments is widening every quarter.

For CTOs, the mandate is clear: move past deployment metrics and into operational maturity. That means

• Investing in orchestration architecture before you scale agent networks,
• Building identity and access frameworks before you expose sensitive systems
• Establishing cost governance before your board builds it for you.

The enterprises winning with agentic AI share one characteristic: they stopped asking whether AI agents could do the work, and started engineering the conditions under which agents could do the work well, reliably, securely, and at scale.

The transition from “Should we?” to “How fast?” has already happened. The organizations asking “How do we make this work?” right now are the ones who will be setting the benchmarks everyone else chases in 2027.

The question is whether you are building toward that position or watching others do it.

The Rise of the Agentic Enterprise: What Every CTO Must Know About AI Agents in 2026 was originally published in ShellKode Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

For CXOs looking to achieve true competitive advantage, the challenge isn’t data access; it’s actionable insight. This is why they are choosing Snowflake Intelligence to build their next generation of Data Agents. Snowflake Intelligence shifts the paradigm from mere data access to true data understanding and collaboration across the enterprise. It moves beyond traditional BI and Text-to-SQL by embedding AI-driven intelligence directly into the data fabric, enabling these agents to have a dynamic dialogue with data that evolves with business needs and user context. Snowflake Intelligence addresses the complexity and fragmentation in modern data ecosystems by unifying knowledge and automating relevance, thus empowering Data Agents to not just execute simple queries, but to assess your current business landscape and generate decisions that lead to actions, anticipate risks, and innovate, turning the data cloud itself into an adaptive, intelligent partner essential for business transformation.

Setting the Stage, The Data Dilemma

In today’s fast-paced corporate environment, business stakeholders are acutely aware that their organizations are sitting on a goldmine of data. Yet, the critical challenge remains the same: transforming this data abundance into actionable, decision-driving insight.

The fact is, traditional Business Intelligence (BI) and simple Text-to-SQL solutions are no longer sufficient to meet strategic demands. These tools fail to bridge the pervasive organizational gaps that prevent true data mastery.

Organizations across industries face several critical challenges when trying to leverage their data effectively:

• Siloed Information: Data scattered across multiple systems and sources, like structured, semi-structured, and unstructured Data, makes comprehensive, unified analysis nearly impossible.
• Time-Consuming Processes: Custom dashboard creation and report generation can take weeks, by which time business conditions may have dramatically changed, rendering the insights obsolete.
• Limited Context: Traditional BI tools only answer the historical question of “what happened”, but critically struggle to explain the deep, operational “why” behind trends and anomalies.
• Resource Constraints: Data Analysts spend up to 80% of their time responding to repetitive ad hoc requests rather than focusing on high-value, strategic analysis that moves the business forward.
• Technical Barriers: Business users lack the SQL or technical skills needed to query databases directly, creating high dependency on already constrained data teams.

These fundamental challenges result in delayed decisions, missed opportunities, and enterprise-wide frustration. For the CXO focused on competitive advantage, the need is clear: a paradigm shift is required to empower adaptive, autonomous decision-making a need that transcends simple dashboards and points directly toward building intelligent Data Agents.

Introducing Snowflake Intelligence: The Intelligent Data & AI Partner

​​If the challenge is the gap between data and decisive action, Snowflake Intelligence is the paradigm shift that closes it. It is not just another BI tool; it is the Intelligent Data & AI Partner embedded directly into Snowflake, unifying data, knowledge, and AI capabilities to enable true, adaptive decision-making.

Moving Beyond Querying: The Shift to True Data Understanding

Snowflake Intelligence fundamentally changes the user’s relationship with data. It moves the conversation from the restrictive, technical questions of “Show me the numbers” (traditional querying) to the strategic, open-ended questions of “What should I do next?” or “Why did this happen?”

This shift is powered by:

• Holistic Context: Unlike systems that only look at structured data tables, Snowflake Intelligence allows Data Agents to pull context from all data types, structured tables, unstructured documents, PDFs, customer tickets, and median to synthesize a comprehensive understanding of the business landscape.
• Dynamic Dialogue: It supports multi-turn conversations, allowing users (or agents) to ask follow-up questions that build on previous insights, creating a continuous, learning dialogue that evolves with user context and business needs.

The AI-Driven Fabric: Intelligence Embedded, Not Bolted On

The critical advantage of Snowflake Intelligence is that its AI is embedded directly within the data fabric, powered by the Snowflake Cortex suite of services. This eliminates the complexity, cost, and risk associated with moving data to external, third-party AI platforms.

Core Capabilities: Unifying Knowledge and Automating Relevance

The power of Snowflake Intelligence for building Data Agents lies in its ability to orchestrate tasks across all your data assets using specialized, managed tools:

Practical Guide: Getting Started with Snowflake Intelligence

Prerequisites and Setup

To begin using Snowflake Intelligence, you’ll need:

• Access to a Snowflake account
• Regional access to supported AI models or cross-region inference enabled
• Existing data models, schema, and tables in Snowflake

Step 1: Create the Snowflake Intelligence Object

The first step is establishing the intelligence framework in your account:

 — Create the Snowflake Intelligence object
 CREATE SNOWFLAKE INTELLIGENCE SNOWFLAKE_INTELLIGENCE_OBJECT_DEFAULT;

— Set up the database and schema for agents
CREATE DATABASE IF NOT EXISTS snowflake_intelligence;
 GRANT USAGE ON DATABASE snowflake_intelligence TO ROLE PUBLIC;

CREATE SCHEMA IF NOT EXISTS snowflake_intelligence.agents;
 GRANT USAGE ON SCHEMA snowflake_intelligence.agents TO ROLE PUBLIC;

— Enable cross-region inference if needed
 ALTER ACCOUNT SET CORTEX_ENABLED_CROSS_REGION = ‘ANY_REGION’;

Note: Only the ACCOUNTADMIN role has the CREATE SNOWFLAKE INTELLIGENCE ON ACCOUNT privilege required for this step.

Step 2: Configure AI Agents

Agents are the building blocks of Snowflake Intelligence.

One of the most powerful aspects of Snowflake Intelligence is the ease with which agents can be created and configured. The process is remarkably straightforward:

• Navigate to AI & ML » Agents in the Snowflake interface
• Select the Snowflake Intelligence tab
• Click “Create Agent” to launch the simplified agent builder
• Provide a name and description for your agent
• Select the data sources and tools the agent should have access to
• Configure permissions and grant USAGE privileges to appropriate roles

The entire process can be completed in minutes, without writing complex code or configuration files. The intuitive interface guides you through each step, making it accessible even to users without deep technical expertise.

Tools and Capabilities for Agents

When creating an agent, you can equip it with various tools that dramatically expand its analytical capabilities:

Cortex Analyst Integration

To enable Cortex Analyst for your agent, you’ll work with semantic models defined in YAML format:

— Grant necessary database and schema privileges
 GRANT USAGE ON DATABASE snowflake_intelligence TO ROLE data_engineer;
 GRANT USAGE ON SCHEMA snowflake_intelligence.agents TO ROLE data_engineer;

 — Grant Cortex access via database roles
 GRANT DATABASE ROLE SNOWFLAKE.CORTEX_USER TO ROLE data_engineer;

 — Create a stage for semantic models
 CREATE STAGE IF NOT EXISTS semantic_models_stage;

• Define your semantic model in YAML (example structure)
• Upload semantic_model_sales.yaml to the stage via SnowSQL or UI
• The YAML file defines tables, columns, metrics, and relationships

Example semantic model YAML structure:

name: sales_semantic_model
 description: Sales performance and customer analytics
 tables:

· name: sales_data
 description: Transaction-level sales data
 base_table:
 database: SALES_DB
 schema: PUBLIC
 table: SALES_DATA
 columns: 
o name: order_date
 description: Date when order was placed
 data_type: DATE 
o name: revenue
 description: Total order value in USD
 data_type: NUMBER

• Reference the semantic model when creating agents
• The agent will use this YAML to understand your data structure

Cortex Search Integration

To add Cortex Search capabilities to your agent:

 — Create a stage for documents
 CREATE STAGE sales_documents_stage
 DIRECTORY = (ENABLE = TRUE);

 — Upload documents (via SnowSQL or web interface)
 PUT file://local/path/sales_reports/*.pdf @sales_documents_stage;

— Create Cortex Search service
 CREATE CORTEX SEARCH SERVICE sales_docs_search
 ON sales_documents
 WAREHOUSE = search_wh
 TARGET_LAG = ’30 minutes’
 AS (
 SELECT
 relative_path AS doc_name,
 file_url,
 GET_PRESIGNED_URL(@sales_documents_stage, relative_path) AS content
 FROM DIRECTORY(@sales_documents_stage)
 );

With Cortex Search, your agent can answer questions like “What does our product documentation say about integration with AWS services?” by searching through thousands of documents and synthesizing relevant information.

Additional Tools for Agents

Beyond Cortex Analyst and Cortex Search, agents can be equipped with:

• Cortex Functions: Pre-built AI functions for sentiment analysis, translation, summarization, and text classification
• Custom Python Functions: User-defined functions for specialized analytics or business logic
• External API Access: Integration with third-party data sources and services
• Snowflake Marketplace Data: Access to external datasets for market intelligence and benchmarking
• Data Quality Tools: Automated data profiling, anomaly detection, and validation checks

Agent Configuration Example

Here’s how to create a comprehensive sales intelligence agent through the Snowflake UI:

1. Go to AI & ML → Agents → Create Agent

2. Agent Configuration:

• Name: sales_intelligence_agent
• Description: Analyzes sales performance, customer behavior, and revenue trends

3. Select Tools as required:

• Cortex Analyst (for structured data queries)
• Cortex Search (for document search)
• Sentiment Analysis (Cortex Function)
• Custom forecast function

4. Choose Data Sources:

• Database: SALES_DB, Schema: PUBLIC
• Tables: sales_data, customer_data, product_catalog
• Stage: @sales_documents_stage

5. Set Permissions:

• Grant USAGE to roles: sales_team, sales_manager

6. Click “Create” to deploy

After creation, test your agent directly in the UI:

Simply type natural language questions in the Snowflake Intelligence interface:

• “What were our top-performing products last quarter and why?”
• “Show me customer sentiment trends for premium products”
• “Forecast next quarter’s revenue based on current trends”

The agent now has the combined power of structured data analysis (via Cortex Analyst), document search (via Cortex Search), sentiment analysis on customer feedback, and forecasting capabilities — all accessible through simple natural language questions in the UI.

Why Work With Us for Snowflake Intelligence Adoption

Adopting Snowflake Intelligence lays the foundation for your next-generation Data Agents, but moving from a powerful platform to measurable, enterprise-wide outcomes requires specialized expertise. This is where Shellkode steps in as your dedicated Snowflake Partner Solutions Expert.

We don’t just implement technology; we architect decision-ready intelligence by combining our deep domain knowledge with end-to-end agentic development mastery on Snowflake Intelligence.

The Shellkode Advantage: Transforming Data into Action

We position Snowflake Intelligence as an adaptive, intelligent partner in your transformation journey. Our unique value proposition ensures that your Data Agent implementation is not just technically sound, but strategically aligned with your C-suite objectives:

Conclusion

The fundamental truth driving modern enterprise strategy is clear: data systems and AI systems are inexorably moving closer, forming a single, intelligent fabric.

The adoption of Snowflake Intelligence confirms this convergence. By unifying the high-performance Data & AI Cloud with the native intelligence of Snowflake Cortex, you achieve a catalyst for transformation.

By partnering with Shellkode and choosing Snowflake Intelligence, you transform your approach: you move from manual querying to autonomous, dynamic conversations with data. This accelerates research, provides adaptive foresight, and empowers your organization with decision-ready intelligence. Build the autonomous Data Agents today to secure your competitive advantage and transform your enterprise.

Building a Reliable, Enterprise Data Agent with Snowflake Intelligence was originally published in ShellKode Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today, ShellKode is proud to join AWS as a launch partner in the AWS Pattern Partners program, an invite-only initiative that brings together a select cohort of consulting partners to define how enterprises adopt next-generation AI and emerging technologies on AWS.

As an AWS Pattern Partner, ShellKode brings deep experience in Agentic AI, GenAI, enterprise modernization, and responsible AI operations. The program accelerates AI adoption by codifying success into repeatable, scalable patterns, including architectures, operating models, guardrails, and delivery playbooks that have already been validated with customers.

For enterprise leaders, ShellKode’s selection signals that AWS has reviewed and endorsed both our outcomes and our operating model across multiple real-world deployments. Organizations looking to modernize processes, adopt AI safely, and manage new regulatory requirements now gain access to proven patterns that deliver measurable business impact.

Our Focus Patterns with AWS

As part of the Pattern Partners program, ShellKode will help scale four high-impact enterprise patterns:

• Process to Agent (P2A) — Transforming business workflows into autonomous, production-ready AI agents.
• Agent to Agent (A2A) — Enabling multi-agent collaboration for complex enterprise operations.
• Modernization with GenAI — Accelerating legacy modernization and automated conversions using GenAI.
• Responsible AI Operations (RAI-Ops) — Ensuring enterprise-grade AI governance, evaluation, observability, and safety.

These patterns represent areas where we have already delivered strong customer outcomes across logistics, retail, BFSI, manufacturing, and technology sectors.

Our Flagship Pattern: Enterprise Process-to-Agent (P2A)

The customer challenge

Across industries, organizations face a common reality:

• Processes are fragmented across systems, slowing automation.
• AI proofs-of-concept rarely scale due to governance and compliance constraints.
• Teams lack clear guardrails, roles, and operating models for enterprise AI.
• Regulations and internal standards evolve rapidly, creating adoption friction.

This makes it difficult to move from promising prototypes to enterprise-grade, production-ready Agentic AI.

Our joint approach with AWS

ShellKode and AWS have co-designed the P2A Pattern, a blueprint that allows enterprises to convert complex workflows into autonomous agents using:

• AWS-native architecture leveraging Amazon Bedrock, Agents for Bedrock, Amazon OpenSearch, Amazon ECS, DynamoDB, CloudWatch, and other foundational services.
• A clear operating model covering governance, security, roles, runbooks, and cross-functional responsibilities.
• Accelerators including pre-built connectors, compliance packs, policy templates, dashboards, monitoring frameworks, and reusable agents.

The pattern is currently being refined through a time-boxed incubation with multiple lighthouse customers. As the blueprint stabilizes, AWS will make the Pattern Package available globally for rapid and consistent adoption.

Early results

Early adopters are already reporting measurable results using the P2A pattern:

• Time to first value reduced to under a few weeks, even in regulated environments.
• 30–60% cycle-time reduction in targeted workflows once agents are deployed.
• Higher accuracy and stronger compliance posture due to embedded guardrails.
• Improved analyst and developer productivity through automated workflows and reusable components.

For example, in one implementation for a global enterprise (customer name masked), the P2A pattern helped reduce a multi-step operational workflow by over 40%, without compromising auditability or internal controls. These early wins directly guide which use cases we prioritize next.

As these outcomes are validated across additional lighthouse customers, the Pattern Package will be rolled out to AWS field teams worldwide to support customers across industries and regions.

How the Pattern Partners Program Helps Customers

When enterprises engage ShellKode through the Pattern Partners program, they begin not from scratch but from a validated blueprint:

• Proven architectures
• Repeatable operating models
• Guardrails, runbooks, and compliance frameworks
• Pre-built accelerators and integrations
• Joint engagement with AWS Consulting COE, AWS service teams, and ShellKode experts

This enables fast but responsible experimentation. Organizations can move from idea to pilot within weeks while ensuring enterprise-grade security, auditability, and governance.
The program also includes a structured path from pilot to scale, enabling multi-region, multi-unit deployments with clear controls and observability.

ShellKode works closely with AWS solution architects and product teams to ensure the patterns remain aligned to the latest AWS innovations in GenAI, agentic systems, and responsible AI.

Partner Perspective

“Joining AWS Pattern Partners is a strategic milestone for ShellKode. Through patterns like P2A, we are transforming our strongest customer successes into a clear, repeatable path for enterprise AI adoption enabling organizations to move from pilots to production with greater speed, safety, and confidence.”
 — Bhuvanesh, CTO, ShellKode

AWS Perspective

“AWS created Pattern Partners to work with a select cohort of builders who can set the standard for how enterprises adopt emerging technology on AWS. ShellKode brings deep expertise in Agentic AI and a proven P2A pattern that is already delivering measurable outcomes. We look forward to scaling this work together across regions and industries.”
 — Brian Bohan, Managing Director, Consulting COE, AWS

Next Steps

Customers interested in understanding how these patterns can accelerate their AI roadmap can contact ShellKode at Link.

Organizations that want to explore the P2A pattern in detail can request a focused discovery session. In this session, AWS and ShellKode jointly map business challenges to the pattern, estimate potential impact, and define a practical path to adoption.

Together, ShellKode and AWS look forward to helping enterprises turn complex business challenges into repeatable, scalable patterns for growth.

ShellKode Named AWS Pattern Partner to Accelerate Enterprise AI Modernization was originally published in ShellKode Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

What Worked in Retail, Doesn’t Work Anymore

Retail used to be about great products. Now, it’s about great experiences. The way people discover, evaluate, buy, and seek customer service capabilities has changed dramatically. Customers expect speed, personalization, real-time updates, and intelligent automation at every step. And they’re no longer comparing brands within the same category they’re comparing against their best digital experiences.

This is where Generative AI in retail sets a new standard. For D2C brands and retail leaders, it’s more than automation, it’s the engine behind intelligent, human-like interactions across the entire retail customer journey. GenAI understands context, adapts in real time, and delivers hyperpersonalized experiences at scale

According to McKinsey, retailers using AI for personalization see a 10–20% uplift in customer satisfaction and 5–15% increase in revenue.

The Shift: Customers Changed, Tech Didn’t Until GenAI

Today’s shoppers want more than transactions. They want:

• Real-time personalization
• Language and cultural fluency
• Visual confidence (great images, rich info)
• Conversational support that solves, not stalls
• Omnichannel support from web to WhatsApp to checkout

Legacy tech can’t meet these demands at scale. GenAI can. Where older AI automated tasks, Generative AI in retail crafts:

• Personalized product descriptions
• Localized content in seconds
• Conversational flows that adapt
• Smart agents that handle 80%+ of support interactions autonomously

It’s not just smarter tech — it’s a scalable, human-like customer experience.

You can’t just drop AI into your business and expect magic. The real impact starts with asking the right question

How do we become more relevant to the people we serve?

Walmart didn’t lead with technology they led with empathy. Customers were unsure about their purchases, overwhelmed by options, and lacked real-time guidance. So Walmart focused its GenAI strategy on solving that. They used LLMs to make product discovery smarter, recommendations more personal, and Q&A actually useful. Suddenly, shoppers could find exactly what they needed and feel confident about their choices.

That shift from just selling to serving with relevance turned GenAI from a tool into a true differentiator. Because the most powerful AI? It’s built on understanding human problems.

From Discovery to Loyalty: A Smarter Retail Customer Journey with GenAI

Let’s look at the D2C customer journey through the customer’s lens, what they want at each stage, where friction shows up, and how GenAI in Retail can remove it.

With Retlocx, powered by Shellkode, you’re not just streamlining operations, you’re creating experiences that feel thoughtful, seamless, and tailored at every step.

1. Discovery That Understands What Customers Want

Customers don’t want to hunt, they want to find. When they search, they expect:

• Results that are relevant and fast
• Product pages in their language
• Descriptions tailored to their needs

Capabilities

Attribute Generation - starts with being findable. If customers can’t filter, search, or sort properly, they’re gone. Attribute Generation uses Claude 3.5 to tag products with relevant details like material, fit, and use case. It updates in real-time and keeps your catalog structured, searchable, and optimized for SEO.

• Result: +80% improvement in discoverability for context-based search

Translation for Product Description and Content - If your pages speak one language, you’re leaving revenue on the table. Translation engine rewrites your product content in multiple Indian and global languages, making sure your brand speaks to every customer with cultural accuracy.

• Result: +30% customer satisfaction

2. Check out That Feels Like a Conversation

Customers want speed, clarity, and confidence when they buy. They expect AI agents in retail that:

• Guide users step-by-step
• Offer personalized upsells
• Provide real-time answers

Capabilities

Smart Checkout Assistant — It enhances the final steps of the buying journey by dynamically surfacing personalized offers, alternatives based on customer behavior, cart content, and real-time trends. It identifies hesitation signals (like color or cart idle time) and responds with the right nudges such as better-matched products, timely discounts, or contextual upsells to prevent drop-offs and boost order value. By integrating browsing history, purchase patterns, and local demand signals, it delivers a checkout experience that feels tailored, not transactional.

Result: 15–30% increase in Average Order Value (AOV) and reduced cart abandonment

3. AI-Powered Customer Support That Listens and Learns

Frustrated customers want to feel heard. They need:

• Fast answers
• A human tone
• No repetition or handoffs

Capabilities

Agent Assistant for Retail & E-commerce — The Agent Assistant for Retail and E-commerce enhances customer experiences with GenAI-powered real-time query resolution, order tracking, and automated payments. It recommends personalized products based on browsing and purchase history, driving conversions and customer satisfaction. This solution streamlines operations while improving engagement and revenue.

• Result: Upto 50% automation in customer query handling using AI Agents

Post-Call Analytics - Support teams can’t fix what they can’t see. This solution automatically analyzes and summarizes every call, highlights patterns, and feeds back insights to improve agent training and workflows.

• Result: Fewer repeat tickets, faster resolutions

4. Predictive Retail Operations with GenAI

Customers don’t think about inventory. They just want their size in stock, their order delivered, and their experience uninterrupted. Retail leaders like Zara and Decathlon use data to anticipate demand and act quickly. This is where GenAI helps retailers stay one step ahead, forecasting demand, optimizing stock, and ensuring the experience runs like clockwork.

Capabilities

Inventory Optimization - Using GenAI and advanced data analytics the solution transforms inventory management by analyzing real-time and historical sales data, seasonal trends, and supply chain metrics. It provides real-time visibility into inventory, automates stock-level recommendations, and balances stock between warehouses and stores to ensure products are always available where needed and minimizes both overstocks and stockouts.

• Result: 20% lower holding costs, 40% fewer stockouts

Footfall Analytics - What if your stores could tell you when they’re busy, where people walk, and what they ignore? This solution does just that. It tracks footfall, dwell time, and zones to help you plan layouts, staff, and promos smarter.

• Result: 80% improvement in decisions, 25% boost in sales

Retlocx: Reimagine Retail and Logistics CX with GenAI

Retlocx leverages domain-specific GenAI agents to improve customer experience across retail and logistics functions.

• Integrates with existing systems
• Easy to make using low-code tools
• Continuously trains on enterprise data
• Offers pre-built GenAI agents via the Retlocx marketplace

Whether you’re scaling customer support, automating fulfillment, or enhancing store decisions — Retlocx delivers smart, self-optimizing AI experiences. Learn More Here

Conclusion

Generative AI is not just a technological shift it’s unlocking entirely new ways for retailers to operate, engage, and grow. From hyper-personalized customer journeys to autonomous AI agents handling support, merchandising, and marketing, the possibilities are expanding rapidly.
For retailers, adopting GenAI early in the journey can serve as a powerful differentiator, helping you evolve into a modern, adaptive, and customer-centric business. That said, this is still an emerging space. The technology is evolving fast, and success will depend on your ability to stay agile, experiment thoughtfully, and build systems that are frictionless to adapt and scale as the landscape matures.

GenAI in Retail: Transforming the Customer Journey from Checkout to Agents was originally published in ShellKode Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

When it comes to cloud deployments, it’s not just about technology, it’s about making strategic choices that drive real business impact. Selecting the right deployment platform is one such critical aspect that is paramount to ensuring long-term success. While AWS offers powerful container orchestration tools like EKS and ECS, AWS App Runner stands out as a game changer, particularly for use cases like containerized web apps.

In fact, its benefits far outweigh those of Amazon EKS and Amazon ECS. For instance, choosing AWS App Runner enabled us to accelerate deployment by up to 65%, and cut infrastructure costs by up to 40%, for a critical synchronization application migration- all while ensuring scalability and reliability. And this is just the tip of the iceberg — there’s much more to explore. Dive into this blog to learn more about how AWS App Runner can make a real difference for your cloud deployment strategy.

What is AWS App Runner?

AWS App Runner is a fully managed service that deploys containerized applications in under ten minutes, with zero infrastructure management. It automates scaling, load balancing, and monitoring, freeing developers to focus on innovation rather than operations.

With built-in CI/ CD integration, App Runner simplifies deployment workflows by automatically pulling the latest code changes from repositories like GitHub and AWS CodeCommit. It also provides end-to-end security, including HTTPS encryption and automatic certificate management, ensuring secure application deployment. And the best part? It scales on demand, adjusting resources based on traffic so businesses only pay for what they use- maximizing efficiency while keeping costs in check.

Why Choose App Runner over EKS and ECS?

Selecting the right deployment platform is a critical decision that impacts scalability, cost, and operational efficiency. While Kubernetes-based solutions like EKS and ECS offer robust container orchestration, they often come with added complexity, requiring significant setup and ongoing management.

For small-scale deployments, EKS and ECS can be unnecessarily complex. These services are built for managing large-scale, multi-container environments, requiring extensive setup, ongoing maintenance, and operational overhead. By leveraging App Runner, we streamlined the process- eliminating 3–5 days of setup time typically spent configuring clusters, nodes, and load balancers. Our focus was on delivering a seamless, cost-effective, and scalable solution without the burden of unnecessary complexity.

Benefits of App Runner

By choosing AWS App Runner, we optimized deployment speed, reduced costs, and enhanced scalability, delivering tangible business benefits. Let’s take a closer look at how it made a difference.

Simplified Deployment with Minimal Overhead

Traditional container orchestration platforms require configuring clusters, managing nodes, and setting up complex orchestration tools- all of which add operational overhead. With App Runner, deployment is simplified to just a few clicks, considerably reducing the set up time from days to minutes. This also has been proved to reduce operational overhead by up to 50%, allowing teams to focus on innovation rather than infrastructure management.

Optimized Costs with a Pay-as-You-Go Model

Managing infrastructure costs efficiently is crucial for any business, especially in cost-sensitive industries like FinTech. App Runner’s pay-as-you-go pricing eliminates upfront investment and significantly reduces operational expenses. In this case, compared to EKS, our customer saw by up to 40% decrease in monthly infrastructure spending, as they only paid for actual usage-without the overhead of maintaining idle resources.

Seamless Auto-Scaling for Peak Performance

In a high-demand environment like FinTech, performance consistency is non-negotiable. App Runner’s traffic-driven auto scaling dynamically adjusts resources based on real-time demand, ensuring smooth operation even during peak loads. This automated scaling reduced latency by up to 25%, maintaining seamless user experiences without manual intervention. With 99.9% uptime, the system remained highly reliable, a critical factor for real-time transactions where even a slight delay could impact business operations and customer trust.

Now that we’ve explored AWS App Runner, let us take a look at how to set it up.

Step-by-Step Guide to Deploying Applications with AWS App Runner

As established, deploying applications with AWS App Runner ensures seamless scaling and automation. Here, we walk you through the key steps to get AWS App Runner up and running.

• Click on create app runner

• Choose the appropriate repository type
  - If your application images are in ECR you can opt for ECR
  - If you’re using other 3rd party repositories like Git, Bitbucket you can choose Source Code Repository
• Automatic deployment mode will detect the latest version of images being pushed at ECR and will deploy it on the App Runner on behalf of you using the ECR access role

• Assign the right amount of resources to the container. (i.e: this is for single container)

• Following that create a new Auto Scaling Configuration for your container to handle the workload with minimum and maximum capacity

• Assign the right role for your container if they need to communicate with any other AWS services
• In my sample case I have added a role that can read and write from secrets manager

• For networking, we can choose the outgoing network to be either to Public or via VPC
• We need to create a VPC connector on our custom VPC with only a private subnet to route the traffic
• We can leverage this VPC connector if the container needs to communicate with any compute services running inside VPC like EC2 or RDS

• Now validate the configurations on the final page then click on create and deploy

• App Runner is created and running
• On the main dashboard we can see the App Runner logs, Deployment logs and Application Logs

Potential Challenges and Pitfalls

As we saw, AWS App Runner is a managed service that simplifies the deployment of containerized applications, allowing developers to focus on their code rather than infrastructure management. However, while it offers ease of use and automation, it may not be the ideal solution for every workload. Here are some limitations to consider:

Single -container Focus

App Runner is designed for single-container applications, making it less suitable for complex deployments that require multi-container orchestration. If your application depends on sidecars, catching layers, or service mesh configurations, solutions like Amazon ECS or EKS provide greater flexibility.

Limited Customization

The abstraction of infrastructure management makes App Runner easy to use, but it also limits control over networking, security policies, and runtime configurations. For businesses that require fine-grained tuning of their cloud environment, ECS or EKS may be a better fit.

Scaling Constraints

While App Runner provides automatic scaling, it has predefined vertical and horizontal scaling limits. This may not be sufficient for high-demand applications that require precise scaling strategies or need to handle unpredictable traffic spikes.

Limited Runtime Support

Currently, App Runner does not support ARM-based runtimes, restricting its usability for workloads optimized for ARM architectures. If your application relies on ARM for cost or performance benefits, you may need to consider alternative AWS services.

By understanding these constraints, businesses can make an informed decision about whether AWS App Runner aligns with their operational needs and long-term scalability goals.

Conclusion

In the digital transformation journey, selecting the right cloud platform is critical. AWS App Runner offers a streamlined solution for deploying and managing containerized applications, automating tasks like infrastructure management and scaling. Designed for single-container applications, it reduces operational complexity, accelerates time-to-market, and minimizes overhead.

By eliminating the need for manual intervention, App Runner allows businesses to focus on innovation and customer value. Its simplicity and efficiency enable organizations to adapt quickly to changing market demands, ensuring long-term agility and competitiveness. For companies seeking a faster, smarter approach to cloud deployments, AWS App Runner is a game-changer.

Author: Nirmal Prabhu, Head Cloud Practice

Why AWS App Runner Outshines Amazon EKS/ECS for Containerized Web App Deployments was originally published in ShellKode Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

AI agents are quickly replacing traditional automation solutions and chatbots. It’s safe to say that they are the next big thing in the tech landscape that organizations cannot afford to overlook. They bring a level of autonomy and operational efficiency that revolutionize the way businesses operate. While agents demonstrate remarkable capabilities in handling complex tasks, their response times in customer-facing applications can present challenges, leading to frustration and a less engaging experience for users.

A solution to this challenge is streaming responses, which allows partial results to be displayed as soon as they are available. In this blog, we’ll demonstrate how to implement a streaming solution using AWS Bedrock Agents, Boto3 API, Flask, and ReactJS. We’ll walk you through creating a Flask API to stream responses from Bedrock Agents, building a ReactJS frontend to handle the streaming API, and combining these components into a seamless real-time user experience.

Addressing Response Latency with Real-time Streaming

When invoking Bedrock Agents, one significant challenge is the response generation time, which can take up to 15–20 seconds depending on the complexity of the request. This latency can create a perception of a slow application and diminish user engagement, as waiting for the complete response delays interaction and reduces communication efficiency. To address this challenge, we implement a streaming solution that delivers responses incrementally to the frontend, allowing users to see partial outputs in real-time, enhancing responsiveness and usability. Let us see how to implement this solution.

Prerequisites:

Before we dive into the implementation, make sure you have:

• Python 3.10 or higher installed
• Node.js and npm installed
• Any IDE as per your need
• AWS credentials configured on your machine
• Basic understanding of Flask and React

Backend: Flask API with Streaming

First, let’s set up our Python environment and install the necessary packages.

• Create a virtual environment and activate it:

python3 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate

• Install the required Python packages:

pip install flask boto3 #And other packages to run your application

• The backend will use Flask to set up an API endpoint. It streams the agent’s response using AWS Bedrock’s invoke_agent method. Here’s a simplified implementation:

from flask import Flask, Response, request
import boto3
import os
import json

app = Flask(__name__)

# Initialize Bedrock client
bedrock_client = boto3.client('bedrock-runtime', region_name='us-east-1')

@app.route('/api/stream-agent-response', methods=['POST'])
def stream_agent_response():
    user_input = request.json.get('input')
    session_id = request.json.get('sessionId')
    agent_id = os.getenv('AGENT_ID')

    def generate():
        try:
            response = bedrock_client.invoke_agent(
                agentAliasId=os.getenv('ALIAS_ID'),
                agentId=agent_id,
                enableTrace=False,
                endSession=False,
                inputText=user_input,
                sessionId=session_id,
                streamingConfigurations={'streamFinalResponse': True}
            )

if response.get('completion'):
   for event_chunk in response['completion']:
       if 'chunk' in event_chunk and 'bytes' in event_chunk['chunk']:
           chunk_text = event_chunk['chunk']['bytes'].decode('utf-8')

           yield f"data: {json.dumps({'response': chunk_text, 
                                      'sessionId': session_id})}\n\n"

        except Exception as e:
            error_response = json.dumps({'error': str(e)})
            yield f"data: {error_response}\n\n"

    return Response(generate(), mimetype='text/event-stream')


if __name__ == '__main__':
    app.run(debug=True)

Key Points

1. Generator Function: Streams chunks of data as they are received from Bedrock
2. Error Handling: Captures exceptions and streams error messages to the client
3. Session Management: Supports unique sessions for different users or conversations

Frontend: ReactJS with Streaming API

The frontend will consume the streaming API using fetch and display the incremental responses in real time. Here’s an implementation snippet:

const handleSendMessage = async (query, languageCode) => 


 const request = {
     input: query,
     sessionId: sessionId,
 };

  try {
     const response = await  
     fetch(`${backendURL}api/stream-agent-response`, {
         method: "POST", body: JSON.stringify(request),});

     const reader = response.body.getReader();
     const decoder = new TextDecoder();
     let ongoingBotResponse = "";

     while (true) {
         const { done, value } = await reader.read();
         if (done) break;


         const chunk = decoder.decode(value, { stream: true });
         const messages = chunk.split("\n\n");


         for (const message of messages) {
             if (message.trim().startsWith("data: ")) {
                 try {
                     const jsonStr = message.trim().slice(6);
                     const data = JSON.parse(jsonStr);


                     if (data.response) {
                         ongoingBotResponse += data.response;
                         setSessionConversation((prev) => [
                             ...prev,
                     { type: "AGENT", body: ongoingBotResponse },]);
                     }
                 } catch (e) {
                     console.warn("Error parsing chunk:", e);
                 }
             }
         }
     }
 } catch (error) {
     console.error("Streaming error:", error);
 }
};

Key Points

1. Streaming Responses: Processes chunks of data in real-time
2. Error Handling: Logs errors and avoids breaking the UI
3. State Management: Updates the conversation dynamically

End-to-End Workflow

Here’s how the components work together:

1. User Interaction: The user submits a query via the ReactJS frontend
2. API Call: The frontend sends the query to the Flask backend
3. Streaming Response: Flask streams partial responses from Bedrock to the frontend
4. Real-Time Updates: ReactJS updates the UI with each chunk of data

Conclusion

Given their indispensability, agents are here to stay in customer-facing applications, not only for their conversational abilities but also for their adaptability, automation, and contextual understanding.

Minimizing response delays is crucial for delivering a frictionless user experience. As explored in this blog, streaming agent responses can significantly enhance engagement by reducing perceived latency from 15–20 seconds to just 5–6 seconds. This makes applications feel faster, more responsive, and intuitive.

By leveraging the power of AWS Bedrock, Flask, and React, you can build highly efficient, real-time AI-driven applications that provide seamless, intelligent, and instant assistance — ensuring users stay engaged and satisfied.

Author

Sai Chandan — www.linkedin.com/in/sai-chandan

Contributor

Bakrudeen — https://www.linkedin.com/in/bakrudeen-k-6790219b/

Streaming Agent Responses using Bedrock Agent API was originally published in ShellKode Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The AI landscape has long been dominated by models that require enormous financial and computational resources, making them accessible only to tech giants. But the rise of DeepSeek is shifting the narrative. By dramatically reducing the cost of training high-performing models, DeepSeek has triggered a wave of innovation, enabling more entrepreneurs, developers, and researchers to explore AI’s potential without breaking the bank. This democratization of LLMs is fueling a new era of AI adoption, sparking fresh competition, and accelerating advancements in generative AI across industries.

The Rise of DeepSeek-R1

DeepSeek-R1 has rapidly emerged as a game-changer in the AI space. Unlike its costly, proprietary counterparts, DeepSeek-R1 is an open-source marvel, making advanced AI capabilities accessible to a much wider audience. The model was trained using just 2,048 Nvidia H800 GPUs over two months, at a remarkably low cost of $5.6 million. To put this in perspective, many of the top models in the industry require training budgets that reach well over $100 million. By adopting model distillation, chain-of-thought prompting, and reinforcement learning, DeepSeek has effectively slashed the cost barrier, proving that high-performance AI doesn’t have to come with an astronomical price tag.

Breaking Down the DeepSeek Models

To bring you this analysis, we went deep into our own pockets (seriously, our wallets are crying) to benchmark these models across key tasks: reasoning, coding, creative writing, and overall cost. Here’s what we found:

• 1 — Excellent
• 2 — Satisfactory
• 3 — Could Be Better

So, after burning a hole in our budget, here’s what we learned: If you’re looking for a model that delivers excellent reasoning, coding, and creative writing, the 651B variant is king — but it comes at a premium price. On the other hand, if cost-efficiency is your top priority, the Qwen-1.5B version is an absolute steal, though it struggles with more complex tasks. The LLama 8B variant strikes a middle ground, making it a solid choice for balanced workloads.

Choosing the Right Model for Your Needs

• DeepSeekR1-Distill-Qwen-1.5B 🏗️ Best for: Low-cost applications, basic chatbot interactions, and lightweight text analysis. Why? Cheap to run but struggles with complex reasoning or creative writing.
• DeepSeekR1-Distill-LLama 8B ⚖️ Best for: Mid-tier AI applications, general-purpose coding, and reasonable creative tasks. Why? Balances cost and performance without breaking the bank.
• DeepSeekR1–651B 🚀 Best for: High-end AI applications, enterprise-level reasoning, and top-tier generative tasks. Why? Unmatched power, but at a hefty price.

Final Thoughts

DeepSeek isn’t just another AI model — it’s a movement towards making cutting-edge AI more accessible. Whether you’re a solo entrepreneur experimenting with LLMs or a company looking to integrate AI into critical workflows, there’s a DeepSeek model that fits your needs. And as long as they keep pushing boundaries (and helping us keep a few extra bucks in our pockets), we’re here for it!

Author

Bakrudeen — https://www.linkedin.com/in/bakrudeen-k-6790219b/

DeepSeek R1 Benchmark & Comparison Evaluating Performance & Cost Efficiency was originally published in ShellKode Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Author: Manasa K, Head Data Practice | Co-Author: Sruthi

In today’s data-driven world, businesses face a critical challenge in extracting meaningful insights from unstructured data such as documents, emails, and reports, which often complement structured datasets. While traditional BI tools excel at analyzing structured data, they fall short when it comes to incorporating unstructured data sources, limiting the depth of analysis and decision-making. AWS is addressing this challenge by integrating Amazon Q Business with QuickSight, leveraging Generative AI to enable businesses to analyze both structured and unstructured data across various formats, unlocking a more comprehensive and insightful view of their data.

In this second blog of the series Conversational AI and Semantic Insights, a New Era in BI, we will explore Data Stories and Amazon Q Business Integration with QuickSight. Discover how this latest advancement helps in creating a unified experience for advanced decision-making.

Amazon Q Business: Bridging the Gap Between Structured and Unstructured Data

For a long time, the focus has been on extracting valuable insights from structured data, while equally valuable, if not more, insights from unstructured data remained untapped. Unstructured data, combined with structured data can be a treasure trove of information for organisations. So naturally, integrating both ensures a complete view for decision-making. Amazon Q Business integration with Quicksight effectively bridges the gap by seamlessly combining insights from structured data, such as databases and spreadsheets, with unstructured data from pdf and so on.

Lets say a business analyzing internal sales data and customer feedback also wants to understand external factors like market trends or competitor actions, with much of this data taking the unstructured format like PDFs. With Q Business in the picture, the business can easily extract and analyze insights from both structured and unstructured data, enabling a more comprehensive understanding of the factors influencing its performance and helping to drive more informed, strategic decisions.

Crafting Compelling Narratives with Data Stories

The amount of data you have does not directly determine its value but it’s how effectively you leverage it that influences your potential for growth. While Amazon QuickSight helps turn data into clear, actionable insights through simple and interactive visuals, sometimes narrative-driven visualizations can add a touch of cohesive storytelling. This is exactly what Amazon QuickSight’s ‘Stories’ feature helps with, enabling you to present your findings in an engaging and easy-to-understand format, great for explaining data to non-technical audiences.

Now let us see how data story works.

How Data Story Works

1. Go to the Data Stories section and click New Data Story

2. Enter a Prompt

In the data story builder, you can provide a prompt that specifies the focus of the story. For instance, describe the data insights you want to explore, we can prompt it to generate a data story around revenue growth analysis, customer segmentation, and so on by selecting relevant visuals based on the narrative or executive summary you want to generate.

3. Add Visuals, choose the dashboard containing the visuals you need

4. Select the visuals to include (up to 20)

How to use Q Business with QuickSight?

As you can see Data stories brings a new way of storytelling with compelling narratives introducing dynamic and interactive elements that make data exploration seamless. However, Data Stories is limited to the data available in QuickSight, and this is where integration with Q Business plays a key role in bringing your external data, specifically unstructured data, for a more holistic storytelling.

To get started with Amazon Q Business in QuickSight, the first step is to set up an application within Q Business for QuickSight, which will allow you to integrate and leverage To get started with Amazon Q Business in QuickSight, the first step is to set up an application within Q Business for QuickSight, which will allow you to integrate and leverage unstructured data alongside your structured datasets for more powerful insights and analytics.

5. Go to Amazon Q Business in AWS console and create an application

6. Upload all the documents to the Q Business application that are required or connect to the data source of your choice

7. You will see the uploaded files as below. For example the uploaded file here has employee feedback data like feedback from the managers, peers and self feedback

We can integrate this Q Business Application in QuickSight as below.

8. Navigate to the Manage QuickSight option in the QuickSight console

9. In Security & Permissions section, click on Manage option

10. Choose Amazon Q Business and select the application created

Now the Q Business Application is integrated with QuickSight.

Amazon Q Business integration can be leveraged in topics and data stories in QuickSight. We will talk about both the integrations below.

Q Business Integration with Data Stories

By integrating structured and unstructured data into Data stories, users gain access to more comprehensive insights, ensuring key context isn’t missed. This capability allows organizations to create richer, more impactful data stories. Enable Use insights from Amazon Q Business option so now a narrative will be created along with the unstructured data in Q business

1. Click Build to generate your story. You can review and either keep, edit, or discard the draft

2. This part of the summary is from the selected visuals and the prompt added

3. This part of the summary generated in data story is from Q Business integration

Q Business Integration with QuickSight Q Topics

To generate detailed insights through PDF documents along with QuickSight Q, we need to enable Q business within the topic settings to unlock enhanced insights.

In our scenario, we are going to analyze Employee Performance where we have employee ratings, projects, KPIs, etc as in the datasets and we have feedback from the managers, peers, and self feedback from an unstructured text file in Q business, now let’s see how the integrations work.

Below, we asked a query about top performing employee and following is the response generated through QuickSight Q.

Along with insights from text documents, it also provides information on what source files it has utilised to generate the insights.

Potential Technical and Business Pitfalls

While the integration of Amazon Q Business with QuickSight is all about offering powerful capabilities, it is imperative to consider the following points when adopting this solution when it comes to ensuring efficient data processing.

Larger the Dataset, longer the processing time

The process of generating outputs can be slow, as the system often requires considerable time to process and aggregate large datasets, particularly when complex queries or multiple data sources are involved. This is common in systems that combine machine learning-driven insights with large-scale data analytics​.

Accuracy issues, due to overlapping Data Sources

When two documents provide overlapping data, Amazon Q Business may face certain challenges in selecting the correct one, leading to random data source selection. This happens because the model might not properly resolve data conflicts between sources. Although QuickSight identifies the data source in the output, it still presents potential issues when multiple sources have similar or identical content​. This may occur when the metadata and contextual rules around data are not robust enough to enforce consistency in selection.

Conclusion

As we saw in this blog, the integration of Amazon Q Business with QuickSight enables organisations to seamlessly analyze and extract insights from both structured and unstructured data types, paving the way for a more holistic approach. Further, the introduction of advanced features like data stories marks yet another significant stride in the business intelligence landscape by AWS.

Evolving from NLP-powered BI to more advanced, AI-driven insights, QuickSight Q has set the stage for Generative BI. Want to learn more about QuickSight Q and how you can enhance and tune the output from it, and eventually drive better decision-making? Check out our previous blog Mastering QuickSight Q: Setting the Stage for Semantic Insights by Manasa K and Priyanka Gopalakrishnan.

Embracing GenBI with Data Stories and Amazon Q Business Integration with QuickSight was originally published in ShellKode Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Author: Manasa K, Head Data Practice | Co-Author: Priyanka Gopalakrishnan

Without question, companies leveraging data insights thrive with improved operational efficiency and profitability. As such, the increasing need for enterprises to effectively leverage data, coupled with paradigm shifts in technology is reshaping the business intelligence landscape more so in recent times. This evolution is marked by a move toward a semantic understanding of data and intelligence, enabling deeper insights and more informed decision-making.

For instance, the emergence of Generative AI has revolutionized business intelligence tools, offering capabilities far beyond traditional interactive dashboards. These tools now enable processing complex natural language queries, allowing users to “chat” with their data to uncover insights effortlessly. AWS is at the forefront of this transformation with Amazon QuickSight, introducing innovative features like Amazon QuickSight in Q, Data Stories, Data Scenarios, and integration with Amazon Q Business. These advancements empower users to engage with their data more intuitively.

As we move towards a new approach in data analytics, this marks the beginning of a transformation in how businesses interact with their data. While we will be delving into all things new about QuickSight in the upcoming blogs, in this first edition of a three-part blog series, we will explore more about QuickSight in Q and the steps to enhance and tune the output from QuickSight Q.

What is Amazon Q in QuickSight?

Amazon QuickSight is a powerful, cloud-native BI service offered by AWS. It allows businesses to create interactive dashboards and visualize data from multiple sources. The latest version, Amazon Q in QuickSight, integrates generative AI capabilities to significantly enhance the user experience by enabling natural language queries. This integration transforms how users interact with their data, making it easier to access, analyze, and derive insights without needing advanced technical skills. With Amazon Q in Quicksight, businesses can now explore their data through natural language, bringing a new level of accessibility and efficiency to business intelligence.

Getting Started with QuickSight Q

Now that we’ve introduced the transformative potential of Amazon Q in QuickSight, let’s dive deeper into how you can start leveraging this tool to maximize your data analysis capabilities. One of the fundamental concepts in QuickSight Q is topics, which is the backbone of how the system understands and processes natural language queries. It includes metadata, user-friendly field names, synonyms, and filters to make the data more intuitive and relevant for specific business use cases.

Creating and managing these topics effectively is crucial to making the most out of QuickSight Q and having a better user experience. A topic can be mapped to one dataset or multiple datasets with relevant data, so business users can get relevant insights.

• When working with multiple datasets in QuickSight Q, you might typically create separate topics for each dataset and switch between topics every time you need to ask a question specific to a dataset.
• However, this can be time-consuming and inconvenient. Instead, you can simplify this process by creating a single topic that includes all the datasets that are relevant and needed.
• Add multiple datasets by clicking on “ADD DATASETS“ in the DATASETS section.

• With this functionality, we can seamlessly ask queries without switching topics, allowing for more efficient analysis across datasets.

Tailoring QuickSight Q to Fit your Data and Business needs

Data holds greater value when it is not just available, but accurate, accessible, and actionable. Tailoring Amazon Q in QuickSight to suit your business needs plays a key role in unlocking its value by delivering the most relevant insights — like a genie by your side, giving the right data at the right time, just the way you want to see it.

Now let us explore how to finetune QuickSight Q effectively.

The Role of Context in Simplifying Data Retrieval and Analysis

Sifting through data without any context can be inefficient and overwhelming. Context, however, provides clarity by offering additional information about the data, like its nature and the type of information it contains. This organized approach enables QuickSight Q to gain a deeper understanding of your data, allowing it to better interpret and respond to queries with accuracy.

Synonyms

Adding synonyms for columns makes the data more accessible and lets users make queries using alternative words or phrases.

For instance, the column Employee ID could have synonyms like ID or Code, allowing users to ask questions in multiple ways and still get accurate results. This flexibility ensures a smoother and more intuitive user experience.

Semantics

Assigning semantic types to fields is one of the key ways to improve the quality of answers in QuickSight Q. Doing so helps QuickSight Q better understand how to interpret and utilize your data effectively, ensuring more accurate and contextually relevant responses.

Semantic types define critical aspects of a field, including field role, data type, default aggregation and additional context. By regularly updating and refining the semantic types for your fields, you empower QuickSight Q to deliver precise, high-quality answers, creating a smoother and more intuitive user experience.

Description

We can enhance QuickSight Q’s ability to understand and retrieve accurate data by providing a description of each field. Serving as a bridge between the dataset and user queries, these descriptions, without doubt, improve both the accuracy of results and the overall user experience.

Enhancing Q’s Understanding with Business Terms

Business-specific terms can sometimes, if not always, be confusing, or complex to interpret if not properly defined. Defining these terms allows QuickSight Q to better interpret queries and provide results tailored to your business context. It effectively bridges the gap between technical data and business terms, improving the precision of your reports and making the analysis more intuitive.

Named Entity

Group relevant columns together using named entities in QuickSight Q for better query performance and accuracy. This approach simplifies querying by providing a structured way to reference and organize related data fields, ensuring more efficient and precise results.

Without adding a named entity:

Adding named entity:

For example, under a Product Named Entity, you can configure the following columns:

• Category
• Discount
• Price
• Rating

• By leveraging Named Entities, whenever you ask a question related to Category, QuickSight Q will directly pull the configured columns instead of searching randomly through all 100 columns.

Improving the Reliability of Insights Generated by QuickSight Q

Sure, you can simplify data retrieval by adding context and enable Q to better understand your queries using named entities to get the results you want. But how can you ensure optimised query performance, allowing QuickSight Q to refine how it processes similar questions in the future and ensuring the reliability of responses? Well, there are not one, but two ways to do that too.

Feedback

• QuickSight Q allows users to provide feedback so it can learn and improve over time, refining how it processes similar questions in the future. If the results do not meet the expectations, users can send feedback with suggestions on how the system could better match the query intent.
• Positive feedback reinforces the system’s understanding of user preferences and ensures consistent performance across similar queries.
• By actively providing feedback — whether for improvements or validations — you contribute to optimising query responses, making the tool more effective and reliable for all users.

Mark as Verified

As you go about making queries and getting accurate results, also make sure to verify answers to the questions asked. Verified answers are prioritized in search results, helping users quickly find reliable information. Additionally, the Mark as verified option also plays a key role in driving constant improvements by helping users track how often questions are asked, and if users find the answers helpful.

Conclusion

As outlined in this blog, following these best practices optimizes the output from QuickSight Q, and enhances the value of your data, driving better decision-making and, ultimately, better business outcomes. QuickSight Q continues to evolve, becoming smarter and all the more powerful, not only addressing the limitations of conventional BI tools but also paving the way for a deeper understanding of data.

To continue your journey toward a deeper understanding of your data, stay tuned for the upcoming blogs, where we will explore the latest advancements in QuickSight Q, including Data stories and Data Scenarios and how they can further elevate your business intelligence capabilities.

Mastering QuickSight Q: Setting the Stage for Semantic Insights was originally published in ShellKode Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Amazon OpenSearch a managed service for OpenSearch provides fully managed and scalable search services. It also provides a serverless variant to run the workloads on any scale. OpenSearch pipelines provide an out-of-the-box Zero ETL to sync data from multiple data sources. Zero ETL is not a new technology, AWS already introduced Aurora to RedShift replication using Zero ETL and it became a huge success. Then they introduced many Zero ETL integrations with various data services. Here we are going to set up a Zero ETL pipeline from Amazon DynamoDB to Amazon OpenSearch service using OpenSearch Pipeline and use CloudFormation template to deploy this. Even though it's a Zero ETL, we need to set up a pipeline configuration in JSON/YAML format. Behind the scenes, AWS using OpenSearch Data Prepper to integrate the DynamoDB data into OpenSearch.

OpenSearch Serverless Concept:

• Collection — An OpenSearch Serverless collection is a group of OpenSearch indexes.
• Data Access Policy — Fine-grained access control is at the collection and index levels.
• Network Access Policy — Control the traffic internally (VPC via endpoints) and externally (Public).
• Pipeline — Data ingestion pipeline from various sources.

Create an OpenSearch collection:

This is very straightforward, if we do it via console we can give the collection name and optional fields for network policy and data access policy.

We’ll not cover the basic stuff like creating vpc, etc, it just creates one Security group. VPC and KMS details have to be provided while launching the template.

But it will create a new VPC endpoint on point, make sure you don't have any existing endpoints, because for 1 VPC only one VPC endpoint is allowed.

AWSTemplateFormatVersion: '2010-09-09'
Description: 'CloudFormation template for OpenSearch Serverless Collection'

Parameters:
  VpcId:
    Type: AWS::EC2::VPC::Id
    Description: VPC ID where the OpenSearch collection will be deployed

  VpcCidr:
    Type: String
    Description: CIDR block of the VPC

  PrivateSubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
    Description: List of private subnet IDs for the VPC endpoint

  KmsKeyId:
    Type: String
    Description: KMS Key ID for encryption

  CollectionDescription:
    Type: String
    Description: Description of the OpenSearch collection

  CollectionName:
    Type: String
    Description: Name of the OpenSearch collection

  EtlRoleArn:
    Type: String
    Description: ARN of the ETL role

Resources:
  OpenSearchSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Security group for OpenSearch VPC endpoint
      VpcId: !Ref VpcId
      SecurityGroupIngress:
        - IpProtocol: https
          FromPort: 443
          ToPort: 443
          CidrIp: !Ref VpcCidr
      SecurityGroupEgress:
        - IpProtocol: -1
          FromPort: -1
          ToPort: -1
          CidrIp: 0.0.0.0/0

  OpenSearchVPCEndpoint:
    Type: AWS::OpenSearchServerless::VpcEndpoint
    Properties:
      Name: !Sub '${CollectionName}-vpcendpoint-os'
      VpcId: !Ref VpcId
      SubnetIds: !Ref PrivateSubnetIds
      SecurityGroupIds: 
        - !Ref OpenSearchSecurityGroup

  OpenSearchEncryptionPolicy:
    Type: AWS::OpenSearchServerless::SecurityPolicy
    Properties:
      Name: !Sub '${CollectionName}-encryption-policy-os'
      Type: encryption
      Description: Encryption policy for OpenSearch collection
      Policy: !Sub |
        {
          "Rules":[
            {
              "ResourceType":"collection",
              "Resource":[
                "collection/${CollectionName}"
              ]
            }
          ],
          "AWSOwnedKey":false,
          "KmsARN":"${KmsKeyId}"
        }

  OpenSearchServerlessCollection:
    Type: AWS::OpenSearchServerless::Collection
    DependsOn: OpenSearchEncryptionPolicy
    Properties:
      Name: !Ref CollectionName
      Description: !Ref CollectionDescription
      Type: SEARCH
      StandbyReplicas: ENABLED

  OpenSearchServerlessAccessPolicy:
    Type: AWS::OpenSearchServerless::AccessPolicy
    Properties:
      Name: !Sub '${CollectionName}-access-policy'
      Type: data
      Description: Access policy for OpenSearch collection
      Policy: !Sub |
        [
          {
            "Rules":[
              {
                "Resource":[
                  "collection/${CollectionName}"
                ],
                "Permission":[
                  "aoss:CreateCollectionItems",
                  "aoss:DeleteCollectionItems",
                  "aoss:UpdateCollectionItems",
                  "aoss:DescribeCollectionItems"
                ],
                "ResourceType":"collection"
              },
              {
                "Resource":[
                  "index/${CollectionName}/*"
                ],
                "Permission":[
                  "aoss:CreateIndex",
                  "aoss:DeleteIndex",
                  "aoss:UpdateIndex",
                  "aoss:DescribeIndex",
                  "aoss:ReadDocument",
                  "aoss:WriteDocument"
                ],
                "ResourceType":"index"
              }
            ],
            "Principal":[
              "${EtlRoleArn}"
            ],
            "Description":"etl"
          }
        ]

  OpenSearchNetworkPolicy:
    Type: AWS::OpenSearchServerless::SecurityPolicy
    Properties:
      Name: !Sub '${CollectionName}-network-policy-os'
      Type: network
      Description: Network policy for OpenSearch collection
      Policy: !Sub |
        [
          {
            "Rules":[
              {
                "ResourceType":"collection",
                "Resource":[
                  "collection/${CollectionName}"
                ]
              },
              {
                "ResourceType":"dashboard",
                "Resource":[
                  "collection/${CollectionName}"
                ]
              }
            ],
            "AllowFromPublic":false,
            "SourceVPCEs":[
              "${OpenSearchVPCEndpoint}"
            ]
          }
        ]

Outputs:
  CollectionId:
    Description: ID of the created OpenSearch Serverless Collection
    Value: !Ref OpenSearchServerlessCollection

  VpcEndpointId:
    Description: ID of the created VPC Endpoint
    Value: !Ref OpenSearchVPCEndpoint

  SecurityGroupId:
    Description: ID of the created Security Group
    Value: !Ref OpenSearchSecurityGroup

Configure DynamoDB Table:

To support ZeroETL from DynamoDB to OpenSearch collection, we need to enable Point in Time recovery and Streams on the DynamoDB side.

• Point-in-time recovery will export the existing data from the DynamoDB to the S3 bucket.
• Streams will do the change data capture(CDC) and do the continuous replication to OpenSearch.

DynamoDB Streams only stores data in a log for up to 24 hours. If ingestion from an initial snapshot of a large table takes 24 hours or more, there will be some initial data loss. To mitigate this data loss, estimate the size of the table and configure appropriate compute units of OpenSearch Ingestion pipelines.

Go to DynamoDB, select your table, and navigate to Backups.

Under the point-in-time recovery, click on Edit and turn on the point-in-time recovery feature.

Now, go to the Exports and Streams options, then turn on the streams.

Create OpenSearch Pipeline from DynamoDB:

We created a CloudFormation template to provision the OpenSearch pipeline.

1. This will create an IAM Role for the pipeline that will do DynamoDB export to S3.
2. It required the KMS access to encrypt the export data files.
3. Create Network access policy and add the VPC endpoint

AWSTemplateFormatVersion: '2010-09-09'
Description: 'CloudFormation template for DynamoDB to OpenSearch pipeline (Zero ETL)'

Parameters:
  DynamoDBTableName:
    Type: String
    Description: 'Name of the DynamoDB table'

  S3BucketName:
    Type: String
    Description: 'Name of the S3 bucket'

  S3Prefix:
    Type: String
    Description: 'S3 prefix for DynamoDB export'

  OpenSearchIndex:
    Type: String
    Description: 'OpenSearch index name - LowerCase Only'

  DLQPrefix:
    Type: String
    Description: 'DLQ prefix for OpenSearch'

  OpenSearchCollectionId:
    Type: String
    Description: 'OpenSearch Collection ID'

  SubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
    Description: 'List of Subnet IDs for the pipeline'

  SecurityGroupId:
    Type: AWS::EC2::SecurityGroup::Id
    Description: 'Security Group ID for the pipeline'
  
  PipelineName:
    Type: String
    Description: 'Name of the Pipeline - lowercase, numbers and hypends'   

  ZeroETLRoleName:
    Type: String
    Description: 'Name of the role'   

  NetworkPolicyName:
    Type: String
    Description: 'Name of the NetworkPolicyName'   


Resources:
  ZeroETLRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: !Ref ZeroETLRoleName
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - osis-pipelines.amazonaws.com
            Action: 'sts:AssumeRole'
      Policies:
        - PolicyName: DynamoDBExportPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: allowRunExportJob
                Effect: Allow
                Action:
                  - 'dynamodb:DescribeTable'
                  - 'dynamodb:DescribeContinuousBackups'
                  - 'dynamodb:ExportTableToPointInTime'
                Resource:
                  - !Sub 'arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTableName}'
              - Sid: allowCheckExportjob
                Effect: Allow
                Action:
                  - 'dynamodb:DescribeExport'
                Resource:
                  - !Sub 'arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTableName}/export/*'
              - Sid: allowReadFromStream
                Effect: Allow
                Action:
                  - 'dynamodb:DescribeStream'
                  - 'dynamodb:GetRecords'
                  - 'dynamodb:GetShardIterator'
                Resource:
                  - !Sub 'arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTableName}/stream/*'
              - Sid: allowReadAndWriteToS3ForExport
                Effect: Allow
                Action:
                  - 's3:GetObject'
                  - 's3:AbortMultipartUpload'
                  - 's3:PutObject'
                  - 's3:PutObjectAcl'
                Resource:
                  - !Sub 'arn:aws:s3:::${S3BucketName}/*'
        - PolicyName: KMSAccessPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Sid: KMSAccess
                Effect: Allow
                Action:
                  - 'kms:Decrypt'
                  - 'kms:GenerateDataKey'
                Resource: !Sub 'arn:aws:kms:${AWS::Region}:${AWS::AccountId}:key/*'
        - PolicyName: AOSSIngestionPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement: 
              - Sid: "VisualEditor0"
                Effect: "Allow"
                Action: 
                  - "aoss:APIAccessAll"
                  - "aoss:BatchGetCollection"
                Resource: 
                  - !Sub 'arn:aws:aoss:${AWS::Region}:${AWS::AccountId}:collection/${OpenSearchCollectionId}'
              - Sid: "VisualEditor1"
                Effect: "Allow"
                Action: 
                  - "aoss:CreateSecurityPolicy"
                  - "aoss:UpdateSecurityPolicy"
                  - "aoss:GetSecurityPolicy"
                Resource: "*"
              - Sid: OSISIngest
                Effect: Allow
                Action: 'osis:Ingest'
                Resource: !Sub 'arn:aws:osis:${AWS::Region}:${AWS::AccountId}:pipeline/*'
        - PolicyName: CloudWatchLogsPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: !GetAtt ZeroETLPipelineLogGroup.Arn

  ZeroETLPipelineLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub '/aws/vendedlogs/OpenSearchIngestion/${AWS::StackName}-ZeroETLPipeline'
      RetentionInDays: 30

  ZeroETLPipeline:
    Type: 'AWS::OSIS::Pipeline'
    Properties:
      PipelineName: !Ref PipelineName
      MinUnits: 1
      MaxUnits: 1
      PipelineConfigurationBody: 
        Fn::Sub:
          - |
            version: '2'
            dynamodb-pipeline:
              source:
                dynamodb:
                  acknowledgments: true
                  tables:
                    - table_arn: arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTableName}
                      stream:
                        start_position: LATEST
                      export:
                        s3_bucket: ${S3BucketName}
                        s3_region: ${AWS::Region}
                        s3_prefix: ${S3Prefix}
                  aws:
                    sts_role_arn: ${ZeroETLRoleArn}
                    region: ${AWS::Region}
              sink:
                - opensearch:
                    hosts:
                      - ${OpenSearchHost}
                    index: '${OpenSearchIndex}'
                    index_type: management_disabled
                    document_id: ${DocumentId}
                    action: ${Action}
                    document_version: ${DocumentVersion}
                    document_version_type: external
                    aws:
                      sts_role_arn: ${ZeroETLRoleArn}
                      region: ${AWS::Region}
                      serverless: true
                      serverless_options:
                        network_policy_name: '${NetworkPolicyName}'
                    dlq:
                      s3:
                        bucket: ${S3BucketName}
                        key_path_prefix: ${DLQPrefix}
                        region: ${AWS::Region}
                        sts_role_arn: ${ZeroETLRoleArn}
          - ZeroETLRoleArn: !GetAtt ZeroETLRole.Arn
            OpenSearchHost: !Sub 'https://${OpenSearchCollectionId}.${AWS::Region}.aoss.amazonaws.com'
            DocumentId: "${getMetadata(\"primary_key\")}"
            Action: "${getMetadata(\"opensearch_action\")}"
            DocumentVersion: "${getMetadata(\"document_version\")}"
      VpcOptions: 
        SubnetIds: !Ref SubnetIds
        SecurityGroupIds: 
          - !Ref SecurityGroupId
      LogPublishingOptions:
        IsLoggingEnabled: true
        CloudWatchLogDestination:
          LogGroup: !Ref ZeroETLPipelineLogGroup

Outputs:
  ZeroETLRoleARN:
    Description: 'ARN of the Zero ETL IAM Role'
    Value: !GetAtt ZeroETLRole.Arn
  ZeroETLPipelineARN:
    Description: 'ARN of the Zero ETL Pipeline'
    Value: !Ref ZeroETLPipeline
  ZeroETLPipelineLogGroupName:
    Description: 'Name of the Zero ETL Pipeline Log Group'
    Value: !Ref ZeroETLPipelineLogGroup

Lessons learned:

• The index will be automatically created, but the data types might not be relevant.
• In the DynamoDB table, the rounded numbers like 10, 20,30,…. are exported to S3 as flot numbers(10.0,20.0,30.0,..), but the other numbers like 1,3,5,34 etc are proper integer values. So make sure you create the index with Flot or do a transform in your pipeline.
• You cannot pause and resume the pipeline, and it uses serverless capacity units. So you might pay for this even if your pipeline is idle(minimum OCU).

Conclusion:

Once the pipeline has been created, you can monitor it on the Pipeline console via CloudWatch. Hope you found this helpful.

Author

This blog post is written by our CTO Bhuvanesh and Lalit Khatter (PSA, AWS Ambassador APJ Lead)

ZeroETL — DynamoDB to OpenSearch Serverless Integration with CFT was originally published in ShellKode Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.