To be honest, I wasn't actively looking for a new backend solution. But Serverpod kept showing up—in my feeds, in Flutter newsletters, in conversations with other developers. The buzz was real.

So I blocked out a weekend to try it. No expectations, no pressure. Just a genuine attempt to see if the hype had any substance. Here's what I found.

Setup: The First Surprise

I expected the usual setup friction: Docker configs, environment variables, database connection strings, authentication middleware. Sure, I could ask Claude or Cursor to help with boilerplate, but even with AI agents speeding things up, getting a backend stack running still means an hour of setup before writing actual business logic.

Instead:

$ serverpod create weekend_experiment
Creating project...
✓ Server setup complete (packages/weekend_experiment_server)
✓ Client library generated (packages/weekend_experiment_client)
✓ Flutter app created (weekend_experiment_flutter)
✓ Database migrations ready
✓ Docker compose configured

The project structure was immediately familiar to any Flutter developer:

weekend_experiment/
├── weekend_experiment_server/     # Backend code
├── weekend_experiment_client/     # Auto-generated API client
├── weekend_experiment_flutter/    # Flutter app
└── docker-compose.yaml            # PostgreSQL + Redis ready

I ran docker compose up in the server directory. PostgreSQL and Redis spun up. Then dart run bin/main.dart --apply-migrations to initialize the database.

The Good: What Actually Impressed Me

1. The ORM That Doesn't Fight You

I've written enough SQL to know what good database tooling feels like. I've also suffered through ORMs that generate horrific queries and make simple things hard.

I created a model file to test Serverpod's ORM:

# user.spy.yaml
class: User
table: user
fields:
  name: String
  email: String
  age: int?
  premium: bool
  createdAt: DateTime

indexes:
  - fields: email
    unique: true

Ran serverpod generate. The framework generated:

• A Dart class with all fields
• JSON serialization/deserialization
• Database access methods
• Client-side models
• Migration files

Here's where it got interesting. The query API felt natural:

// Find all premium users over 25, ordered by creation date
final users = await User.db.find(
  session,
  where: (t) => t.premium.equals(true) & t.age.greaterThan(25),
  orderBy: (t) => t.createdAt.desc(),
  limit: 20,
);

// Complex join with relations
final postsWithAuthors = await Post.db.find(
  session,
  where: (t) => t.publishedAt.isNotNull(),
  include: (t) => [t.author, t.comments],
);

Type-safe. No string column names. No SQL injection possible. IntelliSense shows available fields.

I tested it with a deliberately broken query:

where: (t) => t.nonExistentField.equals(true)  // Compile error!

The Dart analyzer caught it immediately. No runtime surprises. No debugging production issues caused by typos.

2. Real-Time That Actually Works

Most real-time implementations involve setting up WebSocket servers, handling connections/disconnections, managing subscriptions, dealing with reconnection logic, and synchronizing state.

I wanted to test Serverpod's streaming capabilities with a simple counter.

Server endpoint:

class CounterEndpoint extends Endpoint {
  Stream watchCounter(Session session) async* {
    for (int i = 0; i < 100; i++) {
      await Future.delayed(Duration(seconds: 1));
      yield i;
    }
  }
}

Client usage:

client.counter.watchCounter().listen((count) {
  print('Counter: $count');
});

That's it. Serverpod handles WebSocket connection management, automatic reconnection on network loss, backpressure when the client is slow, and clean stream disposal.

For database-driven real-time updates:

class ChatEndpoint extends Endpoint {
  Stream watchRoom(Session session, int roomId) async* {
    await for (var message in Message.db.watch(
      session,
      where: (t) => t.roomId.equals(roomId),
    )) {
      yield message;
    }
  }
}

Any database change to messages in that room automatically streams to all connected clients. No manual pub/sub setup. No Redis channels. No message queues.

3. Code Generation That Makes Sense

Serverpod's generation is straightforward: models defined in YAML, code generated from your server structure. The output is readable Dart—no magic, no abstractions hiding complexity. When something's wrong, you can actually see what's happening.

$ serverpod generate
Analyzing server...
✓ Found 5 models
✓ Found 3 endpoints
Generating...
✓ Client library (packages/weekend_experiment_client/)
✓ Protocol models
✓ Database migrations

Time: ~5 seconds

4. Authentication Without the Tears

Authentication setup has gotten easier over the years—plenty of libraries handle OAuth flows. But integrating them still means configuration, testing callbacks, and making sure everything works across environments.

Serverpod includes serverpod_auth module with support for Google, Apple, and Email authentication.

Server setup:

import 'package:serverpod_auth/server.dart';

// Authentication is configured through the serverpod_auth package

Client setup:

import 'package:serverpod_auth/client.dart';

// Sign in with Google
final result = await client.auth.signInWithGoogle();
if (result.success) {
  print('Welcome ${result.userInfo.userName}');
}

The session automatically includes authenticated user:

class ProfileEndpoint extends Endpoint {
  Future getMyProfile(Session session) async {
    final userId = session.userId;
    if (userId == null) throw Exception('Not authenticated');

    return await Profile.db.findById(session, userId);
  }
}

No JWT parsing. No middleware chains. No authorization header management.

(Pro tip: If you add Google Sign-In to your iOS app, don't forget Apple Sign-In too, or Apple will happily reject your app.)

5. The Development Experience

This is subtle but important: everything is Dart.

• Server code? Dart.
• Client code? Dart.
• Database models? Dart (via YAML, but feels like Dart).
• Tests? Dart.
• Tooling? Dart CLI.

Writing both client and server in Dart eliminates language context switching. The tooling, patterns, and conventions are consistent across the stack. You can move between frontend and backend work without the cognitive overhead of switching ecosystems.

The Bad: Where It Falls Short

Here's where things weren't perfect.

1. Smaller Package Ecosystem

The third-party package ecosystem is limited compared to Express or Django. While you'll find some community packages (like Stripe integrations), you won't have the same breadth of options. Need a specific CMS integration? Elasticsearch connector? Twilio wrapper? You'll likely be building it yourself or adapting existing Dart packages.

AI coding assistants can help with general Dart patterns, but their Serverpod-specific knowledge is limited. They're useful for standard CRUD operations and basic server patterns, but you'll be reading documentation and source code more than you might with mainstream frameworks.

If your project heavily depends on integrating with specialized services or requires extensive third-party tooling, account for additional implementation time.

2. The Learning Curve for Complex Features

The basics are smooth. Creating endpoints, defining models, simple queries - all intuitive.

Advanced features? The documentation assumes you understand the underlying concepts.

Complex authorization patterns, advanced transaction handling, and intricate database relationships require piecing together solutions from Github issue and AI.

3. Migration Complexity

Database migrations work well for simple changes—adding nullable fields, creating tables. Complex changes require manual intervention:

• Data transformations
• Renaming columns while preserving data
• Splitting tables
• Complex constraint changes

The migration system doesn't automatically handle these scenarios. You can manually edit the generated SQL migrations, but this reduces some of the type-safe ORM benefits and requires careful testing.

4. Production Documentation Gaps

Getting started locally? Perfect.

Deploying to production? The docs cover Docker basics and provide a health check endpoint, but you'll need to figure out the production DevOps details yourself:

• Kubernetes-specific configurations (liveness/readiness probes, HPA)
• Graceful shutdown for zero-downtime deployments
• Database backup and disaster recovery strategies
• Log aggregation and centralized monitoring
• Production-grade security hardening

There's a community serverpod_vps package that helps, and GitHub discussions fill some gaps, but comprehensive production operations guides would be valuable.

5. Performance Data

User testimonials mention Serverpod handling 100,000+ daily requests in production, and there's a community benchmark comparing it to other frameworks. But comprehensive performance documentation is sparse.

• Latency distribution under various loads
• Horizontal scaling patterns
• Memory footprint under load
• Database connection pooling best practices
• Caching strategy guidelines

This makes capacity planning for larger applications more uncertain.

The Surprising: What I Didn't Expect

1. The File Upload System Is Actually Good

Most file upload implementations are painful. Serverpod surprised me here.

// Server endpoint
class FileEndpoint extends Endpoint {
  Future getUploadDescription(Session session, String fileName) async {
    return await session.storage.createDirectFileUploadDescription(
      storageId: 'public',
      path: fileName,
    );
  }
}

// Client uploads directly to cloud storage
final uploadDescription = await client.file.getUploadDescription('photo.jpg');
await uploader.upload(uploadDescription, fileBytes);

This generates presigned URLs for direct cloud storage uploads. No file data passing through your server. Supports S3 and Google Cloud Storage.

I didn't expect this level of sophistication in a young framework.

2. The Caching Layer Is Built-In

Redis integration isn't bolted on—it's part of the core API:

// Cache user profile
await session.caches.redis.put(
  'profile:$userId',
  profile,
  lifetime: Duration(minutes: 5),
);

// Retrieve from cache
final cached = await session.caches.redis.get('profile:$userId');

Type-safe caching with automatic serialization. The cache is session-scoped, available everywhere without manual dependency injection.

3. Future Calls for Async Background Jobs

I needed to send an email after user registration without blocking the response:

class UserEndpoint extends Endpoint {
  Future register(Session session, User user) async {
    final created = await User.db.insertRow(session, user);

    // Fire and forget - executes in background
    await session.messages.sendFutureCall(
      'email',
      'sendWelcome',
      {'userId': created.id},
      delay: Duration(seconds: 10),
    );

    return created;  // Response sent immediately
  }
}

These "future calls" are persisted to PostgreSQL, survive server restarts, and provide exactly-once execution semantics. No AWS SQS, No RabbitMQ.

This replaced what would normally require a separate message queue infrastructure.

The Verdict: Would I Use It Again?

After a weekend of exploration, here's my honest assessment:

Use Serverpod When:

• Building Flutter-first applications where the entire team works in Dart
• Type safety across the full stack is a priority
• Real-time features are core to the product (chat, collaboration, live updates)
• Development velocity matters more than ecosystem maturity
• The team prefers staying in a single language ecosystem

Consider Alternatives When:

• Third-party integrations are extensive and specialized
• The team spans multiple languages and stacks (Python ML services, Go microservices, etc.)
• Production deployment requires comprehensive operational documentation
• Integration with legacy systems or proprietary APIs is significant
• The project needs proven scalability patterns at massive scale

Conclusion

Serverpod is legitimately impressive for what it is: a modern, full-stack Dart framework that eliminates significant boilerplate while maintaining type safety from client to database. The code generation is clean, the ORM is practical, and real-time features work without managing WebSocket complexity.

The ecosystem is young. Documentation has gaps. You'll write more integration code than with established frameworks. But for Flutter teams building real-time applications, the developer experience is genuinely productive once you're past the initial learning curve.

Would I recommend it to my teams?
Yes, with clear expectations about ecosystem maturity and integration limitations.

Would I choose it over Node.js + Express?
If the team knows Flutter? Absolutely. If they don't? Probably not.

I had the opportunity to speak at D-Plus in Osaka back in May with a presentation called Task Management for Engineering Managers. Although I've only attended once, it was such a warm and welcoming community that I'd love to participate again when I can balance it with my household and parenting responsibilities.

Introduction

Recently, I attended a training session for middle managers at Monstarlab. During the training, I had the chance to reflect on my people management experience, and I had a shocking realization.

What I thought was people management turned out to be performance management.

In this article, I'd like to share that realization and what I learned from the training.

Monstarlab's Four Management Domains

In the tech industry, when it comes to management, Daichi Hiroki's "Four Ps" framework is well known. It organizes an engineering manager's areas of activity into four categories: People, Project, Product, and Platform (Technology).

At Monstarlab, we define manager roles across four domains—not limited to engineering:

1. People Management - Supporting team members' growth
2. Performance Management - Managing results and productivity
3. Strategy Management - Planning and executing strategy
4. Philosophy Management - Instilling organizational culture and values

(As a side note, if only we could rename "Strategy" to something starting with "P," we'd have our own Four Ps... How about Portfolio Management?)

What I Thought Was My People Management

As an engineering manager, I believed I was genuinely engaging with my team members.

• Conducting regular 1-on-1s
• Checking in on project status
• Identifying project challenges and brainstorming solutions together
• Making sure no one feels isolated

I even gave a presentation titled Connecting People and Organizations Through "En" - Value Creation and Unlocking Potential as an EM in Contract Development, discussing team management with a focus on three "Ens": Engagement, Empowerment, and Encouragement. I thought I was doing a decent job at people management—drawing out team members' potential and alleviating their anxieties.

But as I reflected during the training, I realized I wasn't actually engaging with the people themselves—I was engaging with their "work."

By focusing on project status and challenges, I was essentially concentrating on improving productivity and performance. In other words, I was doing performance management.

What I Learned About True People Management

People management is about supporting team members' growth. To do this effectively, it's crucial to understand each member's Will (what they want to do), Can (what they can do), and Must (what they need to do).

During the training, a manager from another team brilliantly drew out the Will, Can, and Must from someone playing the role of a team member. This person had a strong background in branding design, and it made sense—branding is all about extracting and organizing strengths and weaknesses. I found it fascinating that such skills translate so well to people management.

I learned that by truly engaging with team members and helping them articulate their Will, Can, and Must, we can effectively support their growth. At the same time, I also realized that I'm not particularly good at delving into people's inner thoughts and feelings.

After the training, I found myself buying books on branding design.

I'm hoping to pick up some insights that will help me level up my people management skills—from "not good at it and not doing it" to "not good at it but can manage reasonably well."

Summary

So that's the story of how I thought I was focusing on people management as a manager, when I was actually focusing on productivity ≈ performance.

Of course, performance management is important work too. But realizing that I was missing the perspective of supporting team members' growth was a significant takeaway.

During the training, another manager shared their experience about how even if you're not a perfect manager yourself, it's okay to leverage other managers' strengths to bring out the best in your team members. This was incredibly encouraging, especially since I had just become aware of my own people management challenges.

Going forward, I want to focus more on each team member's Will, Can, and Must during 1-on-1s and practice true people management.

Finally

Even though I'm still very much a work-in-progress as an EM, Monstarlab is looking for people who want to grow together with us.

In particular, we're working on launching a new team initiative like this:

A New Form of Engineer for the AI Era - What Full-Cycle Engineers Mean at Monstarlab

With the spirit of "Turn Vision into Reality," we're looking for people who want to become full-cycle engineers together. If you're interested, please apply!

Monstarlab Careers | Full-Cycle Engineer

Payload offered an approach that leveraged existing TypeScript skills while providing necessary backend capabilities. After deploying to production and maintaining the system through several high-traffic periods, this post reflects on the practical realities of building backend systems with Payload.

What is Payload?

Payload is an open-source, headless content management system (CMS) built with TypeScript and Next.js. Unlike traditional CMS platforms, such as WordPress or Drupal, a headless CMS delivers content via APIs, allowing any frontend stack to consume the content.

A major selling point of Payload is its code-first philosophy. While other headless CMS, such as Contentful and Strapi, require defining schemas through their admin panels, Payload lets you define everything in TypeScript. This approach appeals to developers who are comfortable with modern Javascript frameworks and want to extend the same development experience to their backend infrastructure.

Built on top of Next.js, Payload takes advantage of the framework's server-side rendering and API routing, while also providing its own database abstraction layer. It supports both PostgreSQL and MongoDB, offers REST and GraphQL APIs, and can be deployed on serverless platforms like AWS Lambda or on traditional servers.

For frontend developers stepping into backend work, this means you can manage content, define data structures, and expose APIs all within a familiar TypeScript environment, without needing learn a completely new backend stack.

Configuration-as-Code

Payload differentiates itself through its management of schema definitions as code, as opposed to GUI configurations or SQL statements. In Payload, collections are Typescript configuration objects, with strongly-typed fields that are automatically translated into database table columns.

export const Articles: CollectionConfig = {
  slug: "articles",
  fields: [
    {
      name: "title",
      type: "text",
      required: true,
    },
    {
      name: "body",
      type: "richtext",
    },
    {
      name: "category",
      type: "relationship",
      relationTo: "categories",
    },
  ],
};

From a frontend developer’s perspective, this syntax feels familiar — more like defining component props than database schemas. Additionally, because the code lives in version control, changes can be tracked, reviewed collaboratively, and rolled back if necessary.

The collection above automatically generates several artifacts:

Database Schema

Database tables are created based on the field configurations. The collection above will produce an articles table with the following columns:

REST API Endpoints

Every collection will also receive a complete REST API without additional configurations:

GET    /api/articles          # Retrieve list with pagination
GET    /api/articles/:id      # Retrieve single entity
POST   /api/articles          # Create new
PATCH  /api/articles/:id      # Update existing
DELETE /api/articles/:id      # Delete entity

Default query parameters are also provided for easy filtering and sorting. For instance, in order to retrieve the first 10 articles containing "frontend" in their title, sorted by the creation date:

/api/articles?where[title][contains]=frontend&sort=createdAt&limit=10

TypeScript Type Definitions

Type interfaces are also generated based on the collection configuration, providing type checking and autocomplete support in modern IDEs.

export interface Articles {
  id: number;
  title: string;
  body?: {
    root: {
      type: string;
      children: {
        type: string;
        version: number;
        [k: string]: unknown;
      }[];
      direction: ("ltr" | "rtl") | null;
      format: "left" | "start" | "center" | "right" | "end" | "justify" | "";
      indent: number;
      version: number;
    };
    [k: string]: unknown;
  } | null;
  category?: {
    relationTo: "category";
    value: number | Category;
  } | null;
  createdAt?: string | null;
  updatedAt?: string | null;
}

This automatic artifact generation eliminates a considerable amount of boilerplate code and maintenance, making backend development faster and more reliable.

Reducing the Backend Learning Curve

Several Payload features make backend development more approachable for frontend developers:

Built-in Admin Panel

Payload includes a pre-built, auto-generated admin panel that leverages Next.js Hot Module Replacement for instant feedback during development. The production-ready admin panel automatically reflects TypeScript schema changes and is highly customizable for projects with complex UI requirements.

Internationalization and Localization

Supporting multiple languages is handled through configuration rather than complex setup. Fields can be marked as localized, allowing content editors to manage translations directly in the admin panel. The system automatically handles locale-specific queries and provides fallback behavior when translations are missing.

export const Articles: CollectionConfig = {
  slug: "articles",
  fields: [
    {
      name: "title",
      type: "text",
      localized: true, // Enable per-locale content
    },
  ],
};

API requests can specify locale preferences via query parameters, and Payload handles the underlying database queries to retrieve the appropriate content version. For instance, requesting /api/articles?locale=ja&fallback-locale=en returns Japanese content when available, falling back to English otherwise.

Data Relationship Modeling

Creating relationships between data structures uses the relationship field type, as shown in the sample collection above. Payload handles the underlying database joins and provides a functional admin UI for selecting related content.

Hooks

Collection-level and field-level hooks (beforeRead, afterRead, beforeChange, afterChange, etc.) mimic React lifecycle methods, allowing you to implement business logic with relative ease.

export const Articles: CollectionConfig = {
  slug: "articles",
  fields: [
    {
      name: "title",
      type: "text",
      required: true,
    },
    {
      name: "slug",
      type: "text",
    },
    //...
  ],
  hooks: {
    beforeChange: [
      ({ data }) => {
        data.slug = data.title.toLowerCase().replace(/\s+/g, "-");
      },
    ],
  },
};

Custom Endpoints

Beyond the standard CRUD operations, you can define custom endpoints to handle more complex workflows.

export const Articles: CollectionConfig = {
  slug: "articles",
  fields: [
    /* ... */
  ],
  endpoints: [
    {
      path: "/:id/summary", // Define endpoint
      method: "get", // Define method
      handler: async req => {
        if (!req.user) {
          return Response.json({ error: "Unauthorized" }, { status: 401 });
        }
        const article = await req.payload.findByID({
          collection: "articles",
          id: req.routeParams.id as string,
        });

        // Execute complex operation, call external API, etc.
        const summary = await generateAISummary(article.body);

        return Response.json({ success: true, summary });
      },
    },
  ],
};

Access Control

Authorization can be enforced at both collection and field levels, ensuring security is handled consistently at the data layer rather than relying on UI visibility.

export const Articles: CollectionConfig = {
  slug: "articles",
  access: {
    // Allow anyone to read "published" articles
    read: ({ req: { user } }) => {
      if (user) return true;
      return { status: { equals: "published" } };
    },
    // Only allow authenticated users to create, update, or delete articles
    create: ({ req: { user } }) => Boolean(user),
    update: ({ req: { user } }) => Boolean(user),
    delete: ({ req: { user } }) => Boolean(user),
  },
};

This declarative approach makes access control simpler and more maintainable than manually checking permissions in every route handler. Security is enforced consistently for every request, giving you confidence that your data layer is protected.

Caveats and Challenges

While Payload provides a compelling developer experience, moving to production surfaces several challenges worth considering.

Database & Schema Management

During development, the TypeScript-to-database workflow feels seamless, but in production with PostgreSQL, explicit migration management is required. Payload provides migration scripts, but you need to run payload migrate:create for every schema change and carefully manage the order of migrations across environments. This can add coordination overhead when multiple developers are updating schemas simultaneously.

Serverless Deployment Complexity

Deploying Payload as a serverless function can be more complex than expected. Cold starts can impact performance, but in a live production service, they are generally not an issue with typical traffic patterns and proper memory allocation. Additionally, Payload does not natively support scheduled tasks or batch jobs in serverless environments, so features like scheduled publishing require building separate infrastructure using Lambda functions and EventBridge scheduling.

React and Next.js Integration

Since Payload is built on Next.js with the App Router, you also inherit React Server Component challenges. Custom admin panel components must carefully navigate the server/client boundary, and hydration mismatches can occur when building custom fields that interact with browser APIs. If you've struggled with Next.js hydration errors before, expect to encounter them here as well.

Conclusion

As a frontend developer, working with Payload felt like extending my TypeScript toolset rather than learning an entirely new backend framework. Its familiar patterns, automatic API generation, and strong schema typing made backend development more approachable than expected.

There are still challenges — managing migrations, optimizing serverless deployments, and understanding the underlying database structures take deliberate effort. However, Payload narrows the gap between frontend and backend development by letting you apply existing TypeScript experience to backend systems. Payload doesn't cover everything — advanced database optimization, distributed systems, and infrastructure management require dedicated learning. But it offers a practical entry point.

For developers comfortable with React and modern build pipelines, it proves that backend systems can be built without leaving familiar territory.

Payload, the Payload design, and related marks, designs, and logos are trademarks or registered trademarks of Payload CMS, Inc. in the U.S. and other countries.

From monitoring system performance to analyzing customer behavior, the ability to process data in real-time offers invaluable insights for timely decision-making.

However, one critical aspect often overlooked is handling failures gracefully in real-time data processing pipelines.

In this part of our series, we will explore the importance of implementing robust retry strategies.

Important Note: AWS has announced that AWS App Mesh will be discontinued in September 2026. While App Mesh remains fully supported until then, you should consider AWS Service Connect instead. AWS Service Connect provides similar functionality with improved performance and simplified configuration.

Part 1: AWS ECS App Mesh and Retry Strategies

Amazon Elastic Container Service (ECS) is a fully managed container orchestration service that allows you to run, stop, and manage Docker containers on a cluster. It eliminates the need to install, operate, and scale a cluster management infrastructure.

AWS App Mesh belongs to the category of service meshes, which are specialized infrastructure layers designed to manage communication between services within a distributed application architecture.

Essentially, AWS App Mesh simplifies the networking aspect of your applications by offering features like service discovery, load balancing, encryption, authentication, and observability.

In essence, AWS App Mesh streamlines communication between microservices, allowing developers to focus on building application logic rather than worrying about networking setup.

This approach enhances the reliability, security, and observability of modern distributed systems. Other popular service mesh implementations include Istio and Linkerd.

AWS App Mesh, in conjunction with Amazon ECS, offers a robust platform for deploying microservices architectures that are well-suited for real-time data processing.

While App Mesh offers features like service discovery, traffic management, and observability, its inherent ability to handle failures through retry policies is crucial for ensuring data integrity and system reliability.

Importance of Retry Strategies

In real-time data processing, failures are inevitable due to network issues, service disruptions, or transient errors.

Therefore, implementing effective retry strategies becomes crucial. Here are some key aspects to consider:

1. Determining Retry Attempts: Deciding the number of retry attempts depends on factors like the criticality of the data, the likelihood of transient failures, and the impact on downstream processes. It's essential to find a balance between ensuring data delivery and avoiding endless retries, which could lead to resource exhaustion.

2. Exponential Backoff: Adopting exponential backoff strategies can prevent overwhelming downstream systems during high-load scenarios. Gradually increasing the time between retry attempts reduces the likelihood of further failures.

3. Dead Letter Queues (DLQ): Implementing DLQs allows you to capture failed messages for further analysis and manual intervention. It's crucial to set up robust monitoring and alerting mechanisms to promptly address issues identified in the DLQ.

Testing App Mesh Retry Policy

Let's put the App Mesh retry policy to the test by simulating failure scenarios in a real-time data processing pipeline.

We'll set up two services: a Data Ingestion service and a Data Processing service using Terraform.

1. Data Ingestion Service: This service receives data from external sources and forwards it to the processing pipeline.

2. Data Processing Service: This service analyzes incoming data in real-time, performing tasks such as anomaly detection or aggregation.

Terraform Configuration

We'll use Terraform to define the infrastructure for the Data Ingestion service and Data Processing service, along with the App Mesh configuration.

We will omit the full Terraform configuration for brevity, but here's a high-level overview of the key components:

1. App Mesh

Let's start with the core components of the app mesh.

• Service Mesh: The logical boundary for the services that make up the app mesh itself.

resource "aws_appmesh_mesh" "app-mesh" {
  name = "${var.env}-${var.project}-app-mesh"

  spec {
    egress_filter {
      type = "DROP_ALL" # Allow only egress from virtual nodes to other resources within the mesh
    }
  }
}

• Virtual Service: Abstract representation of the Data Ingestion service and the Data Processing service running in the mesh.

# data-processing virtual service
resource "aws_appmesh_virtual_service" "data-processing-service" {
  name      = "${local.services.data-processing}.${var.env}.${var.internal_domain}"
  mesh_name = aws_appmesh_mesh.app-mesh.id

  spec {
    provider {
      virtual_router {
        virtual_router_name = aws_appmesh_virtual_router.data-processing-service.name
      }
    }
  }
}

# data-ingestion virtual service
# ...

The virtual router is responsible for handling traffic for the virtual service. It is defined later in the configuration with routing rules and retry policies for the virtual service.

• Virtual Node: Concrete implementations behind the abstracted virtual services. Each virtual node points to a specific ECS service, where actual code runs.

# data-ingestion virtual node
resource "aws_appmesh_virtual_node" "data-ingestion-service" {
  name      = "${var.env}-${var.project}-data-ingestion-service"
  mesh_name = aws_appmesh_mesh.app-mesh.id

  spec {
    backend {
      virtual_service {
        virtual_service_name = aws_appmesh_virtual_service.data-processing-service.name
      }
    }

    listener {
      port_mapping {
        port     = var.ecs_services["data-ingestion"].app_port
        protocol = "http"
      }

      timeout {
        http {
          per_request {
            value = var.ecs_services["data-ingestion"].app_mesh_timeout.value
            unit  = var.ecs_services["data-ingestion"].app_mesh_timeout.unit
          }
        }
      }
    }

    service_discovery {
      aws_cloud_map {
        service_name   = aws_appmesh_virtual_service.data-processing-service.name
        namespace_name = aws_service_discovery_private_dns_namespace.internal.name
      }
    }

    logging {
      access_log {
        file {
          path = "/dev/stdout"
        }
      }
    }
  }
}


# data-processing virtual node
# ...

In this example, we have defined the virtual nodes for the Data Processing service and the Data Ingestion service.

In the Data Ingestion virtual node, we have defined the Data Processing virtual service as the backend to which the virtual node is expected to send outbound traffic.

We have specified the listener for the virtual nodes, which defines the port and protocol for incoming traffic.

We have also specified the service discovery mechanism for the virtual node, which in this case is AWS Cloud Map.

• Route: Specifies how traffic flows between services. You can define routing rules based on various criteria like service name, attributes, or weighted distribution.

# data-processing App Mesh virtual router
resource "aws_appmesh_virtual_router" "data-processing-service" {
  name      = "${var.env}-${var.project}-data-processing-service-virtual-router"
  mesh_name = aws_appmesh_mesh.app-mesh.id

  spec {
    listener {
      port_mapping {
        port     = var.ecs_services["data-processing"].app_port
        protocol = "http"
      }
    }
  }
}

# data-processing service App Mesh route
resource "aws_appmesh_route" "data-processing-service" {
  name                = "${var.env}-${var.project}-data-processing-service"
  mesh_name           = aws_appmesh_mesh.app-mesh.id
  virtual_router_name = aws_appmesh_virtual_router.data-processing-service.name

  spec {
    http_route {
      match {
        prefix = "/"
      }

      retry_policy {
        http_retry_events = [
          "gateway-error",
        ]
        max_retries = 12
        per_retry_timeout {
          unit  = "s"
          value = 5
        }
        tcp_retry_events = [
          "connection-error",
        ]
      }

      action {
        weighted_target {
          virtual_node = aws_appmesh_virtual_node.data-processing-service.name
          weight       = 1
        }
      }
    }

    priority = 1
  }
}

# data-ingestion service App Mesh virtual router
# ...

# data-ingestion service App Mesh route
# ...

We have defined the route to match any incoming HTTP traffic and route it with retries to the Data Processing service virtual node.

The retry policy specifies that the service should retry on gateway errors and connection errors, with a maximum of 12 retries and a per-retry timeout of 5 seconds.

Note that the action specifies the weighted target, which is the virtual node to which the traffic should be routed. We have set the weight to 1, indicating that all traffic should be routed to the Data Processing service virtual node.

Note that when defining virtual services, we could have used a virtual node as the provider for the virtual service and omitted a virtual router, which would not provide granular control over the routing rules.

# data-processing virtual service
resource "aws_appmesh_virtual_service" "data-processing-service" {
  name      = "${local.services.data-processing}.${var.env}.${var.internal_domain}"
  mesh_name = aws_appmesh_mesh.app-mesh.id

  spec {
    provider {
      virtual_node {
        virtual_node_name = aws_appmesh_virtual_node.data-processing-service.name
      }
    }
  }
}

• Cloud Map Service Discovery: This allows services to discover and communicate with each other using custom DNS names.

# Private DNS Namespace
resource "aws_service_discovery_private_dns_namespace" "internal" {
  name        = "${var.env}.${var.internal_domain}"
  description = "${var.env}-${var.project}-private-dns-namespace"
  vpc         = var.aws_vpc-vpc-id
}

# data-processing
resource "aws_service_discovery_service" "data-processing" {
  name = local.services.data-processing

  dns_config {
    namespace_id = aws_service_discovery_private_dns_namespace.internal.id

    dns_records {
      ttl  = 10
      type = "A"
    }

    routing_policy = "MULTIVALUE"
  }

  health_check_custom_config {
    failure_threshold = 1
  }
  
  lifecycle {
    create_before_destroy = true
  }
}

2. AWS ECS

We'll now define the ECS cluster configuration for the Data Ingestion service and the Data Processing service.

• ECS Cluster

# ECS cluster
resource "aws_ecs_cluster" "ecs-cluster" {
  name = "${var.env}-${var.project}-ecs-cluster"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }
}

# ECS cluster capacity provider
resource "aws_ecs_cluster_capacity_providers" "ecs-cluster" {
  cluster_name = aws_ecs_cluster.ecs-cluster.name

  capacity_providers = ["FARGATE"]

  default_capacity_provider_strategy {
    weight            = 1
    capacity_provider = "FARGATE"
  }
}

• ECS Services

# data-processing
resource "aws_ecs_service" "data-processing-service" {
  name                               = local.services.data-processing
  cluster                            = aws_ecs_cluster.ecs-cluster.id
  task_definition                    = aws_ecs_task_definition.ecs-data-processing-task.arn
  launch_type                        = "FARGATE"
  deployment_maximum_percent         = var.ecs_services["data-processing"].deployment_maximum_percent
  deployment_minimum_healthy_percent = var.ecs_services["data-processing"].deployment_minimum_healthy_percent
  desired_count                      = var.ecs_services["data-processing"].desired_count
  force_new_deployment               = true

  lifecycle {
    ignore_changes = [desired_count, task_definition]
  }

  network_configuration {
    subnets         = var.aws_subnet-protected-ids
    security_groups = [aws_security_group.ecs-service.id]
  }
  
  service_registries {
    registry_arn = aws_service_discovery_service.data-processing.arn
  }
}

# data-ingestion
# ...

We have created the ECS cluster and defined the ECS services for the Data Processing and Data Ingestion services.

• Task Definition

# data-processing task definition
resource "aws_ecs_task_definition" "ecs-data-processing-task" {
  family                   = "${var.env}-${var.project}-data-processing-service"
  requires_compatibilities = ["FARGATE"]
  cpu                      = var.ecs_services["data-processing"].task_definition.cpu
  memory                   = var.ecs_services["data-processing"].task_definition.memory
  network_mode             = "awsvpc"
  execution_role_arn       = aws_iam_role.task-exec.arn
  task_role_arn            = aws_iam_role.ecs-task.arn

  container_definitions = jsonencode([
    {
      name      = "data-processing-service"
      image     = "${aws_ecr_repository.data-processing-service.repository_url}:${var.app_version}"
      essential = true
      environment = [
        {
          name  = "APP_PORT"
          value = tostring(var.ecs_services["data-processing"].app_port)
        }
      ]
      portMappings = [
        {
          containerPort = var.ecs_services["data-processing"].app_port
          protocol      = "tcp"
        }
      ]
      healthCheck = {
        command = [
          "CMD-SHELL",
          "curl -s http://localhost:${var.ecs_services["data-processing"].app_port}/health-check || exit 1"
        ]
        interval    = 20
        retries     = 5
        startPeriod = 10
        timeout     = 5
      }
      depends_on = [
        {
          "containerName" : "envoy",
          "condition" : "HEALTHY"
        }
      ]
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group         = aws_cloudwatch_log_group.data-processing-service.name
          awslogs-region        = var.region
          awslogs-stream-prefix = "awslogs-data-processing"
        }
      }
    },
    {
      name      = "envoy"
      image     = "000000000000.dkr.ecr.${data.aws_region.current.name}.amazonaws.com/aws-appmesh-envoy:v1.27.2.0-prod"
      essential = true
      cpu       = var.ecs_services["data-processing"].task_definition.envoy_cpu
      memory    = var.ecs_services["data-processing"].task_definition.envoy_memory
      environment = [
        {
          name  = "APPMESH_RESOURCE_ARN",
          value = aws_appmesh_virtual_node.data-processing-service.arn
        },
        {
          name  = "ENVOY_LOG_LEVEL",
          value = "info"
        },
        {
          name  = "ENVOY_INITIAL_FETCH_TIMEOUT",
          value = "30"
        },
        {
          name  = "ENABLE_ENVOY_XRAY_TRACING",
          value = "1"
        },
      ]
      portMappings = [
        {
          protocol      = "tcp",
          containerPort = 9901
        },
        {
          protocol      = "tcp",
          containerPort = 15000
        },
        {
          protocol      = "tcp",
          containerPort = 15001
        }
      ]
      healthCheck = {
        command = [
          "CMD-SHELL",
          "curl -s http://localhost:9901/server_info | grep state | grep -q LIVE"
        ],
        interval    = 5,
        retries     = 3,
        startPeriod = 60,
        timeout     = 2
      }
      user = "1337"
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group         = aws_cloudwatch_log_group.data-processing-service.name
          awslogs-region        = var.region
          awslogs-stream-prefix = "awslogs-envoy-data-processing"
        }
      }
    },
    {
      name              = "xray-daemon"
      image             = "public.ecr.aws/xray/aws-xray-daemon:latest"
      essential         = false
      cpu               = var.ecs_services["data-processing"].task_definition.xray_cpu
      memoryReservation = var.ecs_services["data-processing"].task_definition.xray_memory
      portMappings = [
        {
          hostPort      = 2000
          containerPort = 2000
          protocol      = "udp"
        }
      ]
      user = "1337"
      logConfiguration = {
        logDriver = "awslogs"
        options = {
          awslogs-group         = aws_cloudwatch_log_group.data-processing-service.name
          awslogs-region        = var.region
          awslogs-stream-prefix = "awslogs-xray-data-processing"
        }
      }
    }
  ])
  proxy_configuration {
    type           = "APPMESH"
    container_name = "envoy"
    properties = {
      AppPorts           = var.ecs_services["data-processing"].app_port
      EgressIgnoredIPs   = "169.254.170.2,169.254.169.254"
      IgnoredUID         = "1337"
      ProxyEgressPort    = 15001
      ProxyIngressPort   = 15000
      EgressIgnoredPorts = 22
    }
  }
}

# data-ingestion task definition
# ...

The Data Processing and Data Ingestion services task definitions include:

• Individual container configurations: This includes specifying the image used, necessary environment variables, port mappings for communication, health checks to ensure proper functioning, and logging setups for analysis.

• App Mesh integration: The defined proxy configuration enables the Envoy sidecar proxy to intercept and route both incoming and outgoing traffic through the App Mesh service mesh. Envoy acts as a high-performance data plane component, essentially managing the network interactions within the mesh. For deeper understanding, please refer to the official Envoy documentation.

• Distributed tracing: The inclusion of a xray-daemon container allows for tracing the flow of requests across various services, providing valuable insights into system performance and potential issues via AWS X-Ray.

Here's a high-level overview of the Terraform configuration for the Data Ingestion and the Data Processing services.

Backend Services

Let's proceed to develop the backend applications for the Data Ingestion and Data Processing services using Golang.

1. Data Ingestion Service

func main() {
	// Create a new router
	router := mux.NewRouter()

	// Define routes
	router.HandleFunc("/health-check", HealthCheckHandler).Methods("GET")
	router.HandleFunc("/ev", ElectricVehicleHandler).Methods("POST")

	// Start the server
	log.Fatal(http.ListenAndServe(":3000", router))
}

// ElectricVehicleHandler handles the /ev endpoint
func ElectricVehicleHandler(w http.ResponseWriter, r *http.Request) {
	log.Printf("Request: %s %s", r.Method, r.URL.Path)

	// Read request body
	body, err := io.ReadAll(r.Body)
	if err != nil {
		log.Println(err)
		http.Error(w, "Failed to read request body", http.StatusInternalServerError)
		return
	}

	// Decode JSON payload
	var payload ElectricVehiclePayload
	err = json.Unmarshal(body, &payload)
	if err != nil {
		log.Println(err)
		http.Error(w, "Failed to decode JSON payload", http.StatusBadRequest)
		return
	}

	// Do whatever pre-processing is required
	// ...

	// Create a request to the data processing service endpoint
	req, err := http.NewRequest("POST", dataProcessingEndpoint, bytes.NewBuffer(body))
	if err != nil {
		log.Println(fmt.Errorf("failed to create request to data processing service: %w", err))
		http.Error(w, "Failed to POST to data processing service", http.StatusInternalServerError)
		return
	}

	// Clone headers from the incoming request to the outgoing request
	req.Header = r.Header.Clone()

	// HTTP POST request to data processing service
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Println(fmt.Errorf("failed to create request to data processing service: %w", err))
		http.Error(w, "Failed to POST to data processing service", http.StatusInternalServerError)
		return
	}

	// Check if response status code is not 200
	if resp.StatusCode != http.StatusOK {
		msg := fmt.Sprintf("Data processing service returned non-200 status code: %d", resp.StatusCode)
		log.Print(msg)
		http.Error(w, msg, resp.StatusCode)
		return
	}

	// Copy the response from the data processing service to the current response writer
	_, err = io.Copy(w, resp.Body)
	if err != nil {
		log.Println(err)
		http.Error(w, "Failed to copy response", http.StatusInternalServerError)
		return
	}
}

The Data Ingestion service is a simple HTTP server that listens for incoming POST requests via the /ev endpoint.

This service is supposed to receive data from external sources (Electric Vehicle data in this case).

In this example, we've defined the ElectricVehicleHandler function to handle incoming EV data. It decodes the JSON payload, performs any pre-processing required, and forwards the request to the Data Processing service.

2. Data Processing Service

func main() {
	// Create a new router
	router := mux.NewRouter()

	// Define routes
	router.HandleFunc("/health-check", HealthCheckHandler).Methods("GET")
	router.HandleFunc("/ev", ElectricVehicleDataProcessingHandler).Methods("POST")

	// Start the server
	log.Fatal(http.ListenAndServe(":3000", router))
}

// ElectricVehicleDataProcessingHandler handles EV data
func ElectricVehicleDataProcessingHandler(w http.ResponseWriter, r *http.Request) {
	// Log the request
	log.Printf("Request: %s %s", r.Method, r.URL.Path)

	// return 503 if x-503 header is set
	if value, ok := r.Header["X-503"]; ok {
		log.Printf("X-503 header is set with values: %v", value)
		w.WriteHeader(http.StatusServiceUnavailable)
		_, err := w.Write([]byte("Data Processing Service will return 503 ==> called with x-503 header set."))
		if err != nil {
			log.Println(err)
		}
		return
	}

	// Read request body
	body, err := io.ReadAll(r.Body)
	if err != nil {
		log.Println(err)
		http.Error(w, "Failed to read request body", http.StatusInternalServerError)
		return
	}

	// Decode JSON payload
	var payload ElectricVehiclePayload
	err = json.Unmarshal(body, &payload)
	if err != nil {
		log.Println(err)
		http.Error(w, "Failed to decode JSON payload", http.StatusBadRequest)
		return
	}

	// Process the payload
	// ...

	// Response back to the client
	w.WriteHeader(http.StatusOK)
	_, err = w.Write([]byte("Payload processed successfully"))
	if err != nil {
		log.Println(err)
		return
	}
}

The Data Processing Service is also a simple HTTP server that listens for incoming POST requests via the /ev endpoint.

To simulate a failure scenario, the ElectricVehicleDataProcessingHandler function returns a 503 status code if the x-503 header is set.

This service is supposed to analyze incoming data in real-time, performing tasks such as anomaly detection or aggregation.

After running terraform apply and deploying the backend services, we can check from the Dashboard that the services are running and healthy.

and the App Mesh configuration is also set up correctly.

Simulating Failure Scenarios

We'll simulate failure scenarios by passing the x-503 header to the Data Processing service, causing it to return a 503 status code.

Using AWS Session Manager to get inside the protected subnet, let's first confirm that calls to our services are going through the envoy proxy.

We can see that services are healthy and calls are going through the envoy proxy.

• server: envoy: Indicates that the request is being handled by the Envoy Proxy.

• x-envoy-upstream-service-time: 1: Indicates the time (in milliseconds) taken by Envoy Proxy to communicate with the upstream service (backend services in this case).

Now, we'll call the Data Ingestion service with the x-503 header set, and observe the behavior of the App Mesh retry policy.

curl --location 'http://data-ingestion.dev.smn-app-mesh-ecs.internal:3000/ev' \
     --header 'Content-Type: application/json' \
     --header 'x-503: true' \
     --data '{
      "vehicle_id": "EV-001",
      "timestamp": "2024-02-15T10:30:00Z",
      "location": {
        "latitude": 37.7749,
        "longitude": -122.4194
      },
      "battery": {
        "percentage": 75,
        "voltage": 390,
        "temperature": 25.3
      },
      "speed": 60,
      "odometer": 12500
    }
'

• Data Ingestion service logs:

• Data Processing service logs:

For a single request sent to the Data Ingestion service, we observed that the Data Processing service returned a 503 status code. Consequently, the request was retried 12 times in accordance with the retry policy.

In this case, since the Data Processing service is implemented to return a 503 status code when the x-503 header is set, the request failed after 12 retries.

In a real-world scenario, the Data Processing service would eventually recover from the transient failure, and the request would be successfully processed.

In the absence of the App Mesh retry policy, the request to the Data Processing service would have failed, and the Data Ingestion service would have returned an error to the client.

It's worth noting that in the absence of the App Mesh retry policy, one would have implemented an HTTP retry mechanism in the Data Ingestion Golang code. Utilizing the App Mesh retry policy offers a more robust and centralized approach, allowing developers to focus on application logic rather than network setup concerns.

Conclusion

Incorporating retry strategies is imperative for ensuring the resilience and reliability of real-time data communication between microservices.

With AWS ECS App Mesh, coupled with effective retry policies, organizations can mitigate the impact of failures and uphold data integrity in critical business processes.

In the next part of our series, we will explore additional crucial elements of real-time data processing.

What is Shorebird?

Shorebird is a service that allows Flutter apps to be updated directly at runtime. Removing the need to build and submit a new app version to Apple Store Connect or Play Console for review for every app update.

Flutter already comes with hot reload/restart that allows developers to quickly test code changes during development. Shorebird is similarly used for changes in production apps already in the hands of end-users.

Why?

Enabling production code to be updated without needing to go through the usual store review process, which greatly reduces the time required to release a critical bug fix. With finer control over app updates it would also let us develop mobile apps with smaller release windows and flexibility similar to web site deployments.

How?

Shorebird uses a modified version of the Flutter engine that allows updated code changes to be run at runtime using the Dart Virtual Machine. Since running code in the Virtual Machine is a lot slower then regular compiled code, Shorebird is smart enough to detect only the changed code parts to minimize its usage.

Broadly there are two main concepts.
Release: A full app binary containing the whole Flutter app, the same as you would normally build when releasing an app. With the difference it also contains a custom Flutter engine modified to run code in a virtual machine and functionality to check for updates. The release app needs to be reviewed and distributed through app stores as usual.

Patch: The patch only contains the code changes required for a Shorebird code push update. It's created by comparing the changes to the targeted release binary.

Both are created using the Shorebird CLI tool with the shorebird release and shorebird patch commands respectively. Behind the scenes this wraps the usual flutter build command.

Simple usage flow

1. Install Shorebird CLI tools
2. Register your Flutter app with Shorebird, using the shorebird init command
3. Create a new app release with the shorebird release command
4. Distribute the shorebird release app to app stores (or applicable distribution channels)
5. Modify some dart code in the app (adding new functionality/fixing a bug)
6. Create a patch with the shorebird patch command
7. After the patch has been uploaded to Shorebird servers, on the next app start the patch will be downloaded and applied.

By default the app will check for any new patches and update the app at startup. The update check can also be done programmatically for more advanced use cases.

Example demo

For this example we will be following the Getting Started guide from the Shorebird documentation. In short we will be adding Shorebird to the Flutter counter example app, change some code and push a patch to the app.

1. First we need to install the Shorebird CLI tools by using the handy install script.

curl --proto '=https' --tlsv1.2 https://raw.githubusercontent.com/shorebirdtech/install/main/install.sh -sSf | bash

For this demo I'm using a Mac setup but there are also instructions for Windows environments in the documentation. See the Getting Started link above.

2. After the CLI tools have been installed we now have to set up the Flutter project to use Shorebird using the shorebird init command. You will also be prompted to select a name for your app. It's only used to manage your app in the Shorebird console and won't be shown to end-users.

user@my-mac shorebird_test % shorebird init
✓ No product flavors detected. (19.2s)
? How should we refer to this app? (shorebird_test) shorebird_test

🐦 Shorebird initialized successfully!

✅ A shorebird app has been created.
✅ A "shorebird.yaml" has been created.
✅ The "pubspec.yaml" has been updated to include "shorebird.yaml" as an asset.

Reference the following commands to get started:

📦 To create a new release use: "shorebird release".
🚀 To push an update use: "shorebird patch".
👀 To preview a release use: "shorebird preview".

For more information about Shorebird, visit https://shorebird.dev
✓ Shorebird is up-to-date (1.4s)
✓ Flutter install is correct (15.4s)
✓ AndroidManifest.xml files contain INTERNET permission (1 fix applied) (83ms)
✓ Has access to storage.googleapis.com (0.2s)
✓ shorebird.yaml found in pubspec.yaml assets (9ms)

Checking the Shorebird console we can see our app has been added.

3. Now we can create our first Shorebird supported release binary using the shorebird release command. Creating an apk file for Android only just to simplify things. This would need to be run again for the iOS app.

user@my-mac shorebird_test % shorebird release android --artifact apk
✓ Downloading patch... (0.4s)
✓ Extracting patch... (0.6s)
✓ Downloading bundletool.jar... (0.7s)
✓ Extracting bundletool.jar... (1.9s)
✓ Downloading aot-tools.dill... (0.2s)
✓ Extracting aot-tools.dill... (1.5s)
✓ Fetching apps (0.4s)
✓ Building app bundle with Flutter 3.24.1 (3f7041c5e9) (75.8s)
✓ Release version: 1.0.0+1 (1.3s)
✓ Fetching releases (0.4s)

🚀 Ready to create a new release!

📱 App: shorebird_test (ab486a37-ft3d-472d-l520-ba1h7e3c3801)
📦 Release Version: 1.0.0+1
🕹️  Platform: android
🐦 Flutter Version: 3.24.1 (3f7041c5e9)

Would you like to continue? (y/N) Yes
✓ Fetching releases (0.9s)
✓ Creating release (0.7s)
✓ Updating release status (0.3s)
✓ Creating artifacts (9.0s)
✓ Updating release status (1.8s)

✅ Published Release 1.0.0+1!
Your next step is to upload the app bundle to the Play Store:
/Users/christofer_henriksson/shorebird_test/build/app/outputs/bundle/release/app-release.aab

For information on uploading to the Play Store, see:
https://support.google.com/googleplay/android-developer/answer/9859152?hl=en

To create a patch for this release, run shorebird patch --platforms=android --release-version=1.0.0+1

Note: shorebird patch --platforms=android without the --release-version option will patch the current version of the app.

4. Installing and running the release shows the default counter example as expected.

5. Now let's make some small changes to the app.

Changing the colorScheme to green.

- colorScheme: ColorScheme.fromSeed(seedColor: Colors.deepPurple),
+ colorScheme: ColorScheme.fromSeed(seedColor: Colors.green),

Adding a decrement button

+  void _decrementCounter() {
+    if (_counter > 0) {
+      setState(() {
+        _counter--;
+      });
+    }
+  }

-      floatingActionButton: FloatingActionButton(
-        onPressed: _incrementCounter,
-        tooltip: 'Increment',
-        child: const Icon(Icons.add),
-      ),

+      floatingActionButton: Row(
+        mainAxisAlignment: MainAxisAlignment.end,
+        children: [
+          FloatingActionButton(
+            onPressed: _decrementCounter,
+            tooltip: 'Decrement',
+            child: const Icon(Icons.remove),
+          ),
+          const SizedBox(width: 8),
+          FloatingActionButton(
+            onPressed: _incrementCounter,
+            tooltip: 'Increment',
+            child: const Icon(Icons.add),
+          ),
+        ],
+      ),

6. Next we will create the patch for the above changes with the shorebird patch command.

user@my-mac shorebird_test % shorebird patch android
✓ Fetching apps (0.5s)
✓ Fetching releases (0.3s)
Which release would you like to patch? 1.0.0+1
✓ Fetching aab artifact (0.3s)
✓ Downloading aab (2.2s)
✓ Building patch with Flutter 3.24.1 (3f7041c5e9) (41.2s)
✓ Verifying patch can be applied to release (28ms)
[WARN] Your app contains asset changes, which will not be included in the patch.
    Changed files:
        base/assets/flutter_assets/fonts/MaterialIcons-Regular.otf
Continue anyways? (y/N) Yes
✓ Fetching release artifacts (0.9s)
✓ Downloading release artifacts (4.0s)
✓ Creating patch artifacts (4.2s)

🚀 Ready to publish a new patch!

📱 App: shorebird_test (a1ac6a37-bc3d-459d-b820-bd1f4e3c5391)
📦 Release Version: 1.0.0+1
🕹️  Platform: android [arm32 (87.71 KB), arm64 (84.91 KB), x86_64 (89.81 KB)]
🟢 Track: Production

Would you like to continue? (y/N) Yes
✓ Creating patch (1.7s)
✓ Uploading artifacts (2.8s)
✓ Fetching channels (0.3s)
✓ Promoting patch to stable (0.3s)

✅ Published Patch 1!

The tool detected that we added a new Material icon that won't be included in the patch. For our demonstration we choose to continue anyways.

7. Checking the Shorebird console we can see our first patch has been added.

8. Restarting the app to download and apply the patch. After restarting the app we can confirm that the new changes have been remotely updated successfully.

Note that the newly added button's icon is not showing. This is as expected since the Material icon asset is missing.

Limitations and considerations

• Shorebird can only update Dart code. So native iOS (Objective-C/Swift) and Android (Java/Kotlin) code updates are not supported. For any native code changes like plugins we are still required to prepare a new release build as normal.

• Currently assets like fonts or images are not supported but are planned to be added. Also some care needs to be considered when changing icons. Since icon assets are tree-shaken when the app is built, you can't patch an app to use a new icon that is not already used somewhere in your app.

• For now Shorebird only supports Flutter mobile apps (iOS/Android). But Flutter desktop app support is also planned to be added later.

• Shorebird is not a replacement for existing app stores or other side loading solutions. It is only a tool for making code updates more convenient. You still need to have the app released to your users through an app store or other distribution method.

• Since new updates need to be downloaded from the Shorebird backend, apps that are exclusively offline are not supported. In fact the shorebird init command will add the required internet permissions to the Android manifest if it's missing.

• Flutter versions cannot be updated through Shorebird. For example an app released using Flutter 3.24 cannot be patched to use another version of Flutter. Conversely, since Shorebird is using a customized version of Flutter, there might be a small delay until a new Flutter version has been prepared for Shorebird.

• Both App Store and Play Console have both technical and policy rules that limit what a remote update is allowed to contain. Shorebird is built to comply with the technical limitations, but for policy rules it's up to the developers to make sure they are not broken. For example we are not allowed to "engage in deceptive behavior" via pushed updates. Like changing the original purpose of the app. Failing to follow the content policies can result in both your Shorebird account and developer account being terminated. More details can be found at the Shorebird’s Q&A here.

For more information I recommend checking out the official documentation and the helpful Discord server.

Problems We Saw

Football fans often engage in discussions on “what-if” scenarios. They want to explore different scenarios in their heads, talk about them on social media, and even compete with each other. Apart from that, they cherish the memory of their favorite team winning the trophy, feel heartbroken if their team didn’t make it, and ponder what went wrong. These highly emotional experiences can be enhanced in a way that introduces healthy challenges and keeps fans engaged for years to come.

We feel that, as fans, we would like to have a more immersive experience while watching the game and engage in meaningful conversations with our friends. These open up a new social dimension to the sports we enjoy. Being able to explore the “what-if” scenario in a novel way is always enjoyable and worth paying for. We believe that it will be particularly interesting to see these interactions between fans, which will drive engagement to the roof.

Solutions We Found

We are introducing UCL Max, a companion app concept for fans where not only will they be able to watch the game but also explore the “what-if” scenarios. Fans can sign up for a subscription to watch the game, get instant highlights and advanced match facts, interact with AI-powered simulations in AltPlay, and compete in the global leaderboard. Let’s dive into more details to see how this works.

AltPlay

We use cutting-edge machine learning models to predict when a crucial event happens within a live game. As soon as we get that game footage from the cameras, we can determine the relative positions of the players and various metrics from those moments. All of these data are fed into a pre-trained machine learning model, which can be used to predict alternative behaviors of any player in any moment and position.

We have identified the following parameters to train our models:

• Timestamp: represents a single frame of the video.
• Object Type: distinguishes between players and the ball.
• Object ID: unique identifier for each player.
• X-Coordinate, Y-Coordinate: player or ball position on the field relative to the field.
• Speed: the instantaneous speed of the object.
• Heading: direction of movement in degrees.
• Ball Possession: Indicates the player ID with the ball (if any).
• Pass Origin: The player ID of the passer.
• Pass Target: The player ID of the intended receiver.
• Pass Type: Basic classification of the pass.
• Action: Placeholder for basic actions like shots, tackles, etc.
• Action Type: A new column to categorize the actions more specifically: Shot, Tackle, Dribble, Foul
• Shot Type: Ground, volley, header, chip, free kick, etc.
• Shot Direction: Angle relative to the goal or general direction (left, right, center)
• Shot Outcome: Goal, saved, missed, blocked
• Tackle Location: General pitch area (right side, midfield, penalty area, etc.)
• Tackle Outcome: Successful (won possession), unsuccessful, foul

We use AWS Lambda, Fargate, and DynamoDB to process the live match events as well as the previous match events. Data aggregations from these services are sent into the Amazon Sagemaker pipeline to generate inferences and train our existing models. We use AWS Lambda to pre-process the video data and perform calculations on the inferred data. AWS Rekognition is used to determine various events in the game, such as passes, tackles, goals, etc.

Here’s a diagram of the AWS stack:

Using this data, we can predict with confidence when a player makes a successful pass or releases the ball when under pressure. So we incorporate all of this information into a simulation that can be played with each turn based on the actual game. Here’s a formula to determine the pressed player:

How It Works

Users of the UCL Max app can watch the live broadcast of the games and access the moments of the match that can be simulated. When they initiate a simulation, they are presented with a layout of the initial conditions of that moment. The real match path is already highlighted, and the users have the option to play an alternate game. Tapping on the player's position will reveal a possible path that the player can take, and based on the target player’s statistics, we simulate what’s going to happen.

After every turn, the positions of the players will be updated based on the ML inference, and the user will be able to run the next step of the simulation. Once completed, the user will be able to finalize their simulation and play it back. Based on events, for example, a goal scored, if the user can simulate a goal event on an alternate path, then they will be awarded points. This simulation can be shared on social media outlets.

The more points an individual collects, the higher up they go on the leaderboard. We can decide the winner of the leaderboard after a season and offer prizes in collaboration with the sponsors.

Technologies We Used

Here’s a list of the technologies we used to create our simulation pipeline:

• AWS Lambda - Data preprocessing, mathematical calculations for events such as player pressure, goal probability, sprint speed, duration, etc.
• AWS Fargate - Serverless cluster computing.
• AWS API Gateway - Provide access points to ingest data and inference output.
• AWS DocumentDB - High-performance JSON database to store parameter data.
• AWS DynamoDB - Database to store various data
• AWS S3 - Model archival, JSON data archival, caching.
• Amazon SageMaker - AI and ML pipeline.
• AWS Rekognition - Computer vision technologies, video analysis, and image recognition.
• AWS Amplify - Web and mobile app solution, access to other AWS services and streaming.
• AWS SQS - Messaging service between other AWS services.

Why It’s Unique

Our AI-powered AltPlay simulation explores the alternate reality fans often crave. Also, our solution uniquely enhances the user experience and long-term engagement with sports. We believe this novel technology, social aspects, and user retention make our solution innovative.

The main target of our solution is the fans who enjoy sports and are very passionate. Also, the amount and scope of the data we designed to process can be repurposed to enhance team management and player training improvement as well.

We found that the services we are offering to our fans are unique, as we provide a way to watch the game and interact with the highlights in a completely different way. With the emergence of AR/VR platforms, we think that our solution is particularly suited to enhance the experience of watching games and discussing them with friends.

How It Impacts Us

Users will be able to satisfy their curiosity about how they feel about the game and participate in healthy challenges among friends. This will significantly improve the sports experience and add a new dimension to social aspects. Users will get to learn more about their favorite teams and how the players work together. We believe that this will be beneficial for the fans of the sport.

Scalability and Business Opportunities

We use a myriad of AWS services to ensure that our product remains robust and scalable from a technical perspective. Since the football fanbase is one of the largest in the world and spans across continents, we have a potential userbase that is global. Official live broadcasts of football matches are already paid for. By subscribing to UCL Max, users will be able to enjoy the live broadcast from anywhere. We believe that fans will pay the right amount of money for live broadcasts and additional perks and features that open up a new dimension of sports. Leaderboard winners of previous seasons may be rewarded with match tickets for the next season and may receive merchandise promoting the sponsors. This also creates scope for targeted advertising for users. A monthly subscription or one-time season pass can be possible payment strategies.

Our platform and ideas are not limited to football. We can implement similar strategies to other sports such as American Football, basketball, etc. Those leagues have strong fanbases; if they are persuaded to realize the value of our simulations, we could even host betting tournaments where it’s legal. The possibilities are endless.

Challenges We Faced

Our primary challenge was to figure out how we could satisfy the questions of what-ifs for our fans and, in turn, make that competitive. Also, we needed something economically viable. When deciding on how our simulation will work, we found that if we allow the user to simulate a whole match, this is going to be a completely different thing, like a video game. Then we thought instead of simulating the whole game, we would just play the highlights and let them simulate the specific scenarios. It felt like the right direction since we don’t talk about all the moments in a game but only some interesting moments. The next step was to determine what our simulation would look like inside the app. After a lot of brainstorming sessions, we found our inspiration and designed it.

Final Words

MonstarHacks 2024 presented some real challenges this year. We were amazed to learn about the progress in sports that AWS had unlocked. We were so influenced by the BundesLiga Match Facts that we designed our solution based on those ideas, and AWS provided the perfect tools to aid in our solution. Overall, it was really fun and engaging, and we learned a lot from our experience and team efforts. We are looking forward to the next MonstarHacks. Until then, best wishes to all.

Article Photo by Image by master1305 on Freepik

Together with Amazon Web Services, the world’s most comprehensive and broadly adopted cloud, this event allowed us to explore over 200 fully featured services from data centers globally including AWS’s Generative AI and Machine Learning technologies.

The Team

![Team Photos](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/TheTeam.webp

As a team of 2 architects, Ross and Graham were like a band with 2 bassists. They were cool, but they needed some drums.

So they drafted in long-time collaborator and drinking buddy, Ian Gifford from Atom Bank (UK FinTech), who took their vision and ran with it in Figma for mockups and ident.

Drum & Bass.

The Problem

We decided to focus on Performance Analytics (Athlete View), looking at holistic and historic sports data to try and consider insights that could improve athletic quality and wellbeing.

• Develop solutions centred around athlete data including tracking for performance, health, and safety.
• Employ machine learning to predict in-game strategies and prevent injuries.
• Generate real-time insights that can influence game-day decisions and strategies.
• Enhance scouting and recruitment processes.
• Predictive Analytics and Insights

As we’ll see, this proved more difficult in reality… The business case proposed turns this hurdle into an opportunity.

Staying close to home

All members of Team 5 grew up within a few miles of Newcastle United Football Club and are all lifelong fans.

![NUFC Crest](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/NUFC.webp

For us, Newcastle as a place can’t be easily separated from Newcastle the club. It’s in our DNA, so choosing football was easy for us.

The 2023/24 season for Newcastle has been a nightmare due to the amount of injuries suffered by players.

This injury problem can probably be directly related to overachieving in the 2022/2023 season and having a European competition to compete in with a limited squad. Our success has been our demise.

![Data table from BBC Sport showing NUFC incurring the most injuries and days lost this season](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/NUFCInjuries.webp

We thought - is there a way to utilise publicly available data along with historical injury data to enable a way to help predict the likelihood of an injury reoccurring?

Is there a correlation between different sources of data available?

• Fixtures & Results
• Travel Distances
• Weather
• Injury history

We decided to explore this first.

Public Data

Data on fixtures, location, players and injuries is available along with other sources such as Weather which could be ingested - however this data is difficult to obtain in bulk.

In addition, it turns out that injury data publicly available is under-reported compared to underlying data - “The overall injury cumulative incidence of 0.63 injuries per player-season was substantially lower than in reports using other data collecting methods (ranging from 1.5 to 2.5 injuries per player-season)”.

![Publicly available injury history from transfermarkt website](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/InjuryHistory.webp

Predictive modelling of football injuries is possible given sufficient data, however the availability of the data is limited given most clubs who invest heavily in Sports Science have commercial reasons to keep that data private.

Manchester City Academy’s Head of Sports Science, for example, oversees the data collection of nine teams and over 200 players, from Under-9 through to the Under-23s.

Data Ownership

So the question becomes - who owns the data? UK Players’ performance data could be considered to be personal data, which requires players’ explicit consent for clubs and leagues to use. The argument is that this consent may not yet been given, and that ownership by clubs and leagues has just been assumed.

![Plot of Goals per Game vs Shot Accuracy in Europe's Top 5 Leagues](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/ShotAccuracy.webp

It’s not yet something clubs or leagues have universally tried to claim ownership of, with a view to commercialising, and this is something which is currently being discussed, however in the US it’s different, as both the national American Football (NFL) and Basketball (NBA) leagues have a level of ownership over this data, more so in the NFL.

Money is shared, but not data or insights

While top clubs have the finances to invest heavily into Sports Science, this investment is only available down to tier 4 or 5 of the EFL Pyramid and although TV Revenue does flow down, although only so far, the insights available from Sports Science does not.

![Distribution of TV Revenue by tier in the English Football LEague](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/TVRevenue.webp

This seems like a waste of potential, and likely hampers efforts to grow better footballers at lower levels in the English football community.

Non-commercial benefits of sharing data

Given the concerns mentioned around data ownership and commercialisation - how else could the sharing of data benefit the members of the Football Association themselves directly or indirectly?

The French National Agency for Sport, together with the National Institute for Sport, Experience and Performance (INSEP) and the General Directorate for Sport, have developed the Sport Data Hub - FFS.

![French Football Federation Logo](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/FFF.webp

The project was born in 2020 with the idea of boosting the individual and collective performance of French sport in the run-up to the Olympic Games in Paris 2024. It consists of the creation of a collaborative tool for all those involved in the sports movement, (federations, athletes, coaches, technical teams, institutions and researchers), to share data that allows for aggregate comparative analysis at national and international level.

![French Olympic Committee Logo](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/Olympic.webp

Who would benefit from the sharing of data, or at least these insights from data, across the entire English Football League and who would want this?

Who wants this?

Below are Mission statements from both the FA (Football Association) and the PFA (Professional Footballer Association). The FA governs the UK National Game with the PFA providing Union and wellbeing support for professional players in the UK.

Promoting health and wellbeing, and the power of teamwork, with our new strategy for 2020- 2024, we have a plan for all. Positively impacting every community across the country. Everyone can win if we build on the progress made over the previous four seasons. We’ve moved forward in every area, modernising our organisation to serve a game for all. We have a great platform to build on. We will keep pushing forward. It’s time to deliver real change.

The Football Association (FA)

We believe in the unifying power of football in society, and are committed to empowering footballers to recognise their value as people, not just players. We protect players' rights, represent their views and provide support through a wide variety of educational, financial and wellbeing services.

The Professional Footballer Association (PFA)

FA Vision

From team managers and club secretaries through to players and parents, Whole Game is for the whole of grassroots football.

Whatever your role in the game and whatever you do, Whole Game can help you to do it faster and more effectively.

You’ll find club data – such as player contacts, qualifications, discipline records and so much more all in one place, which is great news for managers and club secretaries when it comes to club admin and monitoring player availability.

You only have to input a player’s details once – and the system fully integrates with both Full-Time and the Matchday app to make access and management of information simple.

![The FA Vision](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/Vision.webp

The problem revealed

At this point, we realised the problem to be solved wasn’t looking at the correlations between different data sets. This could come later.

The actual problem was how to coordinate an ecosystem of different stakeholders, with different levels of access to data, to ensure that insights from the most privileged could benefit the least privileged.

How could Professional UK Football and Grass Roots football share insights to benefit the national game in a long-term and sustainable way.

![Pivot](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/Pivot.webp

Pivoting Early, but read on for our exploration of this problem and a proposition that supports our solution.

The Solution

![AgoraMetrics Logo](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/Logo.webp

Measure, Learn, Win. Together

At AgoraMetrics^*, our mission is to democratise the use of advanced data analytics and machine learning in football, making cutting-edge sports science accessible to clubs of all sizes and resources all the way down to grass roots level.

We're committed to levelling the playing field by providing smaller clubs with the insights and tools previously available only to top-tier teams.

By joining our ecosystem of data sharing, clubs can enhance player performance, prevent injuries, and optimise team and individual management strategies, all while supporting the growth and competitiveness of football at every level.

_{* Inspired by the Greek "agora", the central public space in ancient Greek city-states, symbolising the gathering and sharing of ideas and now, sports data analytics.}

The AgoraMetrics Proposition

![The AgoraMetrics proposal](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/Proposition.webp

The AgoraMetrics proposition is aimed initially at the UK Football Association, the body that holds accountability for governance of both Professional and Grass Roots football in England.

The benefits outlined in this section detail granular positive outcomes for English football at all levels, but at a high level our mission is to Win an International competition by 2030.

We would expect returns on investment to include:

• Positive outcomes at Grass Roots level between years 1-3
• Positive Professional Football outcomes from years 2-5
• Positive England Team outcomes within 4-7 years
• A self-sustaining AgoraMetrics Service within years 1-4

![World Cup](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/WorldCup.webp

These outcomes assume 4 considerations to achieve our mission:-

• AgoraMetrics is funded by the FA from 2024-2030
• AgoraMetrics is marketed to and adopted by the Grass Roots football community extensively to drive adoption
• AgoraMetrics quantitatively improves player quality at every level of the UK game
• AgoraMetrics grass roots players take their career to professional level enabling progression ( and outcomes measurement ).

![AgoraMetrics Stakeholders](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/Stakeholders.webp

Solution Design

Solution Landscape

![Solution Landscape](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/SolutionLandscape.webp

While data would initially flow from professional clubs down to grass roots level, data would be fed back into the system to benefit teams and players at all levels.

Solution Detail

![Solution Detail](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/SolutionDetail.webp

The solution itself would comprise several Data Science models produced for the benefit of the community along with an app to onboard coaches, players and data which would feed back into the data layer.

Data Mesh and ML Factory

![Data Mesh and ML Factory](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/DataMesh.webp

Several Amazon Web Services would be utilised to provide the back end Data mesh and Machine Learning models which would be exposed via secure APIs for use by the client application.

Serverless Architecture

![Serverless Architecture](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/Serverless.webp

A Serverless Architecture would be utilised to provide the Web Application layer to prioritise performance and cost effectiveness and allow scalability.

The Agorametrics Application

Design Influence

We were interested in using a colour scheme that resonated with players, coaches and anyone with a wider interest in football.

With this design principle in mind, we couldn’t think of a better way to start than to picture St. James’ Park (Newcastle United’s Stadium) on a Sunny Day.

![Colour Scheme](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/ColourScheme.webp

This image was used to derive a palette that could be included in a design system and includes tones for components such as alerts (orange / yellow from stewards jackets) as well as primary ident colours taken from the pitch and the sky.

Our logo is intended to convey the notion of team sport in general, with data points composing the “ball”. This ensures we’re not locked into “Football” as a sport.

User Features

We included a finite set of App Features to discuss for the prototype. These allowed us to explore both the benefits of possible insights the Machine Learning and Data science might hunt for, and also how players and coaches might use these features.

![User Features](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/Features.webp

User Interface Design

![Green user interface](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/UserInterfaceGreen.webp

![Blue user interface](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/UserInterfaceBlue.webp

Financials & opportunity

![Financial forecast](/assets/img/articles/2024-05-23-Agorametrics-Measure-Learn-Win-Together/Financials.webp

The AgoraMetrics product aims to provide as wide a range of returns on investment as possible.

In this model, “Data Value” could be compared with the Social Media model of “You are the product” from an end user perspective.

In AgoraMetrics case however, rather than monetising this data for corporate profit, we aim to drive financial gains back into Grass Roots initiatives ensuring Social Return on Investment which will contribute directly to the FA’s Mission.

The decision to pitch this to the FA has a leverage value, in so far as the FA governs Pro Clubs in the UK, and controls the throughput of revenue streams such as TV Rights. We would therefore expect that the FA organisation has the necessary ability to promote and project this initiative.

Future Opportunities

Conclusion - The Hackathon Experience

While we both entered the weekend hoping to create a tangible working prototype utilising some cool functionality from Amazon Web Services - the time spent trying to access and retrieve data for our initial idea meant we had to pivot to a more conceptual approach.

We realised that the main problem with our idea was not the data itself - this exists and is studied in depth, but only by those who have the funds and resources to do so. Top clubs have entire teams dedicated to Data Science and utilising the latest and greatest in sports analytics - but, for good reasons, this data is held onto and not shared. However, the insights and models which can be obtained from this data can benefit lesser teams - and in return, more data can be utilised by the system to benefit all.

It is this democratisation of data, insights and models which we explored and feel fits well into the visions of both the FA and PFA and could potentially be implemented across many different sports communities.

We both look forward to the next Hack!

Article Header from Ross Davidson, Article Photos from NUFC, transfermarkt, French Olympic Committee, French Football Federation

In today’s day and age, there is hardly any platform where parents can get guidance to ensure their children are equipped to pick up any sports and get better at it. Be it recreational or professional, we don’t see this often. Moreover, we have seen athletes drop out of their game because of a lack of training and guidance. The cost involved and the difficulty in finding the best trainers in their localities further complicate the situation.

We, team Ready Player Three, wanted to address this problem in this year's MonstarHacks. Let us tell you readers how to approach the hackathon problem and how we tried to solve the issue.

![AI Generated Image](/assets/img/articles/2024-05-23-AthletePro-Empowering-the-next-generation-of-champions/abstract-intro.webp

MonstarHacks partnered with Amazon Web Services (AWS), to provide our teams with challenges in 3 solution areas: Fan360, Performance Analytics, and Smart Venues. Our team made out of Saad Bin Amjad, Tanveer Hassan, and Faisal Islam, choose the Performance Analytics ("Athlete View") category aiming to help athletes and their families navigate their sports journey more effectively. Leveraging AWS technology, especially Generative AI, we aimed to build a solution that would make a tangible difference in the sports world.

We named our application AthletePro, which plans to revolutionize the sports industry with an AI-driven platform that provides hyper-personalized roadmaps for athletes and parents from recreational to elite levels, empowering the next inclusive generation of champions.

Solution Overview

AthletePro offers a comprehensive solution to empower athletes and their families. Even before signing up for the platform, parents are given a tour of the app features and what it can offer them and their children in terms of sports. This includes sneak peeks at coaches, trainers, events, and the financial aspects involved in getting the athletes up to speed. There are curated FAQs that enable parents to make the right call in terms of exploration or being onboarded to the application.

Then we ensure that parents can sign up easily. We first focused on Personalized Onboarding. This feature enables parents to create profiles for themselves and their children by answering a series of questions. The questions range from their location, age, sports they want to explore, end goals if any, and budget. This helps the application tailor recommendations accordingly, as this information becomes part of the knowledge base going forward.

Parents of special needs children can provide additional information to receive more personalized recommendations.

Similar to other ed-tech platforms but specifically for sports, we showcase tailored lessons and suggest sporting activities. To support this feature, we introduced the onboarding of coaches, trainers, and organizers into our CMS-styled platform powered by Strapi. We ensure it is easily accessible, allowing users to enter the system with their certifications and facilities, which will also become part of the knowledge base going forward.

Businesses in the sports industry, such as gyms, training facilities, and equipment suppliers, can easily connect with athletes and parents through AthletePro. The bigger plan for this feature is that when governments, sports organizations, and universities are onboarded, they can sponsor athletes and their families so they can pursue their sporting journey.

Additionally, we understand that when AI generates any roadmap for vulnerable groups in society, such as children, there should be accountability and transparency. It is crucial that we provide accurate roadmaps. Therefore, we have proposed and showcased a notification system for connected coaches and trainers to first approve the content before presenting the roadmap to parents or children.

We also recognize that not everything can be taught or practiced indoors. The courses are designed so that AthletePro provides exclusive content and perks unlocked at specific stadiums or training facilities, enhancing engagement. This means the application can detect your location and then unlock the corresponding training guidance or instructions.

We understand that not everything can be taught or practiced indoors. So the courses are designed so that AthletePro provides exclusive content and perks unlocked at specific stadiums or training facilities, enhancing engagement. This means the application can detect your location and then unlock the corresponding training guidance or instructions.

![Mock Design](/assets/img/articles/2024-05-23-AthletePro-Empowering-the-next-generation-of-champions/athele-pro-trainer.webp

Target Users and Revenue Model

AthletePro caters to a diverse range of users across the sports ecosystem, including athletes, parents/guardians, coaches/trainers, government sports agencies, universities and colleges, and local businesses.

![Revenue Generation](/assets/img/articles/2024-05-23-AthletePro-Empowering-the-next-generation-of-champions/business.webp

To address the needs of these diverse stakeholders, we have developed a solid revenue model for the app based on subscription-based revenue, partnerships, and sponsorships. Additionally, we engage in ethical data monetization through partnerships with research institutions, sports analytics firms, and market research companies. We believe AthletePro has the potential to expand into new markets domestically and internationally by adapting its platform to cater to different sports, cultures, and regions.

Engineering Approach: Seamlessly Integrating the RAG Framework for Contextual Guidance

We emphasized heavily how easy this app should be for parents and athletes alike. We understand that with any GenAI-backed application, most of the time, we resort to chatbots. However, chatbots won't work here, as not everyone is suited to ask the right questions at the right time. Instead, we've innovatively integrated the RAG framework into the app's architecture, embedding contextual guidance seamlessly within the app journey itself. This user-centric approach not only enhances the overall user experience but also reinforces AthletePro's commitment to accessibility and inclusivity in sports technology.

Leveraging the retrieved data from the knowledge bank, AthletePro dynamically generates system messages that resonate with the user's current context and stage in their athletic journey. These messages are strategically woven into the app journey, serving as subtle cues that guide users towards relevant actions and decisions without overwhelming them with explicit prompts or queries.

The final piece of the puzzle involves generating prompts and recommendations based on the contextual cues provided by the system messages. AthletePro's backend formulates queries for our language model systems (LLMs), such as Anthropic's Claude in Amazon Bedrock, drawing upon the user's profile, preferences, and journey stage. These queries are designed to extract actionable insights and deliver tailored guidance that aligns with the user's unique circumstances.

![AI Flow](/assets/img/articles/2024-05-23-AthletePro-Empowering-the-next-generation-of-champions/ai-flow.webp

Core Technologies

• Strapi CMS
• Amazon Bedrock
• Amazon EC2
• Amazon S3
• Amazon RDS
• Docker
• Github
• Prototyping Tools

RAG Framework Utilization

![User Flow](/assets/img/articles/2024-05-23-AthletePro-Empowering-the-next-generation-of-champions/user-flow.webp

In the user flow diagram, the process begins with Sarah and her mother signing up for AthletePro. Sarah shares her interest in volleyball along with her preferences and any constraints. AthletePro’s system then uses this information to create a personalized roadmap recommendation. This recommendation is sent to local trainers, who review and adjust it as needed before returning it to Sarah’s mother, completing the feedback loop. This user flow demonstrates how AthletePro seamlessly integrates user data, AI-generated recommendations, and real-time communication through its backend architecture.

Challenges Faced

One significant challenge we encountered during the hackathon was defining the scope of our idea. AthletePro had the potential to benefit various stakeholders, including athletes, parents, coaches, government agencies, and businesses. We wanted to showcase the platform's capabilities while ensuring clarity and focus. Ultimately, we decided to center our solution around the role of parents, recognizing their pivotal role in supporting young athletes.

To overcome this challenge, we proactively engaged with parents, conducting interviews to gain insights into their perspectives, motivations, and pain points regarding their children's athletic pursuits. By listening to their experiences, we were able to refine our solution and tailor it to address the specific needs of parents navigating their children's sports journey.

Additionally, we faced difficulties in accessing Anthropic's Claude2 model, a crucial subscription service, due to a waiting period and regional restrictions. To mitigate this issue, we improvised by using Amazon's Titan instead of our own knowledge base of parents and athletes. AWS Bedrock only supports Anthropic's models for using knowledge bases for now. Although we set it up, we could not test it due to this limitation.

Conclusion

AthletePro represents a groundbreaking innovation in the sports industry, offering personalized guidance, resources, and support to athletes and parents of all backgrounds and abilities. By leveraging technology and a commitment to inclusivity, AthletePro aims to empower athletes to reach their full potential and foster a more vibrant sports community. As we continue to iterate and expand our platform, we look forward to revolutionizing the sports industry and creating a brighter future for athletes everywhere.

Article Photo by Seth Kane

Problem Statement

Sports fans with visual and hearing impairments face challenges in enjoying game events in detail. Visually impaired fans face difficulties in understanding in-game play, and on the other hand, hearing-impaired fans don’t get thrilled by the live commentary. These difficulties not only take out the excitement but also makes them feel that they are not part of the game.

The Solution

SportsBuddy is a revolutionary platform that opens a new door for sports enjoyment for individuals with sensory disabilities, especially with visual or hearing impairments. SportsBuddy offers detailed commentary for blind individuals and sign language features for deaf individuals, ensuring an immersive sports experience for all.

![SportsBuddy Mobile App Flow Chart](/assets/img/articles/2024-05-22-Revolutionizing-Sports-Enjoyment-for-Sensory-Disabled-Individuals-in-MonstarHacks-2024/sportsbuddy-main-promo.webp

From our research, we found the challenges faced by individuals with sensory disabilities when accessing sports commentary. This involved researching the potential users, and their preferences, and identifying key pain points.

![SportsBuddy Mobile App Screens](/assets/img/articles/2024-05-22-Revolutionizing-Sports-Enjoyment-for-Sensory-Disabled-Individuals-in-MonstarHacks-2024/sportsbuddy-all-screens.webp

By using the power of AWS, we used a range of cutting-edge technologies to build our solution.

SportsBuddy Flow Chart

![SportsBuddy Flow Chart](/assets/img/articles/2024-05-22-Revolutionizing-Sports-Enjoyment-for-Sensory-Disabled-Individuals-in-MonstarHacks-2024/sportsbuddy-diagram-flow.webp

How does SportsBuddy help visually impaired fans?

SportsBuddy provides a detailed narration of sports events, allowing visually impaired fans to follow the in-game action on the playground through audio commentary which includes descriptions of player movements, game strategies, and key moments, and even any fun moments happening on the fields.

It also offers audio descriptions of visual feeds such as player shirts, team formation, etc., helping visually impaired fans visualize the game in their minds and stay fully connected to the game.

Smart Switch

SportsBuddy uses descriptive onscreen statistics, including leaderboards, to surpass the limited coverage offered by the regular standard commentary system. Users can disable this feature if they wish to follow the match commentary only. However, enabling this feature ensures that the audience remains informed of all the key moments during the match. If an exciting event occurs, such as a goal, foul, or a red card, SportsBuddy’s smart switch feature seamlessly transitions to that event and resumes the statistics analysis afterward.

![Architectural diagram for generating detailed commentary](/assets/img/articles/2024-05-22-Revolutionizing-Sports-Enjoyment-for-Sensory-Disabled-Individuals-in-MonstarHacks-2024/sportsbuddy-architectural-diagram-blind-tailored-commentary.webp

How does SportsBuddy help auditory-impaired fans?

SportsBuddy offers AI-based sign language commentary for hearing-impaired fans. Those fans will follow the game action through visual interpretation, which includes sign language interpretation of game commentary, match analysis, and key moments, to ensure an immersive sports experience.

Apart from the sign language commentary, SportsBuddy offers text-based commentary for those who prefer to follow the game through written match events with the descriptive text of game events, player performances, match analysis, etc.

It will show notifications on any important match events during the match time, for example, goals, fouls, cards, penalties, halftime, etc. It ensures that a hearing-impaired fan stays informed of all the key moments of a game.

The sign language avatar[3] is movable, so users can drag the avatar to their preferred place for optimal viewing without missing any in-game key moments. Also, users have the ability to disable the sign language avatar if they are not interested. We are planning on localizing the sign language feature to target a broader user base.

![Architectural diagram for generating sign language commentary](/assets/img/articles/2024-05-22-Revolutionizing-Sports-Enjoyment-for-Sensory-Disabled-Individuals-in-MonstarHacks-2024/sportsbuddy-architectural-diagram-deaf-sign-language.webp

Technology Used

To develop our solution we have used several technologies including Amazon Services. We’ve listed them below.

Flutter: Flutter is an open-source framework by Google for building beautiful, natively compiled, multi-platform applications from a single codebase. We’ve developed our application SportsBuddy using the Flutter framework for Android and iOS mobile devices.

Amazon S3: Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. We’ve used Amazon S3 for storing user data.

DynamoDB: Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. We’ve used DynamoDB for storing the session memory, statistics feed, sign language avatar, etc.

AWS Lambda: AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources, making it the fastest way to turn an idea into a modern, production, serverless application. We’ve used this cost-efficient service mostly for the backend processing.

Amazon Rekognition: Amazon Rekognition is a cloud-based image and video analysis service that makes it easy to add advanced computer vision capabilities to any application. The service is powered by proven deep-learning technology. We’ve used Amazon Rekognition and other services like Kafka for real-time data processing.

Amazon Polly: Amazon Polly is a cloud service that converts text into lifelike speech. It supports multiple languages and includes a variety of lifelike voices. We’ve used Amazon Polly for generating voice commentary.

Amazon Transcribe: Amazon Transcribe is an automatic speech recognition service that uses machine learning models to convert audio to text. You can use Amazon Transcribe as a standalone transcription service or add speech-to-text capabilities to any application. We’ve used Amazon Transcribe for extracting text from live commentary.

Amazon Bedrock: Amazon Bedrock is a fully managed service that makes high-performing foundation models (FMs) from leading AI startups and Amazon available for your use through a unified API. You can choose from a wide range of foundation models to find the model that is best suited for your use case. We’ve leveraged the power of Amazon Bedrock to facilitate our solution better.

Amazon SageMaker: Amazon SageMaker is a fully managed machine learning (ML) service. With SageMaker, data scientists and developers can quickly and confidently build, train, and deploy ML models into a production-ready hosted environment. We’ve leveraged the power of Amazon SageMaker to facilitate our solution better.

Impact & benefits

Impact

SportsBuddy has a great impact on the sports community and the society. By addressing the needs of the blind and deaf community, SportsBuddy promotes inclusivity and equality in the sports community. With the help of SportsBuddy, we can help blind and deaf individuals enjoy all types of sports to the fullest. It will uplift their moral spirits and also will provide them with a sense of belonging. They won’t feel left out anymore in the sports community. Our solution will help raise social awareness regarding the problems that blind or deaf individuals face in their day-to-day lives. It will inspire other communities as well to focus on the needs of people with disabilities.

Benefits

Our solution is designed to benefit a wide range of stakeholders connected to sports, including fans, athletes, and sports organizations.

Fans

Our solution boosts the experience for fans to enjoy sports, particularly those with visual and hearing impairments. By providing accessible features such as real-time captioning, audio descriptions, sign language interpretation, etc, we ensure that all fans can enjoy sporting events to the fullest. This inclusivity helps grow a sense of belonging and engagement among fans, regardless of their abilities, enriching their overall experience and satisfaction.

Athletes

For athletes, our solution provides accessibility by ensuring the experience is the same and accessible to all fans, regardless of their sensory capabilities. By reaching a wider range of audience, athletes can amplify their impact and inspire individuals from diverse backgrounds. SportsBuddy provides athletes with chances for greater exposure and recognition, contributing to their personal and professional growth within the sports industry.

Sports Organizations

Sports organizations can also benefit from our solution through increased audience engagement, brand loyalty, and revenue generation. By prioritizing accessibility and inclusivity, organizations can showcase their commitment to social responsibility and diversity, growing their reputation and appealing to fans and sponsors alike. Moreover, our solution may open up new revenue streams through subscription services, pay-per-view options, and targeted advertising, driving financial sustainability and growth for sports organizations. Overall, our solution positively impacts users by promoting inclusivity, accessibility, and engagement within the sports community. By breaking down barriers and embracing diversity, we envision a future where sports experiences are amplified for all individuals, leading to positive changes in their overall enjoyment, participation, and operations within the sports industry.

Commercial Viability

In a world where technology is evolving every day, ensuring inclusivity for all individuals, including those with visual and hearing impairments, is crucial. With approximately 253 million[5] visually impaired people and over 430 million[6] people with disabling hearing loss all over the world, there exists a significant market demand for products that address their needs.

![Recent statistics showing 253 million visually impaired and 430 million hearing impaired people](/assets/img/articles/2024-05-22-Revolutionizing-Sports-Enjoyment-for-Sensory-Disabled-Individuals-in-MonstarHacks-2024/sportsbuddy-recent-stats.webp

Market Opportunity

This huge number represents more than just statistics; it points to an immense market opportunity waiting to be captured. Despite the growing number of individuals with visual and hearing impairments, this market remains almost uncaptured, presenting a huge opportunity for innovation and impact. Projections suggest that by 2050, the number of people with disabling hearing loss worldwide will surpass 711 million[7] and vision impairments by 588 million[5].

Sports Engagement

Sports is considered a powerful platform for social engagement, and it's no different for the blind and deaf communities. No matter whether they are participants or spectators, a significant part of these communities actively engage in sports. But here's room to enhance their experience further. Imagine the impact of AI-based detailed commentary for blind audiences or real-time sign language translation for deaf audiences at sporting events. By providing these specialized experiences, we can increase their engagement and boost their enjoyment of sports, contributing to a more inclusive sporting environment for everyone.

Forging Collaborative Partnerships

In our solution for visually and hearing-impaired individuals, we believe that collaboration is the key. Many government and non-governmental organizations are eager to support initiatives that can benefit these communities. By partnering with these organizations, we can have access to resources, expertise, and funding opportunities that can significantly uplift our efforts. These partnerships not only increase our reach but also demonstrate a collective commitment to improving the lives of individuals with visual and hearing impairments. We can take meaningful steps towards a more inclusive and accessible future.

Challenges and Learnings

Our team set out on the hackathon with the dual goals of developing a solution that would be profitable and have a significant social impact. But this admirable goal wasn't without its difficulties, from figuring out the technological environment to understanding the business aspects of our solution.

Challenge 1: Ideation and Social Impact

During the ideation phase, we targeted to find a hack that has both commercial viability and substantial social impact. We brainstormed, considering real-life problems around us.

Solution: Focusing on accessibility in sports enjoyment, we did some collaborative sessions and research and came up with this idea. Serving a neglected audience using our solution inspired us the most.

Challenge 2: Technological Feasibility

At first, we were a bit unsure about the feasibility of our chosen technologies. We did some initial research and sorted it out with our mentor.

Solution: Our mentor, Iaroslav Ustinov, a solutions architect at AWS, guided us to choose the correct technologies. His expertise helped us assess the feasibility of our technologies.

Challenge 3: Commercialization Strategy

At first, we were a bit unaware of how can we make this solution more commercially appealing to the judges and investors.

Solution: Peter Abdelkodos guided us, introduced some terms, and helped us to rethink the commercial viability of the solution.

Conclusion

For our team, participating in the hackathon has been an enjoyable experience. We've overcome obstacles, welcomed creativity, and worked well together to develop a solution that has enormous potential to change society, from conception to implementation. Along with improving our technical proficiency, our journey has deepened our understanding of the value of collaboration and goal-driven creativity. Looking back on our hackathon experience, we're appreciative of the chance to make a difference and motivated to keep using technology to achieve important goals.

Resources

• [1]: Engage online sports fans with live event commentary using generative AI on Amazon Bedrock
• [2]: Leveraging AWS Bedrock to Revolutionize Live Sports Commentary
• [3]: HandTalk Avatar
• [4]: AWS re:Invent 2023 - How to build generative AI–powered American Sign Language avatars
• [5]: NIH - World blindness and visual impairment
• [6]: WHO - Deafness and Hearing Loss
• [7]: Projected number of people with disabling hearing loss worldwide in 2019, 2030, 2040, and 2050

Article Photo by Chaos Soccer Gear on Unsplash

The main drivers were the need to improve efficiency of staff to offset a reduction in resource, reduce average wait time for patients and replace existing disparate legacy systems which were lacking support.

The Monstarlab team conducted a discovery phase, laying the groundwork for a compelling business case to pilot the selected platform. Upon demonstrating an improvement over current technology, the decision was made to rollout Amazon Connect to the PTAC service.

Amazon Connect Features for Pilot Phase

Identified features to explore during the pilot phase were as follows:

Telephony

The incumbent telephony system provided no queue information or expected wait times to users and suffered from frequent system disconnects if no active agents were available.

Amazon Connect offers the ability to configure Comprehensive IVR with better call routing, handling and queuing. We were also able to enable callback requests along with a separate callback queue with opening hours to ensure callbacks were handled at low peak usage times and within normal operating hours.

Real-time and historical metrics dashboards and reports

Obtaining reports from the incumbent system took days or weeks and required manual mapping between separate systems which meant any insights may be up to a week old.

Amazon Connect offers real time and historical reports with data visualisation which provided the ability to manage staff scheduling and feedback in real time without manual intervention. The available metrics by default also provided additional insights which simply weren't possible before.

Contact Lens

In addition to the improved reporting, Contact Lens also offered real-time contact center analytics and quality management powered by natural language processing. This included sentiment analysis and the ability to categorise calls based on the content of the call without the need to manually review the call at a later date. These insights were only possible with significant manual resource investment previously which would only be available to resolve significant issues. By utilising contact Lens further, similar issues could be prevented or prioritised in real time.

Outcome of Pilot Phase

Through the delivery of Amazon Connect handling all phone lines and agents for a period of 3 weeks, the following improvements were realised:

1. Average waiting time reduced by 45%
2. Calls serviced per agent increased by 24%
3. Average contact talk time reduced by 5%

Along with drastically improved management information visibility and agent satisfaction it was decided to roll out Amazon Connect to the full live service to replace the existing telephony.

Next steps and Additional Amazon Connect Features

During the pilot phase, the following additional services provided by Amazon Connect were to be considered and discussed for future phases:

Omni-channel customer experience

The current service is only available during office hours by phone.

Amazon Connect Flows can be utilised to offer patients their channel of choice, while saving them time and effort. Telephony and Website Chat Widget is provided out of the box but integrations with other chat clients, (such as Messenger, Slack, Teams etc). SMS, Email can also be configured and utilised to provide out of hours support and signposting.

Amazon Lex

Limited Patient Data is provided "up-front" to the Agent handling the contact.

By utilising Amazon Lex, an ML Powered conversational engine, pre-eligibility questions could be provided to pre-populate case details for the agent so the beginning of a call isn’t spent gathering details. This will enable the agent to move straight to the eligibility assessment phase to provide a more streamlined service

Amazon Customer Profiles and Cases

The current service tracks customer cases in a separate system to that handling the telephony.

Utilising Amazon Connect Profiles and Cases would allow agents to track and manage customer issues including those which require multiple interactions, follow-up tasks, and teams in the contact center. Agents can document customer issues with all the relevant case details, such as date/time opened, issue summary, customer information, and status, in a single unified view.

Guides

The current eligibility assessment is completed in an in house system by an agent, supported by documentation in several formats along with team support via chat.

By implementing and utilising Step by Step Guides and Flows for the agents, they can be guided through the process during the call to populate the Amazon Connect case with the outcome.

Amazon Q

In addition to the Step by Step Guides, real time information and assistance based on the content of the contact can be supplied to agents from existing knowledge repositories.

API integrations and Tasks

API Integrations with 3rd party systems and APIs can be provided via Lambda functions within Flows, or the utilisation of Amazon Connect Tasks to automate agent tasks such as emails or bookings.

Forecasting and Scheduling

Real time forecasting, capacity planning, and scheduling capabilities can be enabled to efficiently forecast or plan resources automatically based on historical metrics.

Article Photos from Amazon

How we envisioned TALOS

We believe that by gamifying fan engagement, we can encourage and reward supporters for their unwavering support for their club. Through our platform, followers earn points for various interactions such as social media engagement, event attendance, and participation in club-related activities. These points act as currency, offering fans exciting prizes ranging from team jerseys and caps to the most desired tickets to the finals, as well as the chance to meet their favorite players which we think is the ultimate reward for a fan.

But our innovation doesn’t stop there. In today’s digital age, social media has become an integral part of the sports fans's lives. However, it’s not uncommon for this to turn toxic and spoil the spirit of the game. That’s where our solution steps in with its unique characteristic, it uses data from social platforms to identify and prevent toxic behaviour. By implementing a system that deducts points for toxic behavior, we aim to create a healthy and inclusive online community by biasing towards positive attitudes and encourage true sportsmanship among the followers.

For sports teams, understanding fan behavior and emotions is essential for improving their strategies both on and off the field. That’s why we created a robust dashboard that gives teams access to valuable insights gained from fan interactions. From monitoring engagement metrics to monitoring sentiment analytics, teams gain a holistic understanding of their fan base, allowing them to tweak their marketing efforts, improve fan experience, and deepen their relationships with their supporters.

Essentially, our solution is a win-win for all stakeholders. Followers are rewarded for their fandom and teams gain valuable insights of their fans and business.

Technical architecture insights

The architecture that supports our solution is serverless, which enables a pay-as-you-go service model. This approach will prove highly cost-effective in the long term as demand increases for our app. We utilize AWS Fargate to seamlessly deploy our three services. These services connect our app with social media platforms, extract data from social media posts, and securely store it in an S3 bucket. Additionally, AWS Step Functions orchestrate the smooth processing of steps for our AI/ML tools, ensuring efficient and reliable execution of tasks.

At the core of our architecture, we embrace the cutting-edge capabilities of three powerful AI/ML services offered by AWS. Firstly, AWS Personalizes categorizes sports fans into loyal and seasonal groups, allowing for tailored recommendations based on their preferences. Next, Amazon Comprehend analyzes social media posts to understand fan sentiments, entities, and key phrases, providing valuable insights into fan emotions and preferences. Lastly, Amazon Rekognition deciphers visual content from images and videos shared by fans, identifying objects, people, text, scenes, and activities, enriching our understanding of fan engagement. Together, these services form the foundation of our solution, enhancing the overall fan experience in the APAC region and beyond.

As we will analyze large sets of data that are coming from the users' social media accounts, we need to ensure every piece finds its perfect place. To generate precise insights we will collect, extract, and organize the raw data using S3 Bucket, AWS DynamoDB, AWS Glue, AWS Athena and AWS RDS.

Last but not least, to ensure that our analysis is thorough and insightful, we employ AWS Location Service to pinpoint regions with the highest social media activity, helping us target specific demographics effectively. Additionally, AWS QuickSight provides visual data analytics for our B2B services, enabling us to gain valuable insights and make informed decisions based on the data collected and analyzed from social media platforms.

Let's simplify our large AWS diagram by breaking it into smaller pieces for easier explanation.

User Authentication And Data Extraction

In this step, users sign up using AWS Cognito for authentication. Then, they answer questions to identify their favorite club/league, with AWS Personalize guiding the process. Next, users add their social media accounts to our app, which is managed by AWS Fargate. After adding their social media accounts, AWS Personalize again analyzes users' social media activity to identify their true fan identity and develop personalized recommendations. Additionally, another AWS Fargate service fetches public data from users'social media accounts and saves the data to an S3 bucket for further use.

User Social Media Post Analysis

In this step, AWS AI/ML services play a big role. Two services run together: one analyzes users' social media posts with AWS Comprehend, while the other analyzes users' photos/videos with AWS Rekognition. AWS Step Functions coordinate these analyses, with multiple Lambda functions executing based on our algorithm. Additionally, AWS Location Services identify the location of social media activity. After analyzing and extracting all the data, it's saved in AWS DynamoDB.

Analyze data in DynamoDB with Amazon Athena Federated Query

Since we're dealing with various data sources and are uncertain about the scale of analytics needed, it's wise to opt for a lightweight querying solution. Athena comes into play here, using data source connectors running on AWS Lambda for federated queries. These connectors translate between our data sources and Athena, automatically detecting metadata before each query. Additionally, we can use the AWS Glue Data Catalog to supplement metadata for connectors, especially useful when a data source lacks its metadata source, like DynamoDB. To share analysis results across teams, a custom dashboard is created using AWS Quicksight

Frontend

In this setup, we're using three databases: two AWS DynamoDB and one AWS relational database. These databases are connected to auto scaling-enabled EC2 services through an API Gateway.

Article Photo by Jimmy Connover

Challenge statement

Our challenge was to transform fishing matches with technology in order to streamline organization, creating immersive watching experiences while collecting valuable insights based on real data analysis for spectators, fishermen, sponsors, advertisers and equipment producers. Fishing competitions involve anglers competing to catch the largest, heaviest, or most fish within a set time, often requiring strategic planning and keen understanding of aquatic environments. Competitions can take place in a variety of settings, from tranquil freshwater lakes and rivers to the vast and challenging environments of deep-sea fishing. These events are open to a wide range of participants, including amateur local anglers, seasoned professionals, and anyone in between looking to test their skills. However, these events can face challenges in fan engagement and operational efficiency such as paperwork, complexity of live data recording, overall event organization and others.

The proposed solution

We want to perform a digital transformation of such fishing competitions by building a platform that targets to solve the challenges by bringing interactable viewing experience and data analysis for competitors, equipment manufacturers and fishing enthusiasts. Event management will get digitized, enhancing organization and sponsor interactions, helping in building digital communities to connect fans more deeply with the sport. This approach will not only streamline the logistics of hosting competitions but also will transform how spectators interact with and enjoy these events.

Beneficiaries of the Platform

The main benefits include the following:

• Event Organizers benefit from streamlined event management.
• Spectators enjoy a more immersive and interactive viewing experience.
• Competitors gain valuable insights from post-event analytics, helping them refine their techniques.
• Sponsors and Advertisers engage more effectively with a captive audience through targeted advertising.
• Equipment Manufacturers use detailed usage data as a feedback to improve their products.

Advanced Viewing Experience

For spectators, the experience is elevated with multi-angle live streams and interactive features. Multi-camera setup, including underwater cameras, will allow users to choose from various camera feeds, including those controlled by producers for dynamic shots and others that can be selected by viewers. Configurable overlay will allow users to view current match statistics, records and other information that will help to understand the current situation on a deeper level. All these ensure that every significant moment of the competition is captured and accessible, bringing viewers closer to the action than ever before.

Data Analysis and Utilization

The platform employs algorithms to analyze data collected during competitions. This analysis includes the effectiveness of different equipment, the strategies used by competitors, and environmental impacts on fish behavior. During a match, spectators will be able to view live data. After the match is done, relevant stakeholders will have access to final data analysis, helping them make informed decisions to enhance performance. This data can be provided to the manufacturers, so they can improve their professional products.

Promoting Sustainability and Education

Our platform also emphasizes sustainability and education by providing viewers with information on responsible fishing practices and the environmental aspects of fishing. This helps to promote a more sustainable approach to the sport and educates the public about the importance of conservation in fishing environments.

Our Tech stack

• Machine Learning and AI using AWS Web Services
• React JS (Web) & Flutter for Desktop (MacOS, Windows, Linux) and Mobile (iOS and Android) Apps
• Python (Backend)

High level AWS Infrastructure diagram and description

In the diagram below you can see the high level AWS architecture with the main services leveraged in our solution.

• S3 buckets to store raw and normalized data
• Lambda functions to trigger other AWS services like AWS Rekognition/SageMaker
• AWS Elemental MediaLive for normalising different feeds from various cameras setups
• AWS Elemental MediaPackage for real time packaging and streaming videos to end users
• AWS Interactive Video Service (IVS) for enabling interaction with streamed videos securely to end users
• AWS Rekognition (Custom Labels) with AWS Sagemaker to provide a suitable and efficient ML models for input raw data analysis
• AWS RDS for storing the data analysis results for further usage
• AWS Step Functions for orchestrating the over all flow of the platform

Conclusion

We believe that the overall interest in fishing matches will increase thanks to our solution, which will lead to fulfilling our vision and mission values.

Article Photo by Stephen Momot

Screenshot from a video by Milliken Fishing

Install Docker on your system

Install Docker into your system using any of the guides online for your preferred system.

Run the following command in your terminal to verify docker has been installed.

docker -v

Installing SonarQube on Docker

To run SonarQube on your machine using docker, first open a new terminal tab and then pull the image from docker hub using the command below.

docker pull sonarqube

Once the image has been pulled, start the container by running:

docker run -d --name sonarqube -e SONAR_ES_BOOTSTRAP_CHECKS_DISABLE=true -p 9000:9000 sonarqube:latest

Once your instance is up, log in to the Sonarqube dashboard by navigating to http://127.0.0.1:9000 and using System Administrator credentials:

• login: admin
• password: admin

Create a new project on Sonarqube

After logging in, you should see this screen.

• Let's select manually.
• Enter your project display name and main branch name.
• Select a baseline for new code for your project
• Your project should now be successfully created in Sonarqube

Analyze your project on Sonarqube

After creating your project, you should be seeing this screen.

Since we are going to analyze our project locally, click on Locally. Then generate your project token. Select the appropriate options under Run analysis on your project.

Let's also run SonarScanner via docker, no need to download and install it.

First create a file in your project in the root directory called sonar-project.properties

# must be unique in a given SonarQube instance
sonar.projectKey={projectKey}

# --- optional properties ---

# Encoding of the source code
sonar.sourceEncoding=UTF-8

# Here, you can exclude all the directories that you don't want to analyze.
# As an example, I'm excluding the node_module and .next directories
sonar.exclusions=node_modules/**, .next/**

Then, to run the SonarScanner and scan your code, navigate to your project in a new terminal tab and execute the following command:

docker run \
    --rm \
    --network=host \
    -e SONAR_HOST_URL="http://127.0.0.1:9000" \
    -e SONAR_SCANNER_OPTS="-Dsonar.projectKey={Project key}" \
    -e SONAR_TOKEN="{authenticationToken}" \
    -v "$(pwd):/usr/src" \
    sonarsource/sonar-scanner-cli

Eg:

docker run \
    --rm \
    --network=host \
    -e SONAR_HOST_URL="http://127.0.0.1:9000" \
    -e SONAR_SCANNER_OPTS="-Dsonar.projectKey=hrbot" \
    -e SONAR_TOKEN="sqp_9f5ff08eba566ab37afe5f36104fcxxxxxxxx" \
    -v "$(pwd):/usr/src" \
    sonarsource/sonar-scanner-cli

After the scanning is complete, go back to the dashboard and you should see the following screen:

You have now completed the scanning of your code for the first time. You can now go ahead and solve any issues and smells that are reported by Sonarqube on your path to a sparkling clean code!

To turn off the Sonarqube container afterwards you can run the command:

docker stop sonarqube

To turn it on again for later use, you can run the command:

docker start sonarqube

Article Photo by Milad Fakurian

Single Page Applications (SPA) revolutionized the web development landscape and provided users emulation of native applications. With it however, also came added complexity. “Meta frameworks” such as NextJS or NuxtJS further add on to the complexity as developers are now tasked with managing complex interactions between server and client. In an attempt to reduce this complexity, HTMX has re-emerged into the scene.

What was formally called “intercooler.js” is now HTMX. HTMX in a nutshell is a small client side Javascript library that attaches attributes to DOM elements. These DOM elements are then used to send requests to the server. What this effectively allows the developer to do, is to emulate SPA reactivity by controlling it from server by sending small, HTML “snippets”, which then HTMX swaps or inserts somewhere in the DOM. This simple concept is enough to power many web applications, which do not require the large baggage that modern frontend frameworks come with.

Among many other benefits, this also reduces the need for managing state in two places (client and server), and also allows developers to code in any backend language they prefer.

In this post I’d like to highlight HTMX's simplicity by building a small todo list application together while discussing its benefits and why one should consider using it in their next projects.

A Todo List

The best way to understand HTMX is by building something with it. As is tradition, let’s build a todo list application with it. Here already we start with the inherent merit of HTMX and that is that we can use it together with any programming language of ones choice. This allows teams not to be locked into some framework that requires specialized infrastructure. In fact, many developers may choose HTMX simply to get away from JavaScript, and we’ll be doing just that with Python and Flask.

This combination offers a wonderful experience for web development, enabling the creation of web applications that emulate Single Page Applications (SPAs) with ease and efficiency. With Flask and HTMX, you can avoid the complexities associated with traditional build systems like webpack, while effortlessly managing state between the client and server. This approach is also ideal for developing proof of concepts at high velocity. We’ll do this with the coding platform Replit. If you wish to follow this on your local machine, please refer to this tutorial, but since the point of this article is to display HTMX’s ease of use, we will disregard application setup.

You can start right away with Replit by navigating to a new Repl and selecting the Flask template.

Let’s start right await by making an index template that wraps our entire page:

<!DOCTYPE html>
<!-- templates/index.html -->
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Todo App</title>
    <script src="https://unpkg.com/htmx.org"></script>
    <!-- Tailwind for styling -->
    <script src="https://cdn.tailwindcss.com"></script>
  </head>
  <body class="display flex flex-col items-center w-full relative h-screen">
    <h1 class="text-3xl flex gap-4">
            {% include 'star.html' %}       Today    
    </h1>
    <div id="todo-list">
      <!-- This is where our todo list will go -->
    </div>
    <form hx-post="/add-todo" hx-target="#todo-list" hx-swap="“innerHTML”">
      <button
        type="submit"
        class="text-white text-3xl absolute right-5 bottom-5 rounded-full bg-blue-600 aspect-square w-6"
      >
        +
      </button>
    </form>
  </body>
</html>

And the following Python file that will contain all our logic:

# main.py
app = Flask(__name__)

@app.route('/')
def index():

  return render_template('index.html')

On the page we simply include HTMX via a script tag, no build step required.

Our first introduction to HTMX begins with the hx-post, hx-target, and hx-swap attributes on the form element. HTMX empowers web developers by allowing the addition of special attributes to any standard HTML element. These enhanced elements gain the ability to initiate HTTP requests. The responses in the form of HTML fragments, can then be dynamically inserted or replaced in any part of the webpage. This feature introduces a seamless way to update content without needing a full page refresh. In our case, we want to put our tasks into the div element with the id of "todo-list".

Let’s do this by inserting an empty task when the user presses the “+” button. We start by adding an endpoint that will create the HTML fragment, and then return that fragment:

# main.py
@app.route('/add-todo', methods=["GET"])
def task():
  return render_template('task.html')

And the following template:

<!-- templates/task.html -->
<div class="bg-gray-800 rounded-lg p-4 max-w-lg mx-auto my-6">
  <div class="flex items-center">
    <input
      type="checkbox"
      class="form-checkbox h-5 w-5 text-yellow-400 rounded border-gray-600"
    />
    <input
      type="text"
      placeholder="New To-Do"
      class="ml-4 bg-transparent border-0 placeholder-gray-400 text-white focus:outline-none"
      style="width: calc(100% - 2rem);"
    />
  </div>
  <textarea
    placeholder="Notes"
    class="mt-2 bg-transparent border-0 placeholder-gray-400 text-white focus:outline-none w-full"
  ></textarea>
  <div class="flex items-center justify-between mt-4">
    <button class="text-yellow-400 focus:outline-none">
                  Today        
    </button>
    <div class="flex space-x-2">
      <button class="focus:outline-none">
                      {% include '/svg/line.html' %}            
      </button>
      <button class="focus:outline-none">
                        {% include '/svg/dropdown.html' %}            
      </button>
      <button class="focus:outline-none">
                        {% include '/svg/hamburger.html' %}            
      </button>
    </div>
  </div>
</div>

With some added styling, the actual important points of this fragment is the input elements which will play a part soon. Clicking on the + button will now display a new, empty task. Let’s see how this works. The button looks like this:

<form hx-get="/add-todo" hx-target="#todo-list" hx-swap="“innerHTML”">
  <button
    type="submit"
    class="text-white text-3xl absolute right-5 bottom-5 rounded-full bg-blue-600 aspect-square w-6"
  >
    +
  </button>
</form>

The special attributes added by HTMX hx-get, hx-target and hx-swap tells us, “when the form is submitted, issue a (hx-)get request, and (hx-)target the element with id “todo-list, and (hx-)swap the inner content of this HTML.”. The concept of emulating SPA-like behavior via attributes is what forms the entire behavior of HTMX. Here’s a few other attributes which would allow us to do other things:

1. hx-trigger would allow us to trigger requests based on behavior, such as on mouse hover.
2. hx-post sends a post request instead of a get request. You can also do this with PUT, DELETE, and PATCH.
3. hx-boost allows us to convert all anchor tags and forms into AJAX requests that, by default, target the body of the page.
4. hx-push-url allows us to interact with the browser history API.

And many others. This also does not include the long list of values for each of these attributes which further allows us to customize behavior. It’s also important to note that these attributes can be applied to any element. These are all well documented in the documents page.

Let’s improve the todo list application by adding attributes to append tasks to the list, and to edit the individual tasks. Instead of swapping the entire inner HTML of the #todo-list element, we can append to the end with beforeend as such:

<form hx-get="/add-todo" hx-target="#todo-list" hx-swap="beforeend"
  …
 

These tasks are ephemeral, so we need to start adding some state. Since HTMX allows us to control the front end from the server side, we don’t need to reach for state synchronization solutions as it is often needed in applications that use larger frontend frameworks. We’ll drive everything from the server as our source of truth:

# main.py
task_list = [{
    "id": 222,
    "title": "Schedule doctors appt",
    "notes": "Make sure its in the evening after the 14th."
}]


@app.route('/todos', methods=["GET"])
def load_task():
  return render_template('tasks.html', tasks=task_list)

We load our state into the template, and further split up our code into fragments, similar to components (create a file templates/tasks.html):

<ul id="todo-list">
      {% for task in tasks %}         {% include 'task.html' with context %}    
  {% endfor %}
</ul>

The individual item templates are ready to consume the data supplied by Flask. This completes our "READ" process in our little CRUD application. Since we already have the "CREATE" process from our first code written, we can continue here onto our "UPDATE". We can add the ability to edit a task with a nice UX pattern. The user should be able to just type and the tasks will be updated seamlessly. We can do this by debouncing on key input. This can typically be quite terse with other frontend frameworks, requiring several lines and some intricate state updates, but with HTMX it is elegant:

<form
  id="task-form-{{ task.id }}"
  hx-post="/update-todo"
  hx-trigger="keyup from:input, keyup from:textarea delay:500ms
"
>
  <div class="flex items-center">
    <input type="hidden" name="task_id" value="{{ task.id }}" class="" />      
      <input type="checkbox" class="“…”" name="checked" />        
    <input
      type="text"
      name="title"
      placeholder="New To-Do"
      class="“…”"
      value="{{ task.title if task.title }}"
    />
  </div>
  <textarea name="notes" placeholder="Notes" class="“…”">
{{ task.notes if task.notes }}</textarea
  >
</form>

The hx-trigger on top of the form does a lot of work for us here, and it almost reads like in plain English, “trigger on key from input or textarea, and delay for 500 milliseconds”.

Finally, we can update the data on our backend:

@app.route('/update-todo', methods=["POST"])
def update_todo():
  # Extract the data from the form
  task_id = request.form.get('task_id')
  title = request.form.get('title')
  notes = request.form.get('notes')
  checked = request.form.get('checked') == 'on'

  # Update the task in the list with some fun list comprehension
  global task_list
  task_list = [{
      **task, 'title': title,
      'notes': notes,
      'checked': checked
  } if str(task['id']) == task_id else task for task in task_list]

  # return nothing so to disallow default swapping from HTMX
  return '', 204

And with that, we've concluded our "UPDATE" process code. I leave the exercise of deletion and sorting (or whatever else you may think of) to the reader. If you’ve been convinced by now, you can probably already see how trivial it can be to add the rest of these features. Up until now we've explored how to do some simple CRUD operations. One reason however that people chose to use the complex modern frontend frameworks is the ability to have state dependent on other state update automatically. While HTMX may not be the best approach here when requiring a complex set of dependencies across many DOM elements, it is still absolutely achievable to accomplish this with HTMX. HTMX provides the hx-swap-oob attribute that lets you swap outside of the target element. As the HTMX documentation says:

This allows you to piggy back updates to other element updates on a response.

With this in mind, we can add a great example for our little app. What we will add is a counter at the top of the application header. This counter is a state that is dependent on the number of todo items. When we add a new todo item, we can update the top header with the new updated count value:

At the end of the task.html file we add:

<!-- rest of HTML -->
{% if new_task_length %}
<span id="counter" hx-swap-oob="true">{{ new_task_length }}</span>
{% endif %}

And then we add the following item to the return statement of the /add-todo endpoint:

@app.route('/add-todo', methods=["GET"])
def task():
# ...rest of logic
	return render_template('task.html', task={}, new_task_length=len(task_list))

this demonstrates how easy it is to manage state dependent updates via HTMX's hx-swap-oob feature. Here, with just a few lines of code, we have created an additional feature in our app - a counter that automatically updates based on the current length of our task list. The length of the task list is a dependent state, reflecting the total count of todo items. When a new task is created, the counter in the header is automatically refreshed to display the new count.

I want to conclude this tutorial exercise section by discussing animations, one reason why many people choose to use Javascript frameworks.

![Github Branches](/assets/img/articles/2023-12-19-Introduction-to-HTMX/img-2.webp

Indeed, animations are a key aspect of modern web applications, often being a primary reason for choosing JavaScript frameworks. However, with HTMX, you don't have to forgo this essential element of user experience.

Adding Animations with HTMX

Animations can be achieved through various methods while using HTMX. A significant portion of interactive feedback can be handled with CSS. Modern CSS provides enough for creating animations and transitions, making it possible to animate changes in your UI in response to HTMX events.

For instance, using CSS transitions, you can smoothly animate the appearance and disappearance of elements, or their transformation in response to user interactions. This can be particularly effective for operations like adding, updating, or deleting items in a list, where visual feedback enhances the user experience. With HTMX, this is taken a step further with great integrations such as swap transitions. Take an example from the docs:

If you want to fade out an element that is going to be removed when the request ends, you want to take advantage of the htmx-swapping class with some CSS and extend the swap phase to be long enough for your animation to complete. This can be done like so:

<style>
  .fade-me-out.htmx-swapping {
    opacity: 0;
    transition: opacity 1s ease-out;
  }
</style>
<button
  class="fade-me-out"
  hx-delete="/fade_out_demo"
  hx-swap="outerHTML swap:1s"
>
  Fade Me Out
</button>

Leveraging JavaScript Libraries

For more complex animations, you can integrate lightweight JavaScript libraries. GreenSock (GSAP) is an excellent example of a library that provides advanced animation capabilities with a simple API. By combining HTMX with such libraries, you can create sophisticated animations that bring your UI to life, without the overhead of a full JavaScript framework.

Lastly, I'd like to point out the emerging Transitions API, currently only available in Chrome. This API is set to offer native support for smooth transitions, potentially reducing the reliance on external libraries for such effects, thus making the necessity of SPA frameworks less relevant. HTMX too, integrates well as it provides access to this API via the hx-swap attribute, which you can learn more about here.

Wrapping Up

HTMX offers a sublime experience for developers with simplicity and functionality. It allows for developers to focus on business logic without compromising on UI/UX. With HTMX, basic REST knowledge is enough to build a fully functional user interface, reducing complexity, development time, and costs.

HTMX isn't a solution for everything, though. For more complex applications, a comprehensive JavaScript framework may be more suitable. The state management and component-based architectures that HTMX doesn't inherently support might be a better fit depending on the application and needs.

Despite this, HTMX's ease of integration and minimalistic approach makes it a valuable tool that should always be considered. It's particularly effective for enhancing server-rendered pages or for projects where a full JavaScript framework is unnecessary. Ultimately, HTMX stands out for its ability to deliver efficient, interactive web experiences with minimal overhead, making it a technology worth considering in various web development scenarios.

Since there is always more to learn, please consider checking out the official documentation and the essays as well as the free book provided by the HTMX author. You can also visit the source code used for this project here: Replit

Resources

Article Photo by HTMX

Nikita is, in his own words, a "healthy perfectionist, stoic and a creative person". In his free time, he likes to play guitar, sing, write stories, poems and books. Nothing is published yet, but hopefully one day we'll get to read his work.

![Prague Team](/assets/img/articles/2024-01-19-Interview-Nikita-Sirovskiy/team.webp

Nikita, can you describe your journey of transitioning from a developer to a project lead? What were the most significant changes you had to adapt to?

“Extreme Ownership”, that’s what sums it up best. I've always been proactive and tried to enhance things on all the projects I was a part of. Questioning decisions made outside of my direct responsibility zone was and is very important to me. I work not only on mobile apps, I work on products. A right question at the right time might turn the tide. I know people say ‘I’m not getting paid for this or that.’, but I never understood this. Of course you must not do things for others, yet, if you see a room for improvement, why not just improve it?

One of the most significant changes is that now, most of the time, I am the person who answers questions, not the one who asks them. Deadlines, scopes, code reviews and all this. After all this, there’s not much time to code myself. While being a lead gives me a bird-eye view on the entire project, which is very interesting, sometimes I miss these days where all I had to do was just to code. Coffee, good music, quietness, flow state… Yes, sometimes I definitely miss this.

Q: During your first project as a lead, what were some of the biggest challenges you faced? Could you share a story where you had to overcome a difficult obstacle, and how you managed to do it?

Self-doubt. That’s the biggest thing for sure. It was hard to accept that now it’s me who is responsible for making decisions and saying yes or no to other’s decisions. Sometimes the pressure was high, sometimes I was not confident in my own solutions and decisions, sometimes the deadlines were really tight. Although people around me — my family, my line manager, my project manager and my teammates — supported me a lot. Not only with valuable advice, but also by saying that the result is fantastic so far. Another day, when looking at the result myself, I realized that we’ve built the best mobile app I ever worked on. It’s stable, developer experience is great, the code is clean, and the UI/UX is amazing. That gave me a lot of confidence and the game changed.

“Don’t pull yourself backwards” — that’s what one of my friends told me.

Q: Given your experience with remote work, how do you handle the dynamics of leading a team remotely, especially when occasionally there is a need to visit client offices as well?

I worked remotely since the beginning of my career, and I still do. Although Monstarlab’s borderless approach took it to another level. Even people who work from office usually sit in different offices all around the world! This is amazing. You can meet so many different people. This is how we work on the current project as well.

We try to be more autonomous and contextualized. Less meetings, more documentation: in-code, Confluence, Google docs, or whatever is more convenient for a specific topic. Less direct messages in messengers, more communication in corresponding tools (GitHub, Figma, etc). Different time zones is a challenge itself. I want us to reach the point when, if someone needs something, they will be able to find it in the docs. So we won’t need to disturb or wait for each other in Slack or in a meeting room.

But working remotely means that colleagues are more disconnected from each other. Sometimes it’s hard to understand the needs or mood of your teammates, so periodic catch ups are very important. I have 1-on-1 calls where I sincerely ask my teammates about what is going on. Is there something to change, a problem that must be fixed or just a question that needs an answer. We find it very beneficial. I was given a lot of suggestions on these calls. It also allows you to discuss off-topic things which leads to better connection with a person. Although it is important not to turn these calls into rituals that you blindly follow just “because”.

Work trips are also very beneficial. All my visits were filled with productive and interesting time. Brainstorming sessions, lunches, morning coffees, debug sessions. This for sure enriches your connection with people. As someone who works remotely most of the time, I value it a lot. Perhaps one day I decide to relocate to be with the team all the time. I’m very curious how it would be.

Q: What advice would you give to developers who are about to take on their first leadership role? Are there any key lessons you wish you had known when you started?

Well, don’t pull yourself backwards, that’s for sure. You are where you are for a reason. Your skills and commitment have led you to this role, so just keep it up and do your best.

For more specific things, learn how to organize yourself. Self-organization helps to keep track of things: questions raised on calls or chats, project goals, tech debt, ideas etc. Proper tracking allows you to plan better, oversee everything and move faster. I started using Apple Notes extensively. I have folders for interviews, TODOs, suggestions, conversations, and basically anything that is important in any way. One thing I love is the new (maybe not that new anymore) feature with the quick note in the corner of your screen. I keep my TODO list there. If I need to note something, I just move my cursor to the right-bottom corner and start noting.

I write a lot of documentation. This is an extremely beneficial tool for time saving and optimization. We don’t need to re-explain rules and there’s less need to sync. If in doubt, one can open the documentation to learn how certain things are done.

Another thing is talking to people. This will sound classic, but everyone is very different, and everyone requires a different approach. You will need to provide constructive feedback, be a mentor for your colleagues, raise problems and concerns to managers and the business, suggest ideas and say no. The latter is a skill itself, a very important one. I like this phrase that I heard somewhere, «If it’s not a 100% yes, then it’s a no». Of course it’s not really applicable here (otherwise we would’ve been blocked most of the time!), but this idea is still a good foundation for making decisions. Remember, you’re in charge now, and raising concerns, whatever those are, is a must. I failed to do this a few times. Those problems came back at a later stage of the project. Nothing big, fortunately, yet still… I learned this very well.

Q: How has your experience been working at Monstarlab? Could you share some highlights or key moments that have shaped your journey here?

It’s been fantastic so far! Working here has been a journey of growth and challenges. What I can highlight is the level of trust that you have here. On my very first project, I quickly shifted to conducting demos, enhancing code practices, and doing code reviews. On my second project, when the lead relocated to a different time zone, I stepped up to handle most of the code reviews, demos and crucial feature finalizations. On my third project I was mostly solo, which made me the one who talks with designers, backend developers, clients and PMs.

The atmosphere on every project was amazing. Ask anyone, and everyone will tell you that people in Monstarlab are special. They are always eager to chat, to laugh, to help and support.

Someone even invited me to their home, and we spent evenings talking about life. Someone drives me to and from the office every time I’m on a work trip, and we have these short morning and evening conversations.

Q: Reflecting back to the beginning of your career, what initially inspired you to start programming? Have there been any specific role models in the tech industry who have influenced or guided your path?

I have a very vivid memory of asking my father “Who creates computer games?” and him answering “programmers”. That was the initial inspiration. I wanted to be either a programmer or a taxi driver. Don’t ask me why. Today I don’t even have any clue about cars. Anyways.

It became serious when my father discovered that you can create custom game maps, or modes, for my favorite computer game, Warcraft 3. This involved coding, designing characters, and crafting stories and worlds – a mix of technical skills and creativity. I think I was 7 years old when he showed me that program, World Editor. I spent countless hours, days and nights, in trying to craft things. My mother also played a crucial role. She forced me to learn more and take additional classes, even when I didn’t want to. But in a soft way. I always had full freedom in experimenting what I want from this life and in learning whatever I think is needed.

But I didn’t become a game developer! Maybe one day, maybe one day. I was lucky to have that map-creating period in my life. It has built a solid foundation for problem-solving and creative skills that I use today.

Q: Every profession has its lighter side, and programming is no exception. Do you have a favorite programming joke or a meme that resonates with you or brings a smile to your face?

Yes! I wanted to find the picture, but I couldn’t. There's this meme comparing users enjoying a new feature to developers tangled in a mess of code trying to figure out how to develop a new feature without breaking the app.

While humorous, it shows the real challenges in coding. I've seen poorly organized codebases that looked like spaghetti. It's a reminder that we should never trade-off maintainability for delivering something new. Organization, requirements, documentation and optimal processes make development cycles efficient and apps cost-effective.

It improves another thing that is often ignored, developer experience. The happier the devs, the more interesting it is to work on a project, the better the project will be. I hope that that meme is not applicable for us. Hope that in our case both of the sides of the picture represent happy people.

GitHub Copilot turned out to be my guiding star in this Python odyssey. With its code suggestions and real-time assistance, it felt like I had a seasoned Python developer right by my side. Copilot not only streamlined my coding process but also served as an exceptional teacher. It provided explanations, alternative solutions, and helped me grasp Python's intricacies faster than I could have imagined.

But Copilot wasn't my only ally. ChatGPT, another AI companion, was there to answer my questions and provide insights whenever I needed them. It didn't matter if it was about Python syntax, or data manipulation with Pandas, or even conceptualizing ETL workflows – ChatGPT was always ready with accurate and thorough responses.

Together, Copilot and ChatGPT became the dynamic duo that guided me through the twists and turns of this project. They helped me build an ETL pipeline that not only met the project's goals but exceeded my own expectations. The pipeline seamlessly extracted data, transformed it to fit the target system, and loaded it with just some adjustments on the code but in general guiding me with really good samples.

As I reflect on this journey, I can't help but marvel at the capabilities of modern technology. With just basic Python knowledge, I accomplished what once seemed like an task I wouldn't have the skills to do. Copilot and ChatGPT not only accelerated my learning but also provided creative solutions and expanded my horizons. This project has taught me that with the right tools and determination, even a novice can achieve remarkable results in the world of programming.

Getting Started

Beyond coding, ChatGPT helped me with setting up my development environment. It guided me through best practices for organizing Python scripts, ensuring my codebase was clean and maintainable. When it came to connecting to various data sources, ChatGPT was my helper when I got blocked. It provided step-by-step guidance on how to integrate with Salesforce, external APIs, and Google Services like Google Drive API. Obviously, I had to explore a little bit directly into the APIs documentation to ensure the current ChatGPT response was in line with the needs. At one moment I was lost on a task to set up the services needed for the project and ChatGPT surprised me. It even outlined the necessary roles and permissions demystifying the process of establishing these connections and setting up the grants needed for this task.

Moreover, ChatGPT helped me navigate the complex world of the Google Cloud (GCloud) infrastructure. It guided me through setting up connections to services like BigQuery and Google Storage. And as mentioned before it also helped me demystifying the process of managing grants between service accounts needed to open a connection between the different services used, ensuring that my ETL pipeline operated smoothly within the Google Cloud ecosystem. ChatGPT showed me step by step instructions on how to do things like creating a Service Account Key and how I could use it to connect with Google services using Python and the libraries used for a specific service.

On the other hand, ChatGPT helped me to set up Salesforce connections and it guided me through the existent APIs. Building the connection between Salesforce and the ETL on Python was a challenge since it implies a different kind of Authentication with OAuth. ChatGPT was able to explain me the ways that we could authenticate and it showed a few approaches that could do. In fact, during the implementation, it was necessary to look at the libraries used for an easier Salesforce API connection. During this research the Simple Salesforce library was recommended which ended up being my tool to build the solution and perform in a easy way queries to retrieve the information from that source.

One more sample that ChatGPT provided was for Google spreadsheets connection. I asked “Which grants needs a Google Service account to read using Google Drive API and get access to Google Spreadsheets?” and ChatGPT showed me what I needed from Google Cloud and how I could use a Service Account:

That was not the only thing, it also helped to understand things like how could I sort and use Pandas dataframes. This was really helpful through my work of developing the solution:

Using Copilot for Code Assistance

Initially, I started thinking about how I could use the tools and libraries to set up my environment. During this phase, Copilot helped me build the initial inputs for tasks like establishing the connection with the Salesforce API and how I could start sending requests through it using SOQL queries to retrieve information from the system. I have to admit that in those cases where a language like SOQL comes into the scene, it was not completely precise since it showed normal SQL in the suggestions, but sometimes it was close enough to what I wanted to achieve, so I just had to modify the proposed query slightly.

One additional task Copilot helped me with was clarifying complex aspects of Python and the Pandas library, providing me with insights and solutions that were instrumental in achieving the project's goals and understanding the libraries used.

Tasks like opening a research to merge two different Pandas or understanding how the Pandas work, are easier with Copilot completion:

Navigating Legacy Code and Pandas Mastery

One early challenge I encountered was dealing with legacy code from an old ETL pipeline, in order to understand the model expected by the final output. There was a huge amount of "spaghetti code" that I had to try to understand, which was not an easy task given my expertise in managing Python and Pandas. However, Copilot proved to be a lifesaver in giving me clarity. It assisted me in understanding the intricacies of the existing code, breaking down its logic, and understanding legacy code that was not following best practices.

As I mentioned, a significant part of my project involved working with Pandas library for data manipulation. Copilot helped me understand how Pandas could make different tasks like merging data from different tables, making transformations, sorting, and more. Its code suggestions and explanations really helped me, enabling me to merge different tables from different data sources and to ensure data consistency throughout the ETL process.

In cases where I was not completely sure what the code was doing Copilot Chat was able to explain it to me and give me more context about what a piece of code was doing.

Let’s say that I have the next piece of code:

trackings['hours'] = (pd.to_datetime(trackings.end_at, format = "%H:%M:%S") - pd.to_datetime(trackings.start_at, format = "%H:%M:%S")) / np.timedelta64(1, 'h')

It was able to read the opened file and explain to me a little bit about the operation of the given code:

In addition it was able to add some improvements suggestions to the code:

This method proved beneficial in not only developing the code but also in documenting it simultaneously. Whenever a line of code had ambiguous intentions, the tool assisted by generating explanations and documentation, ensuring each step of the code was clear and well-documented from the outset.

Copilot helps a lot when you want to document your code to improve its readability.

Transformation Magic

Data transformation was a fundamental aspect of my ETL processes. Copilot and ChatGPT became my virtual mentors in this regard. They guided me through the process of applying transformations to the data, showcasing the versatility of Pandas in handling diverse data manipulation tasks. Whether it was altering data types, aggregating information, or reshaping dataframes, Copilot provided clear code suggestions. It is important to mention that in some cases where Copilot had generated some strange code I used ChatGPT in conjunction with Copilot’s suggestions to give me some clear explanations and better approaches. This enabled me to wield Pandas with confidence.

Considerations such as how to use the Pandas 'loc' function to fill data from other columns only when a specific condition is met, or other suggestions where I just need to adjust the names of the columns in the dataframes, are areas where these tools have proven invaluable. They have also assisted in performing calculations within the dataframes, such as the sample of computing the number of hours between a range of dates. This demonstrates that using both AI tools can unleash your potential and streamline complex tasks, saving significant time.

Even when I had to combine data from multiple tables and sources, providing samples in the prompt or within the method, it was capable of generating an entire method with the required output. In this case, I provided sample inputs of the method along with the expected output, and it generated multiple completion options, which was really helpful as it saved me a lot of time. To successfully accomplish these complex tasks, it was necessary to provide sufficient scenarios of data combinations from the input tables and the expected output.

Things like how I could split column tables on rows assigning numeric values in the same method it was able to actually build the whole structure.

Whit the next explanation on the method:

def transform_work_calendar_to_user_working_days(
    users_df: pd.DataFrame,
    work_calendar_df: pd.DataFrame,
) -> pd.DataFrame:
    """Converts work calendar to user working days mixing users_df and work_calendar_df in order
    to create a user_working_days_df output matching with UserWorkingDays model,
    with the following columns:
    - user_id: User.Username | Contact.Id depending if the user is a resource contact. Each contact has Work_Calendar_id relationship.
    - numeric_day: Day of the week starting from 0 (Sunday) to 6 (Saturday), use the Work_Calendar.{day}_Hours to identify the day.
    - hours: Work_Calendar.{day}_Hours value as hours in float format.

    Args:
        users_df (pd.DataFrame): users df with user data to match with its information.
        work_calendar_df (pd.DataFrame): work calendar df with work calendar.
    Returns:
        pd.DataFrame: with the MPlanner UserWorkingDays schema columns needed.
    """

It was able to produce the following implementation:

    # Transform days to numeric values and placing values to its corresponding numeric day
    # Melt the DataFrame to reshape the data
    working_hours_df = pd.melt(
        working_hours_df,
        id_vars=["id"],
        value_vars=[
            "Sunday_Hours",
            "Monday_Hours",
            "Tuesday_Hours",
            "Wednesday_Hours",
            "Thursday_Hours",
            "Friday_Hours",
            "Saturday_Hours",
        ],
        var_name="numeric_day",
        value_name="hours",
    )

    # Map the day names to numbers
    day_to_number = {
        "Sunday_Hours": 0,
        "Monday_Hours": 1,
        "Tuesday_Hours": 2,
        "Wednesday_Hours": 3,
        "Thursday_Hours": 4,
        "Friday_Hours": 5,
        "Saturday_Hours": 6,
    }
    working_hours_df["numeric_day"] = working_hours_df["numeric_day"].map(
        day_to_number
    )

And it was also able to create the merge output of the data:

...
    user_working_hours_df = user_df.merge(
        work_calendar_df,
        left_on="Work_Calendar_id",
        right_on="Id",
        how="left",
    )

    return user_working_hours_df

The above code is just a sample that I have in the method. With more complex tables, it was also able to combine different tables, rename columns, and make calculations if needed. I just had to make some adjustments to clean the code and align with best practices.

On the other hand, tasks like defining new methods to refactor the code and avoiding duplicated pieces of code are aspects that Copilot does not always identify. When it generates suggestions, I conduct a thorough review of the generated code, removing or calling existing methods or defining new ones as needed. Even in these cases, it helps me to refactor and create new code with minimal effort, without needing specific prompts. Therefore, it's important to pay special attention to these aspects when using a companion tool like Copilot.

Filtering and Beyond

Navigating through data filtering posed a significant challenge for me. The guidance I received in crafting and implementing filter conditions within dataframes was tremendously helpful. It wasn't just about understanding the syntax; it was about discovering innovative ways to accurately and efficiently retrieve the necessary data.

Challenges and Limitations with GitHub Copilot

While GitHub Copilot undoubtedly played a pivotal role in my ETL project, it was not without its share of challenges and limitations. It's essential to acknowledge these aspects to provide a comprehensive view of my experience. It is important to mention that the challenges come with the prompts you use and that is where you start seeing the limitations of this kind of technology.

Contextual Understanding

One notable challenge I encountered was Copilot's occasional difficulty in grasping the full context of my project. This became apparent when it came to fetching columns from tables within my code. Copilot tended to infer common fields found in typical databases, such as 'id,' 'name,' 'full_name,' 'cost,' and others. While these fields are indeed prevalent, my project often required more specific columns that were unique to our data schema and the model on the project used.

In such cases, I found myself having to provide additional context within the prompt, explaining the current columns of a table explicitly. This was necessary to guide Copilot in suggesting the correct fields. It's worth noting that this limitation might have arisen because I couldn't grant Copilot access to the entire codebase due to company privacy agreements so I had to be really careful to not share information relevant to keys and other variables that could imply an exposure risk.

Another noteworthy observation was the evolving precision of Copilot's suggestions as I actively engaged with it. Over time, while Copilot was active and I navigated through the code, it gradually became more precise in its recommendations.

Quality of Suggestions

However, it's important to acknowledge that Copilot isn't infallible. There were instances when it provided suggestions that were not entirely aligned with the project's requirements, particularly in handling Pandas dataframes and processing its data in some cases. For example, I encountered a situation where Copilot recommended iterating through dataframe rows using conventional loops. This approach, while technically viable, was less efficient and more cumbersome compared to utilizing Pandas' built-in functions for data transformation. These kinds of suggestions, though well-intentioned, could sometimes lead to undesirable or even incorrect code snippets. This was especially challenging in scenarios where leveraging Pandas' powerful and optimized methods could significantly streamline processes and improve performance. Such instances highlighted the need for careful review and consideration of Copilot's suggestions, especially when implementing more sophisticated data manipulation techniques in Python.

Recommendations for Improvement

Based on my experience on this project, I'd like to offer some suggestions for enhancing Copilot's effectiveness to new users:

1. Access to the whole codebase: To improve Copilot's contextual understanding, granting it access to the entire codebase would be beneficial. This would enable it to better recognize and adapt to project-specific variables, tables, objects, schemas and so on. In that way it could give better suggestions using existent code and giving more context.

2. Context in prompts: Providing more detailed context within prompts could aid Copilot in generating code that aligns closely with the project's needs. For example you could explicitly specifying the desired columns or explaining unique data requirements which will lead to more accurate suggestions.

3. Iterative learning: As observed, Copilot's suggestions became more precise with active usage. So, encouraging developers to interact with Copilot on a project actively and fine-tune its suggestions during the coding process could lead to better results.

4. Don’t use the first suggestion and instead explore different prompts, it possibly suggest better outputs.

5. Before apply a suggestion from Copilot think about which other solution could be better on terms of performance and also taking into account best practices like clean code or applying SOLID principles for example.

Limitations that I found were the plugins on certain IDEs don’t have all the features that Visual Studio Code IDE offers like interacting with Copilot in a similar way as we were chatting to ChatGPT. This feature at the moment of this article has been written is in Beta and not all people have access to it on other IDEs.

Also it is important to mention that even with this companion tools is necessary to have the knowledge of expertise building applications and applying concepts like clean code and clean architecture principles.

Conclusion: Copilot - ChatGPT

In summary, GitHub Copilot proved to be an indispensable coding companion throughout my ETL project. It bridged the gap between my basic Python knowledge and the intricate demands of data migration and transformation. Copilot's ability to clarify legacy code, demystify Pandas, and provide guidance on data manipulation was instrumental in achieving the project's objectives. With Copilot's assistance, I not only completed the project successfully but also expanded my Python and data engineering skills significantly. Its ability to accelerate coding, offer insights, and simplify complex tasks is undeniable.

However, it's essential to acknowledge its limitations and work towards improving its contextual understanding and the quality of its suggestions. With continuous development and refinement, Copilot has the potential to become an even more indispensable asset in the developer's toolkit.

On the other hand, ChatGPT is another really valuable companion tool that helps you to find in a easy way the punctual topics that you need to achieve your goals, whether in professional or even in your personal life, showing you clarity, guidance, and really useful suggestions in your tasks. Questions, blockers or lack of comprehension in your daily life could be improved and it could unleash more of your potential. I have to admit that my searches on Google and StackOverflow have decreased significantly using these tools. It’s like talking with an expert without those extensive explanations and really long threads that sometimes just lead you to waste your time and energy without a clear answer.

Obviously, this tools won’t replace the expertise you have gained during years of hard work, and here is another good point to underline: even with all this access to exactly what you need using AI, you always have to make smart decisions and not just copy and paste what these companion tools suggest to you. You have to continuously apply best practices, and depending of your challenge it could guide you to make different decisions than the suggestions made by these tools.

Article Photo by Luis Forero generated using DALL-E

Born and raised in Dubai, she indulges in a variety of activities that reflect her dynamic personality. From practicing calisthenics and aerial dance to enjoying video games, paddle boarding, and tackling escape rooms, Nairah loves to challenge herself. And when it comes to food, she will never say no to a delicious bowl of ramen.

Nairah, could you tell us about your experience developing immersive technology apps that use generative AI? Could you share a cool prototype you have worked on recently?

I’ve started incorporating generative AI into my projects as early as March 2023. When the trend took off, I immediately saw its value and got started on ideating and developing new experiences. The latest one I developed was to help train customer facing employees to deal with difficult client cases. In this scenario, the “customer” was a generative AI human that had a specific banking challenge that you needed to converse with to help solve. The conversation would be recorded and at the end you get an evaluation of how well you handled the situation – what you could have done better, statistics on the conversation as a rating from 1-10, etc. Because the “customers” in the prototype are generative AI, it’s very easy to change out their prompt to handle different situations or even industries.

What have been some of the most significant challenges you've faced when creating immersive tech experiences using generative AI? Could you share an example where you had to innovate or think creatively to overcome these challenges?

When I had started off back in March, the generative AI scene in Unity was still quite fresh and there weren’t any helpful tools yet. My earliest prototypes had a pretty dissociative issue which was a long delay between you speaking to a generative AI avatar and it responding back to you. This was due to the processes of speech to text, analyzing the question and generating an answer, and then transforming this text back to speech using generative AI to sound more human. As a workaround, I started processing smaller chunks of text at a time, in order to reduce this delay. Eventually, many helpful tools came to market, such as Inworld, that make this process easier to implement and without the speech delay I was facing.

In a field as rapidly evolving as immersive tech and generative AI, how do you keep your skills and knowledge up to date?

With new challenges and brief timeframes, I’m compelled to find the most efficient and latest technologies to help me develop them. It’s been very helpful to have the freedom to explore new creative solutions and develop them. The constant practice of doing this over the last year has really helped me stay up to date with the latest technology.

Looking forward, where do you see the integration of generative AI and immersive technologies heading? What are some advancements you anticipate in creating more engaging or realistic experiences?

I feel generative AI is already in a very good state and I trust it to continue to get better over time. Where I anticipate more improvement would be in the realm of more realistic human characters to work with. While the avatars look great, there are nuances in real-time facial experiences and mouth movements that I’d be excited to see improve in order to create better immersion.

How has your experience been working at Monstarlab? Could you share some highlights or key moments that have shaped your journey here?

I think it would be difficult to highlight only one particular moment. As a whole, the environment I’m in is set up nicely to foster innovation and push the boundaries of what is possible. I also have a great team I can depend on and I cross collaborate regularly with other departments. This helps us come up with some really great solutions.

As an immersive tech engineer, it’s always important to be up to date with the latest in technological innovation and Monstarlab gives me the platform to do so.

If you had the chance to choose one programming superpower, what would that be and why?

Read documentation once and remember it forever 😊. However, I am super happy to see tools such as Unity Muse come out, which keeps all its documentation alive in a generative AI chat, similar to ChatGPT. It’s very handy and has made programming a lot more efficient.

What's on your current tech wish list? Any specific gadgets or software tools?

Definitely the Meta Quest 3 for its mixed reality capabilities. I’m also looking forward to trying the Apple Vision Pro soon!

Today, we're chatting with Adam Mack. With over 15 years at the company, starting back at Fuzz Productions (a digital agency which was acquired by Monstarlab in 2019), Adam is a Solution Architect or Technical Product Manager, depending on the day of the week!

Living in Massachusetts, USA, Adam enjoys life with his partner Molly and their cat, Cray-cray. When he's not tackling tech problems, he's into pinball, making and playing video games, creating music, and cooking. Let's dive into his world!

Adam, could you share with us the story of how you entered the field of generative AI? What was the inspiration that led you to focus on this particular area in technology?

I was fascinated by Google Deep Dream back in 2015, and the bizarre dreamlike images it could create. That’s when I first became aware of AI image generation as an academic field, and I’ve sort of kept watching the space since then. Fast forward to 2021, and the fairly quiet announcement of Dall-E 1 from OpenAI; I was showing every single person I know the landing page, saying things like “look! It can make an avocado chair out of nothing!!”. I was very excited about the possibilities, to say the least.

For those who are new to the concept of generative AI, how would you explain it in simple terms?

For friends and family who know nothing about computers, let alone neural networks, I usually refer (half joking) to the analogy of waiting for a million monkeys to accidentally write Shakespeare - what if we could train the monkeys over time to learn what words mean, and which words go next to each other more frequently than others? Even if they can’t fully read or write like you or I can, you’d start to see patterns emerge that resemble Shakespeare (or whatever input) more closely than the random noise you’d otherwise get. Now imagine you can give positive reinforcement to specific monkeys that do a great job of generating Shakespeare-like patterns; you’re part of the way toward visualizing a neural network at work.

What do you think are some of the most exciting trends in the field of generative AI? How do these trends influence your work?

The most exciting trend, hands down, is seeing how product teams everywhere are taking the “basic” building blocks of generative AI and layering them with the fundamental building blocks that make up their product, leading to new interaction patterns that would have been unimaginatively complex to engineer just a few years ago. Just opening up natural language as a viable user interface for software is a game changer - I think anyone frustrated with the limitations of last gen voice assistants knows how much unrealized promise there is in the ability for a user to naturally converse with a machine. Especially when it comes to the context of providing enhanced accessibility or representation to audiences that are either underserved or have accessibility challenges.

Looking ahead, where do you see generative AI heading in the next 5 to 10 years? What kind of advancements or changes do you anticipate?

I have so much hope for generative AI’s potential to give an artistic voice to those without the resources or ability to “compete” with commercial interests. Suddenly, if you have a dream for a movie or illustration, it’s no longer locked up in your brain if you lack the artistic ability or tools to realize it. I feel like that could be transformative for society in the same ways that the printing press was transformative, in giving a voice to those who previously lacked a voice, giving access to creating art to those who lacked access before. At the same time, especially as AI outputs become closer to commercially viable replacements for human labor, we’re going to see a huge backlash against these technologies and the use of these technologies in commercial applications. I don’t know who wins that battle for hearts and minds of consumers, but it’ll be interesting. In the meantime, I’m loving that I can create beautiful “oil paintings” with words.

You seem to have a strong passion for image generation. Could you share with us an image you created using generative AI tools that particularly amazed you?

Great timing! At time of writing, Midjourney 6 just came out. In my many hours experimenting, I’ve coaxed it to create some stunning images. I’ve included 4 of my favorites!

How has your experience been working at Monstarlab? Could you share some highlights or key moments that have shaped your journey here?

I’ve been here quite a long time in internet years, growing my career from being a very inexperienced graphic designer, to being a solution architect capable of leading international projects. In many ways, the big opportunities of my career have been largely due to chance, where I accidentally find myself in front of a new opportunity to do something important, and I need to adapt and succeed. So that has always been my takeaway, and my secret to success and longevity in my career growth here:

Always be ready to seize unusual and unexpected opportunities when they arise. It’ll never be what you expected.

Reflecting on your early days, what is your earliest memory related to coding? Could you tell us about your first project and the experience of working on it?

Growing up, before DOS and Windows machines, we had several Atari personal computers around the house, and one I remember most fondly was the 800. I was fascinated by early games like Time Bandit and Pitfall, and it led to me asking my father questions about how games work. That led us to doing little programming exercises together out of magazines, plugging them into BASIC and hoping they ran. I remember one time he led us in daisy chaining a bunch of hardware together for the intended outcome of capturing still images from a VHS and printing them in high contrast black and white on a dot matrix printer as “coloring book” images. Wizardry like that is what sparked my life long interest in software and hardware development.

Is there a book, not necessarily related to programming, that has had a significant impact on your thinking or approach to your work?

I always love an opportunity to plug this book - I read it in college, and it changed my whole way of approaching critical thinking: How We Know What Isn’t So by Thomas Gilovich. It’s all about the simple lapses in reason that affect everyone, every single day, and how you can recognize and work around them. It really impacted how I evaluate my own thinking and beliefs, and how I approach problem solving.

• Lua scripting language.
• curl command-line tool for transferring data specified with URL syntax.
• SDL cross-platform development library designed to provide low-level access to audio, keyboard, mouse, joystick, and graphics hardware via OpenGL and Direct3D.

Python libraries, such as Numpy, also reach out to C in order to speed up certain parts of the code.

Recently, while working with images, we discovered that UIKit can't decompress all JPEG2K images. Since this was a mandatory requirement, we had to find a solution. After a bit of digging, we came across OpenJPEG. We did a few tests and confirmed that it was able to decompress the images that UIKit couldn't. The only issue remaining was that OpenJPEG was a C library.

To use OpenJPEG in an iOS SDK, there were two options:

1. statically compile the library and bundle it in an XCFramework
2. wrap the source code in a Swift package

For option 1, we'd have to deal with CMake, compile it for multiple architectures, archive them in Frameworks and finally package them in an XCFramework bundle. That seemed to be very time-consuming, so we went with option 2, which is not without its quirks, but pretty straightforward once you understand it. So let's C how to do that, wink wink.

In this article, we're going to create a small wrapper around the Xoshiro256++ PRNG (Pseudorandom Number Generator). This is a great PRNG, and it's about 10x to 28x faster than UInt64.random.

If you wanted to implement this algorithm in Swift you could do it like this.

public final class Xoshiro256PlusPlus: RandomNumberGenerator { public typealias Seed = (UInt64, UInt64, UInt64, UInt64) private var s: Seed

public init(seed: Seed = Xoshiro256PlusPlus.randomSeed()) {
    self.s = seed
}

public static func randomSeed() -> Seed {
    return (
        UInt64.random(in: UInt64.min...UInt64.max),
        UInt64.random(in: UInt64.min...UInt64.max),
        UInt64.random(in: UInt64.min...UInt64.max),
        UInt64.random(in: UInt64.min...UInt64.max)
    )
}

/// Generates a pseduo random number.
public func next() -> UInt64 {
    let result = rotl(s.0 &+ s.3, 23) &+ s.0
    let t = s.1 << 17
    s.2 ^= s.0
    s.3 ^= s.1
    s.1 ^= s.2
    s.0 ^= s.3
    s.2 ^= t
    s.3 = rotl(s.3, 45)

    return result
}

private func rotl(_ x: UInt64, _ k: Int) -> UInt64 {
    return (x << k) | (x >> (64 - k))
}

}

This is just a simple example, but it's sufficient to explain how public and private interfaces
work in C and how to structure your Swift package. The C code is still
around 10% faster than the Swift version.

You can get the original Xoshiro code at [Xoshiro Generators](https://prng.di.unimi.it/).


## How to use C in Swift packages

Before anything else let's create the library package.

```bash
$ mkdir Xoshiro && cd Xoshiro
$ swift package init --name "Xoshiro" --type library
$ open Package.swift

Add two new targets, CLib and CLibTests and create the respective folders.

.target(
    name: "CLib"
),
.testTarget(
    name: "CLibTests",
    dependencies: ["CLib"]
),

Then let's add the actual code. By default, all header files added in the include folder are public.

Small helper function for rotating bits left

// Xoshiro/Sources/CLib/include/bitops.h
#ifndef bitops_h
#define bitops_h

#include <stdint.h>

extern uint64_t rotl(const uint64_t x, int k);

#endif /* bitops_h */

// Xoshiro/Sources/CLib/bitops.c
#include "bitops.h"

inline uint64_t rotl(const uint64_t x, int k) {
    return (x << k) | (x >> (64 - k));
}

Xoshiro256++ PRNG

// Xoshiro/Sources/CLib/include/xoshiro256.h
#ifndef xoshiro256_h
#define xoshiro256_h

#include <stdint.h>

void xoshiro256_seed(uint64_t s1, uint64_t s2, uint64_t s3, uint64_t s4);
uint64_t xoshiro256_next(void);

#endif /* xoshiro256_h */

// Xoshiro/Sources/CLib/xoshiro256.c
#include "xoshiro256.h"
#include "bitops.h"

static uint64_t s[4];

void xoshiro256_seed(uint64_t s1, uint64_t s2, uint64_t s3, uint64_t s4) {
    s[0] = s1;
    s[1] = s2;
    s[2] = s3;
    s[3] = s4;
}

uint64_t xoshiro256_next(void) {
    const uint64_t result = rotl(s[0] + s[3], 23) + s[0];
    const uint64_t t = s[1] << 17;

    s[2] ^= s[0];
    s[3] ^= s[1];
    s[1] ^= s[2];
    s[0] ^= s[3];
    s[2] ^= t;
    s[3] = rotl(s[3], 45);

    return result;
}

We can easily test if this is generating uniform distributions by creating a simple test that checks when we generate N random numbers if all possible results share roughly the same percentage.

import XCTest
@testable import CLib

final class CLibTests: XCTestCase {
    func testXoshiro() throws {
        let s = (1...4).map { _ in UInt64.random(in: UInt64.min...UInt64.max) }

        xoshiro256_seed(s[0], s[1], s[2], s[3])

        let results = NSCountedSet()
        let options: UInt64 = 100
        let samples = 1_000_000

        for _ in 0..<samples {
            results.add(UInt8(xoshiro256_next() % options))
        }

        for option in 0..<options {
            let actual = Double(results.count(for: option)) / Double(samples)
            let expected = 1.0 / Double(options)

            XCTAssertEqual(actual, expected, accuracy: 0.01)
        }
    }
}

There's one problem though, we have put everything in the include folder which means every function in both bitops.h and xoshiro256.h are public. The user doesn't need to know about uint64_t rotl(const uint64_t x, int k) in bitops.h, so how do we make it private?

One simple way is to move it to the root folder of its target, like this.

Xoshiro
├── Package.swift
└── Sources
    ├── CLib
    │   ├── bitops.c
    │   ├── bitops.h
    │   ├── xoshiro256.c
    │   ├── include
    │   │   └── xoshiro256.h
    └── Xoshiro
        └── Xoshiro.swift

This causes a 'bitops.h' file not found error.

To fix this you have to tell SPM where to search for header files by adding a header search path.

.target(
    name: "CLib",
    cSettings: [
        .headerSearchPath(".")
    ]
),

Personally, I prefer being more explicit and make it clear what's public and private by creating private and public folders.

Xoshiro
├── Package.swift
└── Sources
    ├── CLib
    │   ├── private
    │   │   ├── bitops.c
    │   │   ├── bitops.h
    │   │   └── xoshiro256.c
    │   └── public
    │       └── xoshiro256.h
    └── Xoshiro
        └── Xoshiro.swift

And then explicitly set the public headers' path.

.target(
    name: "CLib",
    publicHeadersPath: "public",
    cSettings: [
        .headerSearchPath("private")
    ]
),

How to add a Swift interface

Now the easiest part, wrapping the C interface with a Swift interface.

import CLib

public final class Xoshiro256: RandomNumberGenerator {
    public typealias Seed = (UInt64, UInt64, UInt64, UInt64)

    public init(seed: Seed? = nil) {
        if let seed {
            xoshiro256_seed(seed.0, seed.1, seed.2, seed.3)
        } else {
            xoshiro256_seed(
                UInt64.random(in: UInt64.min...UInt64.max),
                UInt64.random(in: UInt64.min...UInt64.max),
                UInt64.random(in: UInt64.min...UInt64.max),
                UInt64.random(in: UInt64.min...UInt64.max)
            )
        }
    }

    public func next() -> UInt64 {
        return xoshiro256_next()
    }
}

Conclusion

Using C in Swift packages is pretty straightforward once you know how to structure your code. Being able to use C is especially useful when dealing with low-level code that needs to be performant. Unsafe Swift syntax/API is also a bit too verbose for my liking, so I think it's great that I get to call out to C whenever needed.

I hope this tutorial has made integrating C in Swift packages less daunting.

Related Articles

• Apple - C Interoperability
• Apple - Unsafe Swift
• Apple - Safely manage pointers in Swift

Article Photo by Iñaki del Olmo

GitHub Branches and Web Servers Architecture Recap:

Before diving into the self-hosted runner setup, let's quickly recap our GitHub branches and web servers architecture. We have multiple branches, including "dev-deploy" and "prod-deploy." The "dev-deploy" branch has a dedicated GitHub runner for the development server. After successful testing on the dev server, we are ready to deploy changes to the production server using the "prod-deploy" branch.

![Github Branches](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/github.webp

![server architecture](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/aws.webp

Options for Self-Hosted Runners:

With self-hosted runners, we have three options to consider:

1. Repository-level: Runners dedicated to a single repository.
2. Organization-level: Runners capable of processing jobs for multiple repositories within an organization.
3. Enterprise-level: Runners assigned to multiple organizations in an enterprise account. Our example will be based on the repository-level, but you can choose the most suitable option based on your requirements.

Setting Up the Self-Hosted Runner:

Navigate to the repository settings page and click on "Actions," then select "Self-Hosted Runner." ![Github Settings](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/github-settings.webp

Choose the appropriate "Runner image" available on your server and select the corresponding "Architecture" (e.g., "Linux" and "x64"). ![select image](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/image-select.webp

The installation and configuration of the self-hosted runner involve three steps:

1. Download
2. Configure
3. Using your self-hosted runner.

![self hosted runner](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/Self-Hosted-Runner.webp

Download:

In this step, we need to log in to our own server and execute given commands.

Configure:

During the configuration, you may encounter some prompts. For the "group name," you can use the default one. Give a name to the runner, such as "production-deploy," which will appear in your runner page on the Github. Finally, set an label, for example, "prod-deploy." ![github action runner configure](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/configure.webp

Using your Self-Hosted Runner:

Now, you can start listening for job requests and to ensure continuous listening in the background by typing

sudo ./svc.sh install
sudo ./svc.sh start

With the self-hosted runner successfully configured and listening, any code changes merged into the "prod-deploy" branch will trigger the runner, facilitating deployment to our production server. ![github action configure](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/configured-successfull.webp

Creating the Deployment Automation YML File:

Step 1:

Create a folder named ".github" in the root of project directory. Inside the ".github" directory, create another folder named "workflows." Within this "workflows" folder, create a file named "prod_deploy.yml." The file extension must be ".yml." All the deployment automation code will be written in this file.

Step 2:

Provide a meaningful "name" for the workflow. This will help in identifying the workflow in the GitHub Actions interface.

![github action name](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/action-name.webp

Step 3:

Configure the "on" event to specify when the GitHub Action should run. For instance, we want the workflow to trigger when code is merged into the "prod-deploy" branch.

![github action name](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/action-run.webp

Step 4:

Set a default directory before running the "jobs" Since our root directory hasn't the WordPress theme, we can set the theme directory to avoid changing directories in each step.

![github action name](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/working-directory.webp

Step 5:

Define the "jobs" within the GitHub workflow. Each job contains a series of "steps," which can be actions or shell scripts. Here, we create a job and specify that it should run only if code is merged into the "prod-deploy" branch. We also add the "prod-deploy" label, which ensures that the self-hosted runner runs on the associated server.

Now let's see what it looks like when we write all the code together.

Let's break down each step in the deployment process:

• checkout: This step involves checking out our files into the virtual server using the GitHub checkout action from https://github.com/actions/checkout. This ensures that we have the latest code available for the deployment process.

• cache: The cache step is used to store cached data, allowing the theme build process to be faster after the first execution, provided the cache exists. For more information on the cache action, you can refer to https://github.com/actions/cache.

• Install Node dependency: In this step, we run an action to install the npm package dependencies required for our project. This ensures that the necessary Node modules are available for the subsequent build process.

• Install PHP dependency: Here, we execute an action to install the PHP package dependencies that our WordPress theme relies on. This ensures that the required PHP libraries and components are available for the theme build.

• Build The Theme: This step involves running an internal package command to build our WordPress theme. For this purpose, we are using the "Tonic starter" theme, which provides a solid foundation for our WordPress Theme development.

• Move Theme: After the theme is successfully built, we need to move it to the appropriate location on our server. This step utilizes basic shell commands to carry out the process. For example, it may involve removing the existing theme using and moving the current build folder to the appropriate location using

sudo rm -rf /theme-location/theme-name/
sudo mv * /theme-location/theme-name/

These commands should not be exposed publicly as they reveal sensitive server file paths. To keep these details secure, we can follow the steps below:

1. Go to the Repository and click on the "Settings" tab ![github Settings](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/setting-one.webp
2. In the left side: menu, you will find "actions" under the "secrets" menu. ![github setting](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/setting-two.webp
3. Click on "New repository secret".
4. Provide a "name" on that specific "secret" which we can access it by its name ${{ secrets.THEME_LOCATION }} here "THEME_LOCATION" was the name.
   

![github setting](/assets/img/articles/2023-12-14-How-to-setup-CI-CD-pipeline-for-WordPress-with-GitHub-Actions-and-AWS-part-2/theme-location.webp

With the self-hosted runner and GitHub workflow configured, we have established an automated deployment process for our WordPress theme. The self-hosted runner listens for changes in the "prod-deploy" branch, allowing us to deploy the latest changes to our production server. This setup ensures smoother and more efficient deployment while maintaining an extra layer of security through the VPN connection.

Resources

Article Photo by Praveen Thirumurugan

This journey's goal is to teach you a thorough understanding of JavaScript's underpinnings as well as the significant enhancements made by TypeScript. By the time you complete reading this article, you will have the knowledge required to apply both in your development projects. Now let's start this illuminating voyage.

Basic principles of JavaScript

JavaScript was first primarily used as a simple scripting language to improve web interactivity. However, it lacked a number of sophisticated capabilities found in contemporary programming languages. JavaScript has now evolved into a potent tool for front-end and back-end development with the addition of TypeScript.

Static typing, interfaces, enums, and generics are among the fundamental concepts introduced by TypeScript, a statically typed superset of JavaScript.

It is now an essential component of contemporary online and application development and supports functional, object-oriented, and asynchronous programming paradigms.

There are three main parts to a core JavaScript engine:

• Call stack
• Memory/variable environment
• Thread of execution

We will address each of these elements in turn in the following sections, offering in-depth analyses for a thorough grasp of how TypeScript improves JavaScript development.

Unlocking the Power of Reusable Code: Exploring JavaScript Functions with TypeScript

Within the realm of programming, functions perform as standalone programs when utilised, particularly in JavaScript (which is now further enhanced by TypeScript), and also code becomes more robust and dependable with greater type safety.

Two key considerations need to be made in order for this to function in TypeScript:

• Thread of Execution: This idea entails going through the TypeScript code for the function line by line, much like when you follow a recipe to the letter.

• Memory: This memory space enables the function to handle and store crucial information or data needed for its activities, much like a defined storage area. Type annotations in TypeScript facilitate the definition and administration of data structures, improving memory management.

Execution context is the term used in TypeScript to describe the confluence of these two fundamental components. To ensure that the function does its assigned tasks precisely and type-safely, this framework serves as its TypeScript operating environment.

A Detailed Overview of JavaScript and TypeScript Function Definition and Usage

JavaScript and its powerful extension TypeScript both employ the 'function' keyword and the function name to build functions. The syntax for the short arrow function (=>) is an alternative.

Just remember that in order to use a function in TypeScript—that is, to call, execute, or run it—it must always be enclosed in parenthesis. Code reliability and maintainability are enhanced when appropriate types of arguments are passed, thanks to TypeScript's type system. With TypeScript, you may define and use functions with more confidence that your code is valid.

Tracking Functions in the JavaScript CallStack

The "global" main function in JavaScript is where all code runs. To comprehend how JavaScript manages this global environment during the execution of a function, consider the "Call Stack".

The stack of plates that represents the call stack is made up of individual function calls. As an illustration:

function greet(name) {
  console.log(`Hello, ${name}!`);
}

function sayHello() {
  greet("Alice");
  greet("Bob");
}

sayHello();

• The execution of sayHello() adds to the call stack. Two calls performed inside of sayHello() to greet() have also been added to the Call Stack. The Call Stack tracks the order in which functions are executed.
• Handling Nested Functions: The Call Stack efficiently handles nested function management. Every function is removed from the stack after it is finished. For example:
  • sayHello() is now included in the Call Stack.
  • sayHello() now includes greet("Alice").
  • greet("Alice") terminates and is removed.
  • An additional greet("Bob") is attached.
  • greet("Bob") terminates and is removed.
  • Upon completion of sayHello(), the Call Stack is cleaned.

The Call Stack is essential for debugging and code optimisation since it ensures that functions in JavaScript's global context are executed in a logical order.

Comprehending Higher Order Functions & the JavaScript Callback

Passing a function as an argument to another function can look strange, especially when execution happens later. However, "callback functions." is the foundation for asynchronous programming with TypeScript and JavaScript.

Callback functions, or functions that are passed as arguments, are crucial for managing asynchronous tasks in JavaScript and TypeScript. For example, they specify what should happen after an API request is processed or a user activity is finished.

These days, higher-order functions like map, filter, and reduce are part of the toolkit in TypeScript and JavaScript. Also, Promises and the succinct async/await syntax have strong type support in TypeScript.

Demystifying JavaScript's Functional Programming: Unleashing Pro-Level Functions and Technical Interview Success.

Using functions, which makes it possible to use strong pro-level functions like reduce, filter, and map, is one of the most misunderstood topics in JavaScript. Functions are important to functional programming. A function is the foundation of a professional mid- to senior-level engineering interview and the means by which our code becomes more declarative and legible in Codesmith.

Using the following syntax, one could write a function that returns the square of 10:

function tenSquared() {
  return 10*10;
}

tenSquared() // 100

function nineSquared() {
  return 9*9;
}

nineSquared() // 81 😞

// And an 8 squared function? 125 squared?
// What principle are we breaking? DRY (Don't Repeat Yourself)

// We can generalize the function to make it reusable
function squareNum(num) {
  return num*num;
}

squareNum(10); // 100
squareNum(9);  // 81
squareNum(8);  // 64

Generalizing Functions

Code flexibility in JavaScript and TypeScript is largely dependent on the idea of generalising functions. "Parameters" (placeholders) allow us to perform our functionality on any data without having to decide on it beforehand.

When we run the function, supply a real value ('argument').

This same approach applies to higher-order functions.

• Until we run our function, we might not want to decide exactly what part of our functionality is.

Understanding Higher-Order Functions in JavaScript & TypeScript

Higher-order functions are essential to TypeScript and functional programming alike. They take functions as parameters and return functions as outcomes, which improves the flexibility and reusability of the code. Code is kept DRY (Don't Repeat Yourself) by using these utilities to simplify it. They make legible code possible through reduce, filter, and map operations. Add to this combination type safety with TypeScript.

Knowing higher-order functions is essential for professional interviews. Additionally, they are essential to asynchronous JavaScript, supporting ideas like async/await and Promises.

Whether for interviews or contemporary online programmes, knowing higher-order functions is essential to becoming proficient in JavaScript and TypeScript.

Understanding JavaScript's Arrow Functions:

// Introducing arrow functions - a shorthand way to save functions

function multiplyBy2(input) { return input * 2; }

const multiplyBy2 = (input) => { return input*2 }

const multiplyBy2 = (input) => input*2

const multiplyBy2 = input => input*2

const output = multiplyBy2(3) // 6

createMultiplier, a higher-order function, uses arrow syntax to return another function. This function multiplies a number by a predetermined amount. One specific example of a createMultiplier that doubles any input integer is multiplyBy2. By utilising multiplyBy2(3), it is evident that arrow functions can be employed to generate specialised functions that exhibit elegant behaviour, such as returning 6.

Understanding JavaScript Closures (Scope and Execution Context)

A closure in JavaScript is a function that works even after the outer function has finished running, allowing it to access variables from its outer scope.

In JavaScript, a closure is a function that returns from the execution of the outer function with access to variables from its outer scope.

Closures are highly helpful even though they were once regarded to be tough. They are now commonly understood. In order to construct reusable modules and improve code structure, they are usually required to maintain private and well-organized data. The usage and understanding of closures in modern JavaScript has increased.

Closures perform Memory-Related tasks

• We establish a live store of data (local memory/variable environment/state) for the function's execution context when our functions are invoked.
• After the function has finished running, its local memory is removed (apart from the returned value).

But what if our functions could save real-time data between calls?

This would allow our function definitions to be linked to a cache or permanent memory.

But it all starts with us returning a function from another function.

The Connection:

Upon definition, a function is bound to the Local Memory (sometimes called the "Variable Environment") in which it is defined.

The "Backpack":

function outer() {
    let counter = 0;
    function incrementCounter() {
        counter++;
    }
    return incrementCounter;
}

const myNewFunction = outer();
myNewFunction();
myNewFunction();

const anotherFunction = outer();
anotherFunction();
anotherFunction();

We obtain incrementCounter's function definition from outer and give it a new name, myNewFunction.

• The link to outer's live local memory is maintained; it is 'returned out' and attached to incrementCounter's function definition.
• As a result, the function outer's local memory is still linked to myNewFunction even if its execution context has long ago terminated.
• Running myNewFunction in global mode causes it to look first in its own local memory and then in its "backpack", as would be expected.

Thanks to closure, our functions now have enduring memory and an entire new toolkit for writing well-written code.

Understanding Asynchronous and Event Loop in JavaScript and TypeScript

Using TypeScript and JavaScript for asynchronous programming is necessary to create applications that are responsive and efficient. Asynchronous operations enable activities to be completed concurrently without interfering with the main thread of processing, in contrast to synchronous operations that run code one after the other.

An essential part of TypeScript and JavaScript, the event loop makes sure that asynchronous operations are carried out without blocking, resulting in responsive behaviour and economical use of resources.

Promises, the async/await syntax, and the event loop simplify asynchronous tasks in modern JavaScript and TypeScript. This improves the readability and maintainability of the code while guaranteeing smooth performance.

Important elements in this procedure consist of:

One essential ES6 feature that makes writing asynchronous programming easier is promises.

• Asynchronicity: The ability to handle concurrent tasks is what enables dynamic web applications.
• Event Loop: The asynchronous task management triage mechanism built into JavaScript. Web browser APIs, microtask queues, and callback queues are crucial parts of asynchronous process management.

The three primary components of the JavaScript engine are as follows:

• Thread of execution
• Memory and variable environment
• Call stack

Beyond the capabilities of ECMAScript 5 (ES5), advancements such as Promises and a more efficient Event Loop have been included in more recent iterations of JavaScript and TypeScript. In comparison to the ES5 era, these improvements increase the efficiency and maintainability of handling asynchronous operations and apply to both Node.js and web browsers.

ES6+ Solution with Promises

Promises provide a two-pronged approach to asynchronous task handling in modern JavaScript and TypeScript. They start web browser work in the background.

• Without delay, they provide a placeholder object, or promise.

These Promises adhere to the following guidelines for running code that is deferred asynchronously: A microtask queue is used to store promise-deferred functions, and a task queue (called the Callback queue) is used for callback functions once the associated Web Browser Feature (API) has finished. Functions are added to the Call stack (that is, executed) once all global code has run and the Call stack is empty. This state is kept an eye on by the Event Loop.

Promises let JavaScript and TypeScript do the following things when used with Web APIs, Callback & Microtask Queues, and the Event Loop:

• Develop non-blocking programmes that enable the execution of multiple threads of code concurrently without waiting.
• Take care of scheduling the completion of Browser features automatically, guaranteeing effective code execution. Act as the foundation for contemporary online applications, making it possible to create web solutions quickly and without stuttering.

Classes & Prototypes (OOP)

For object-oriented programming, JavaScript has historically employed prototypes, which may be confusing to developers used to working with class-based systems.

However, object-oriented programming in JavaScript became more recognisable to developers of other languages with the release of ECMAScript 6 (ES6), which introduced a class syntax. Although JavaScript classes still use prototypes internally, the new syntax offers a more organised and understandable method for generating and extending objects.

Important things to remember:

• Classes are a commonly used paradigm for structuring complex code in TypeScript and JavaScript.
• Prototype chain, an efficient tool in and of itself that enables the replication of Object-Oriented Programming (OOP), is one such background-working component.
• The difference between prototype and proto must be understood for effective OOP in JavaScript and TypeScript.
• Programming is made easier by using the new and class keywords, which automate the generation of objects and methods.

proto:

Every JavaScript object, including arrays, functions, and other objects, has a property called proto. JavaScript initially verifies that a property or function on an object actually exists before allowing you to access it.

prototype:

In TypeScript and JavaScript, every function has the property prototype. The prototype property of a constructor function becomes the prototype of the newly generated object when it is defined and instances are created using the new keyword.

A Guide for Writing and Using Code

1. Save data (for example, user 1 and user 2's quiz scores).
2. Use the data to run code (functions) (e.g., enhance user 2's score)

Simple! Then why is development challenging? I have to save a lot of users in a quiz game, along with administrators, quiz questions, quiz results, league tables, and other data and functionality.

Where's the functionality when I need it in 100,000 lines of code?

• How can I ensure that the functionality is only applied to appropriate data?

To make things simple, let's say I save each user's specific data in my app:

How might my entire set of features and information be in one place?

// Solution 1. Generate objects using a function

function userCreator(name, score) {
    const newUser = {};
    newUser.name = name;
    newUser.score = score;
    newUser.increment = function() {
        newUser.score++;
    };
    return newUser;
}

const user1 = userCreator("Will", 3);
const user2 = userCreator("Tim", 5);
user1.increment();

Problems: Our computer has to allocate memory for all of our data and features each time we add a new user. But all we are doing is copying functions. Does a better approach exist?

Advantages: It's easy to understand and consider!

// Solution 2: Using the prototype chain

function userCreator(name, score) {
    const newUser = Object.create(userFunctionStore);
    newUser.name = name;
    newUser.score = score;
    return newUser;
}

const userFunctionStore = {
    increment: function() { this.score++; },
    login: function() { console.log("Logged in"); }
};

const user1 = userCreator("Will", 3);
const user2 = userCreator("Tim", 5);
user1.increment();

Understanding TypeScript to Enhance JavaScript Development

We studied the hard JavaScript sections as well as the basic concepts and challenges of this dynamic language. All things considered, the rise in popularity of TypeScript, is one of the most significant advancements in JavaScript lately.

Adding TypeScript to a project has several advantages:

1. Maintainability of Code
2. Enhanced Developer Productivity
3. Improved Collaboration
4. Scalability
5. Reduced Number of Bugs
6. Ecosystem and Community

Furthermore, TypeScript facilitates faster code reviews, more comprehensible code, and easier onboarding of new team members—all of which enhance developer collaboration.

Conclusion

As a result of our thorough investigation of JavaScript and its potent extension, TypeScript, we have gained a profound understanding of the underlying ideas that underpin this adaptable programming language. We have studied the management of intricate processes by JavaScript, thanks to its asynchronous features such as callbacks and higher-order functions. The event loop and closures, which are crucial to comprehending JavaScript's execution mechanism, have also been clarified.

We are more equipped to take on challenging coding tasks as we continue to explore the nuances of JavaScript, enhanced by TypeScript's extensions. Our objective is to develop applications that are resilient, adaptable, and scalable to the demands of the ever-changing digital landscape. So, fellow coders, let's embrace TypeScript and JavaScript to the fullest and use their combined strength to innovate and change the digital landscape.

Happy Coding!

Article Photo by Unsplash

This is the initial release of AndroidX Bluetooth APIs that provides a Kotlin API surface covering Bluetooth LE scanning and advertising, and GATT client and server use cases. It provides a minimal API surface, clear thread model with async and sync operations, and ensures all methods be executed and provides the results.

Sounds very promising, but less talk, more action, let's dive deeper into what AndroidX Bluetooth library brings to the table!

How to use AndroidX Bluetooth library

First of all, import the artifact in your build.gradle.

implementation "androidx.bluetooth:bluetooth:1.0.0-alpha01"

Note: current minSdk of the artifact might be too high (33 at the point of writing this article) and it is intended. They are gradually lowering the minSdk as part of integration testing/for people to not use it in production since it is still in alpha.

Working with the library

With the basic introductions done, it is time to buckle up and start coding! For the sake of simplifying the article we will skip the permission handling, checking whether device has bluetooth and if it is enabled and move straight to the specifics of the AndroidX Bluetooth library.

BluetoothLe class

BluetoothLe class is the main entry point of the library. Internally, it contains GattClient and GattServer classes which are responsible for the GATT client and GATT server role, respectively. Every usecase you have will go through this class.

Currently BluetoothLe class has 4 main methods:

• scan - launches peripheral scanner
• connectGatt - handles the connection to the peripheral
• advertise - starts BLE advertiser
• openGattServer - starts GATT server

To instantiate BluetoothLe class, simply call its constructor where you have to supply it with Context.

GATT Client/Central

Now that we have our BluetoothLe class set up, let's dive deeper into GATT client/central role and its main responsibilities. That includes scanning for peripherals, connecting and communicating with them via read/write operations.

Scanning

Let's start with the first operation we have to do when doing any business with the bluetooth devices - the scan.

To launch a scan, simply call the bluetoothLe.scan(List<ScanFilter>) method. It has one parameter, which is a list of scan filters and it returns a Flow of ScanResults, which is very convenient! No more callback hells, instead we have a cold flow that emits devices once they are detected by the hardware scanner (and passes your scan filters).

You can notice that there is no way to set ScanSettings for the scan since it doesn't exist as a parameter for the function. ScanSettings allows developers to set the scan mode, match mode, PHY, etc. Since under the hood it still uses framework's bluetoothLeScanner, it is still mandatory to specify, but for now it is not implemented in AndroidX library and it uses default ScanSettings instead:

val scanSettings = ScanSettings.Builder().build()  
bleScanner?.startScan(fwkFilters, scanSettings, callback)

There is also no method to stop the scan, but by checking the code inside the library we can see the following code:

fun scan(filters: List<ScanFilter> = emptyList()): Flow<ScanResult> = callbackFlow {

  // scan method body
  
  awaitClose {  
    bleScanner?.stopScan(callback)  
  }  
}

Notice the awaitClose? It suspends the function/coroutine until it is either closed or cancelled and invokes the code block. Therefore in order to stop the scan we would need to cancel the Job that collects the scan results. A draft of the code would look like this:

private var scanJob: Job? = null

fun startScanning() {  
  scanJob = viewModelScope.launch {
    try {
      bluetoothLe.scan().collect { scanResult ->
        // do something with the scan results
      } 
    } catch (exception: Exception) {
      // handle scan errors
    }
  }
}  
  
fun stopScanning() {  
  scanJob?.cancel()  
  scanJob = null  
}

Looks quite tidy and clean! Now you can take your device from the scan results and connect to it.

Connecting

To connect to the peripheral, one should make use of the connectGatt method.

bluetoothLe.connectGatt(BluetoothDevice, suspend GattClientScope.() -> R)`

There are two parameters for developers to fiddle with:

• BluetoothDevice - library's wrapper around framework's BluetoothDevice. You can receive it from the ScanResult when scanning for peripherals (more details in the scanning section above)
• suspend GattClientScope.() -> R - lambda which is marked as suspend. It invokes once the connection with the peripheral is established. Since it is also an extension of GattClientScope you can call various methods on it to communicate with the remote device.

Therefore, to connect to the peripheral, you'd need to have something like the following:

// every communication is going through this object once the device is connected  
private var gattClient: BluetoothLe.GattClientScope? = null

try {
  bluetoothLe.connectGatt(bluetoothDevice) {  
    gattClient = this
  }
} catch (exception: Exception) {
  // handle connection errors
}

Since lambda is an extension of GattClientScope, we can save it to the outside variable, similar to how you'd save BluetoothGatt object when working with the framework's BLE SDK.

One thing to keep in mind is that the connection will be dropped the moment suspend GattClientScope.() -> R lambda finishes. To negate that, you need to suspend the lambda at the end by calling awaitCancellation(). That will also allow you to control when to disconnect from the device, similarly to how we stopped the scan earlier. Therefore, the connectivity block might look like this:

// every communication is going through this object once the device is connected  
private var gattClient: BluetoothLe.GattClientScope? = null
private var connectJob: Job? = null

fun connect() {
  connectJob = coroutineScope.launch {
    try {
      bluetoothLe.connectGatt(bluetoothDevice) {  
        gattClient = this
        awaitCancellation()
      }
    } catch (exception: Exception) {
      // handle connection errors
    }
  }
}

fun disconnect() {
  connectJob?.cancel()  
  connectJob = null  
}

After it establishes the connection, the library automatically does the initial flow of setting up the GATT. The flow looks like this: connection success >> request MTU >> 515 >> discover services >> we're ready! The only flaw in this is that if the MTU request fails, then the whole connection is dropped. Granted that the MTU request is not a mandatory operation, that seems quite extreme (at least I wasn't been able to connect to my virtual peripheral, because it was failing the MTU request and had to find another way to host a GATT server)

Other than that, the whole connectivity process was smooth and I was able to have a working connectable device in no time.

Communicating

Once the connection is established, it is time to do some communication, i.e. read/write. Previously we saved GattClientScope to an outer variable. This class has a minimum amount of capabilities to start communicating with the remote device. For example, you can check all the characteristics of a certain service by calling the following method:

gattClient.services.first().characteristics

Or when you need to read from characteristic or write to it, you can do it all by using the same GattClientScope:

// Find the characteristic
// Please note, that this is just an example, you would need to find characteristic
// by its' UUID
val myCharacteristic = gattClient.services.first().characteristics.first()

// read characteristic; this is a suspend function
val result = gattClient.readCharacteristic(myCharacteristic)

// write characteristic; this is a suspend function
gattClient.writeCharacteristic(characteristic, valueToWrite.toByteArray())

Besides those standard operations, GattClientScope also able to subscribe to characteristic and return flow of values:

// not a suspend function; returns Flow instead
val byteArrayFlow = gattClient.subscribeToCharacteristic(myCharacteristic)
coroutineScope.launch {
  byteArrayFlow.collect { byteArray ->
    // process byte arrays coming from peripheral's notifications 
  }
}

Operation queuing

Developers who worked with Android BLE SDK know that it is forbidden to perform operations in rapid-fire succession. Essentially, if you are not using some third-party library, you would need to implement a queuing mechanism that would execute your BLE operations sequentially, one-by-one (any operation, such as discover services, request MTU, read/write, etc). If you don't do that, chances are that some of the operations will be simply dropped/ignored causing software misbehavior.

In AndroidX Bluetooth that seems to be finally taken care of and developers can finally not worry about creating a queuing mechanism for their operations. That is due to the fact that internally the library uses Kotlin's Mutex. Mutex essentially suspends the function until the shared resource is free. But the important part there is that the created mutex is fair: lock is granted in first come, first served order. In another words, Mutex allows us to create a queue for BLE operations. The code snippet below was taken from the AndroidX Bluetooth source code, modified for article's simplicity and enriched with some comments explaining how it works.

val gattScope = object : BluetoothLe.GattClientScope {

  // create Mutex
  val taskMutex = Mutex()
  
  // helper function to make use of Mutex
  // will be used down below in readCharacteristic function
  suspend fun <R> runTask(block: suspend () -> R): R {  
    taskMutex.withLock {  
      return block()  
    }  
  }  
  
  override suspend fun readCharacteristic(characteristic: GattCharacteristic): Result<ByteArray> {  
    // the runTask function locks the mutex  
    return runTask {
      // read the characteristic safely, inside the mutex
      fwkAdapter.readCharacteristic(characteristic.fwkCharacteristic)  
      // the function below suspends until we get the result from the read operation 
      val res = takeMatchingResult<GattClient.CallbackResult.OnCharacteristicRead>(  
        callbackResultsFlow  
      ) {  
        it.characteristic == characteristic  
      }  
  
      // return the result, unlock the mutex
    }  
  }  
}

Mutex is used in read, write and subscribe operations, which means that if you somehow rapid-fire read operations along with discover services, request MTU or some other, then you still can theoretically lose some requests, but we are going into an edge case now since AndroidX Bluetooth doesn't currently have the capability to enqueue discover services request through their API.

GATT Server/Peripheral

Besides being the client most of the time, Android can also be a server! Since Android 5+ some of the devices can act as peripheral/server, but be aware that not all of the devices support that on the hardware level. You can find what phones support that in that list.

To open the GATT server, you would need to call openGattServer method.

bluetoothLe.openGattServer(List<GattService>, suspend GattServerConnectScope.() -> R)

There are multiple design similarities with the previously described operations. Let's dig deeper! There are two parameters, similarly to how client connection works:

• List<GattService> - services and characteristics that the server will hold.
• suspend GattServerConnectScope.() -> R - lambda which is marked as suspend. It executes once the server is opened. Again, similarly to how client connection works, it's also an extension of GattServerConnectScope.

GattServerConnectScope doesn't have much to offer from the first glance. You can update services by invoking the corresponding method updateServices(List<GattService>) and listen to the connect requests by utilizing connectRequests Flow. The latter one is a little bit more interesting since you are collecting GattServerConnectRequests. It contains the BluetoothDevice from which the request is coming from and methods to accept (has yet another lambda) or reject the request. If you accept the request, you are basically establishing the connection with the client and can start processing read/write requests, notify the client via BLE Notifications, etc.

Therefore, to open the GATT server, you'd need to have something like the following:

bluetoothLe.openGattServer(gattServerServices) { // opens GATT server
   connectRequests.collect { connectRequest ->   // collets clients' connection requests
     connectRequest.accept {                     // accepts the connection request (you can also reject it)
       requests.collect { serverRequest ->       // collects client's read/write requests, etc
         // do something with server request
       }
     }
   }
}

You can further improve the snippet above by adding CoroutineScope.launch before opening GATT server like we did in the scanning/connectivity before, cancelling it whenever you need to close the server and/or adding parallelism by launching a separate coroutine for each connectRequest, etc. The sky is the limit!

Advertiser

Lastly, you can also have a device advertiser. You can do it by calling the advertise method.

bluetoothLe.advertise(AdvertiseParams, (suspend (@AdvertiseResult Int) -> Unit)?)

This method has two parameters:

• AdvertiseParams - provides a way to adjust advertising preferences and advertise data packet.
• (suspend (@AdvertiseResult Int) -> Unit)? - an optional block of code that is invoked when advertising is started or failed.

AdvertiseParams has a lot of preferences to modify, let's go over them one-by-one:

• shouldIncludeDeviceAddress - sets whether the device address will be included in the advertisement packet.
• shouldIncludeDeviceName - sets whether the device name will be included in the advertisement packet.
• isConnectable - sets whether the advertisement will indicate connectable.
• isDiscoverable - sets whether the advertisement will be discoverable. Note that it would be ignored under API level 34 and isConnectable would be used instead.
• durationMillis - sets advertising duration in milliseconds. Supports values from 0 to 655350.
• manufacturerData - sets a map of company identifiers to manufacturer specific data.
• serviceData - sets a map of 16-bit UUIDs of the services to corresponding additional service data.
• serviceUuids - sets a list of service UUIDs to advertise.

As you can see, there are a lot of preferences to fiddle with, but more importantly, under the hood, it has all the conditionals to ensure backwards compatibility so you don't need to have a hell of API-level version checks. It is all already done! And that applies to all of the operations we have gone through in this article such as scanning, connectivity, hosting a GATT server, etc!

Lastly, this is how the code would look like if you need to advertise your device:

val advertiseParams = AdvertiseParams(  
  shouldIncludeDeviceAddress = false,  
  shouldIncludeDeviceName = false,  
  isConnectable = false,  
  isDiscoverable = false,  
  durationMillis = 0,  
  manufacturerData = mutableMapOf(),  
  serviceData = mutableMapOf(),  
  serviceUuids = mutableListOf(),  
)
bluetoothLe.advertise(advertiseParams) {
  // do something with the AdvertiseResult
}

Room for Improvement

There is definitely room for improvement for the library. Some of them include:

• Consistency. Some methods of BluetoothLe suspend the function and you need to cancel it to interrupt the operation and other methods don't suspend the function and you have to add awaitCancellation yourself (looking at you connectGatt method 👀).
• Make request MTU not mandatory when establishing the connection with the peripheral.
• Allow developers to modify ScanSettings when scanning for peripherals, but I am sure it will come in future versions.

Conclusion

To wrap it up, here are the key takeaways for the AndroidX Bluetooth library:

• AndroidX Bluetooth is a Jetpack library that adds synchronous way to handle bluetooth operations.
• AndroidX Bluetooth allows developers to scan, connect, communicate, create their own GATT servers and advertise their devices.
• AndroidX Bluetooth is heavily leveraging Kotlin Coroutines, which makes collecting results and/or cancelling operations relatively easy.
• AndroidX Bluetooth handles all API level version specifics for you. Less version checks conditionals = good.
• It is still in alpha and a lot of features are missing and there is room for improvement, but it is a very good start and makes third-party Android BLE libraries a bit less mandatory for the real world development

_ Article Photo by Sten Ritterfeld on Unsplash

The Secure Enclave is a pivotal part of most modern Apple devices. On your iPhone, it's what keeps secure your most sensitive information. Information such as your biometric data, to use with Face & Touch ID and your payment information that can be stored within Apple Pay. It's a dedicated distinct hardware component, isolated from the main processor making it super secure. It being isolated, means it can't be directly accessed by the main processor or any other applications that may find a way to gain un-authorized access. It also means this data will only ever be accessible via the device that encrypted it. In essence, the Secure Enclave acts as a secure vault within the iPhone's hardware architecture, dedicated to managing and protecting sensitive information critical to the device's security features.

Often people's first assumption is that the Secure Enclave includes storage for all types of data, similar to like how we use Apple Keychain or UserDefaults services, but it doesn't. What it can do is use Apple's distinct Secure Enclave algorithms to provide and store encryption keys and other cryptographic material. The Secure Enclave only supports P256 elliptic curve keys and offers around 4MB of storage, which is plenty for key storage.

Recently we had a task to utilise the secure enclave to enhance security for the data an application stored and provide an extra layer to protect that sensitive information. Interested in knowing how we did it? Then let's go...

How we can utilise it

As previously mentioned, the Secure Enclave is not somewhere we can simply store any type of data we have.

In fact, for our utilisation, we are going to continue to use the keychain for storage, but instead of simply storing the raw data in the keychain, we are going to build a wrapper that can make use of the Secure Enclave to encrypt data before it's stored and decrypt at the point it's fetched. This provides us with the security that even if someone somehow accesses that data in the software, you can rest assured that it is encrypted and protected at the hardware level. That's pretty neat. Not only that, with features provided by Apple's CryptoKit library, we can even put restraints on when this data can be decrypted, for example, when the device is locked or when your app is in the background.

We're going to step through the code seeing how we can store our data while encrypting and decrypting it with the Secure Enclave. If you would like to see the full EnclaveWrapper code, it is available at the bottom of this page.

Initialising our EnclaveWrapper

Before we can start, make sure you've imported CryptoKit, LocalAuthentication & KeychainAccess from Apple. We will be making use of all of them in our EnclaveWrapper.

import Foundation
import CryptoKit
import LocalAuthentication
import KeychainAccess

Our EnclaveWrapper requires a dependency on somewhere to store our EnclaveKey. In our case, we want to use our KeychainWrapper but you may have other places you want to store it or you may want this dependency decoupled for testing purposes. Because of this, the EnclaveWrapper has a property keyAccess that conforms to the EnclaveWrapperKeyAccess protocol.

The EnclaveWrapperKeyAccess protocol allows us to read and write our EnclaveWrapperKey when we wrap and unwrap our data. Below we have a class that uses our KeychainWrapper to align with the EnclaveWrapperKeyAccess protocol for use with the EnclaveWrapper.

public class EnclaveWrapper {
    let keyAccess: EnclaveWrapperKeyAccess
    
    public init(keyAccess: EnclaveWrapperKeyAccess) {
        self.keyAccess = keyAccess
    }
    
    ...
}    

public protocol EnclaveWrapperKeyAccess {
    var enclaveKey: EnclaveWrapperKey? { get }
    func updateKey(_ key: EnclaveWrapperKey) throws
}

public class EnclaveKeychainAccess: EnclaveWrapperKeyAccess {
    private let privateKeyTag = "com.demoApp.secureEnclaveDemo.secureEnclavePrivateKey"
    
    public init() {}
    
    public var enclaveKey: EnclaveWrapperKey? {
        return try? Keychain.shared.get(key: privateKeyTag, ofType: EnclaveWrapperKey.self)
    }
    
    public func updateKey(_ key: EnclaveWrapperKey) throws {
        try Keychain.shared.set(value: key, key: privateKeyTag)
    }
}

Sealing our data

The seal data function we're going to look into...

//seals data in AES.GCM seal and returns combined Data
public func sealData<T: Encodable>(data: T) throws -> Data {
    guard SecureEnclave.isAvailable else {
        throw EnclaveWrapperError.secureEnclaveUnavailable
    }
    
    //fetch cached key if we have one
    var enclaveKey: EnclaveWrapperKey? = keyAccess.enclaveKey
    
    //if we dont have a cached key, generate a new one
    if enclaveKey == nil {
        let newPrivateKey = try SecureEnclave.P256.KeyAgreement.PrivateKey()
        let newKey = EnclaveWrapperKey(privateKeyDataRepresentation: newPrivateKey.dataRepresentation,
                                       salt: generateRandomSalt())
        //cache it for future use
        try keyAccess.updateKey(newKey)
        enclaveKey = newKey
    }
    
    //check we have keys at this point, either from cache or generated
    guard let validKey = enclaveKey  else {
        throw EnclaveWrapperError.keyGenerationFailed
    }
    
    let symmetricKey = try validKey.symmetricKey()
    let encodedData = try JSONEncoder().encode(data)
    return try AES.GCM.seal(encodedData, using: symmetricKey).combined!
}

Hardware checks

First off we need to check the secure enclave is available on the device we're using. This will be any device that uses the A7 chip (iPhone 5S or later). If we can't use the enclave, we simply throw an error.

guard SecureEnclave.isAvailable else {
    throw EnclaveWrapperError.secureEnclaveUnavailable
}

Key Handling

Secondly, we will check if we have an enclave private key already cached. We do this by using our keyAccess property we set up at initialisation.

//fetch cached key if we have one
var enclaveKey: EnclaveWrapperKey? = keyAccess.enclaveKey

Generating a key

In the case that this is a users first run through, they wont have a stored key so we will need to create one for them.

For this we have the EnclaveWrapperKey struct. This combines our private key along with salt data that will later be used to generate a symmetric key for sealing and unlocking our data.

public struct EnclaveWrapperKey: Codable {
    let privateKeyDataRepresentation: Data
    let salt: Data
}

Below you can see how we create a new EnclaveWrapperKey.

• We use CryptoKit to generate a new public key using the Secure Enclave.

let newPrivateKey = try SecureEnclave.P256.KeyAgreement.PrivateKey()

• We then generate random salt data that is used for the derivation of the symmetric key.

private func generateRandomSalt() -> Data {
    var data = Data(count: 32)
    _ = data.withUnsafeMutableBytes { buffer in
        SecRandomCopyBytes(kSecRandomDefault, 32, buffer.baseAddress!)
    }
    return data
}

• We then combine the two to make our new EnclaveWrapperKey. We then need to cache a representation of this key using keyAccess once more. It's important to remember that the data representation stored in the Keychain will not simple be enough to unlock our data. We will need to interact with the Secure Enclave using this data to get our PrivateKey.

let newKey = EnclaveWrapperKey(privateKeyDataRepresentation: newPrivateKey.dataRepresentation,
                                       salt: generateRandomSalt())
//cache it for future use
try keyAccess.updateKey(newKey)

Utilising a cached key

Presuming this is our second time storing data and we have already generated a key previously, meaning one is available in the Keychain. We can use the cached key found in our keyAccess dependency.

var enclaveKey: EnclaveWrapperKey? = keyAccess.enclaveKey

Our EnclaveWrapperKey has two helpful functions. The first being a private function, that allows us to retrieve our SecureEnclave key.

func privateKey() throws -> SecureEnclave.P256.KeyAgreement.PrivateKey

This function can use the EnclaveWrapperKey.privateKeyDataRepresentation to interact with the SecureEnclave using CryptoKit and return our SecureEnclave.P256.KeyAgreement.PrivateKey.

Another thing to note is, you need to provide an authentication context whenever accessing these private keys. Depending on the key and configurations, you may need to display context and reasoning as to why you need to authenticate to access this key using biometric or password checks.

private func privateKey() throws -> SecureEnclave.P256.KeyAgreement.PrivateKey {
    let context = LAContext()
    context.localizedReason = "Authenticate to retrieve the private key"
    let privateKey = try SecureEnclave.P256.KeyAgreement.PrivateKey(dataRepresentation: self.privateKeyDataRepresentation, authenticationContext: context)
    return privateKey
}

Symmetric Keys

The second EnclaveWrapperKey function is publicly accessible. This is because it generates a symmetric key by combining the privateKey() function key and the salt data stored within the EnclaveWrapperKey. It first takes a private key and generates a shared secret using its private and public key combined. With this shared secret, we can then derive a symmetric key using SHA256 and the salt. We store the salt and the key together in our EnclaveWrapperKey as one without the other would not work. Both are required to derive our symmetric key. This symmetric key is what will be required for sealing and opening our data.

func symmetricKey() throws -> SymmetricKey {
    //generate symmetric key from shared secret
    let privateKey = try privateKey()
    let sharedSecret = try privateKey.sharedSecretFromKeyAgreement(with: privateKey.publicKey)
    
    // Perform HKDF to derive the symmetric key from the shared secret
    return sharedSecret.hkdfDerivedSymmetricKey(using: SHA256.self,
                                                salt: self.salt,
                                                sharedInfo: Data(),
                                                outputByteCount: 32)
}

For our case, we can use the public key that is accessible via the private key as we are not sharing this data with anyone else. It is purely for security enhancements within our device. Often when sharing data with others, users will share their public keys while keeping their private keys, exactly that, private! Once this key exchange has taken place, both parties will be able to access the data as long as the shared secret is a combination of both their keys. This would be a more common use case and why we need to interact with the Secure Enclave in this way. You can read more about the Diffie–Hellman key exchange here.

The concealing

Now we have our symmetric key, we can encode and then seal our data up by calling AES.GCM.seal(encodedData, using: symmetricKey).combined!

let encodedData = try JSONEncoder().encode(data)
return try AES.GCM.seal(encodedData, using: symmetricKey).combined!

Now I have my seal data function, I can simply call it in my app on any Encodable object and store my encrypted data like so...

let wrappedData = try EnclaveWrapper.sealData(data: user)
storage.set(wrappedData, key: LocalConstants.userStorageKey)

Opening our sealed data

public func openSealedData<T: Decodable>(type: T.Type, data: Data) throws -> T {
    guard SecureEnclave.isAvailable else {
        throw EnclaveWrapperError.secureEnclaveUnavailable
    }
    
    guard let enclaveKey = keyAccess.enclaveKey else {
        //if we dont have a cached key, opening sealed data will not work
        throw EnclaveWrapperError.keysNotFound
    }
    
    let symmetricKey = try enclaveKey.symmetricKey()
    
    let sealedBox = try AES.GCM.SealedBox(combined: data)
    let data = try AES.GCM.open(sealedBox, using: symmetricKey)
    return try JSONDecoder().decode(T.self, from: data)
}

Now we've got our sealed data, there will come a time, where we need to unlock it and access it once more. Lets look into how we can do that.

Just like before, we first need to check the enclave is available and fetch our cached enclaveKey using keyAccess.enclaveKey. Unlike before there is no need to generate new keys if it can't be found. There is no way we'll be able to unlock our data if we don't have the keys and salt that were used to seal it. In this case, we will just throw an error.

guard let enclaveKey = keyAccess.enclaveKey else {
    //if we dont have a cached key, opening sealed data will not work
    throw EnclaveWrapperError.keysNotFound
}

We then access our symmetric key using our same symmetricKey() function.

let symmetricKey = try enclaveKey.symmetricKey()

Finally, we can simply create a sealed box using the data passed in like so...
try AES.GCM.SealedBox(combined: data)

We can then open the box with our symmetric key and decode our data. It's important to remember that in order to use our Enclave Wrapper, all data types will need to conform to Codable.

let sealedBox = try AES.GCM.SealedBox(combined: data)
let data = try AES.GCM.open(sealedBox, using: symmetricKey)
return try JSONDecoder().decode(T.self, from: data)

With this function complete, I can use it in my app to fetch back my User object (that conforms to codable) that I sealed previously...

func getUser() -> User? {
    if let wrappedData = storage.get(key: LocalConstants.userStorageKey, ofType: Data.self) {
        return try? EnclaveWrapper.openSealedData(type: User.self, data: wrappedData)
    }

    return nil
}

Conclusion

Now we have an EnclaveWrapper and have decoupled our dependecies on Keychain, we can even write a test to assert our sealing and opening of our data works as expected. Below we use the EnclaveWrapper to encrypt our Codable data, and once encrypted it can't be decoded into its original form until it's opened with the EnclaveWrapper. We can then test the data matches.

import XCTest
@testable import EnclaveWrapper

final class EnclaveWrapperTests: XCTestCase {

    struct User: Codable {
        let firstName: String
        let lastName: String
    }
    
    class MockEnclaveKeychainAccess: EnclaveWrapperKeyAccess {
        var enclaveKey: EnclaveWrapperKey?
        
        func updateKey(_ key: EnclaveWrapperKey) throws {
            enclaveKey = key
        }
    }
    
    func testNewEncryption() {
        //mock object we will encrypt and test with
        let user = User(firstName: "Joe", lastName: "Bloggs")
        
        let enclaveWrapper = EnclaveWrapper(keyAccess: MockEnclaveKeychainAccess())
        
        guard let encryptedUser = try? enclaveWrapper.sealData(data: user) else {
            XCTFail("Failed to encrypt object")
            return
        }
        
        //prove we can't use encypted data
        let encryptedUserDecoded = try? JSONDecoder().decode(User.self, from: encryptedUser)
        XCTAssertNil(encryptedUserDecoded)
        
        //now decrypt user using open sealed data
        let decryptedUser = try? enclaveWrapper.openSealedData(type: User.self, data: encryptedUser)
        
        //assert data is equal
        XCTAssertEqual(user.firstName, decryptedUser?.firstName)
        XCTAssertEqual(user.lastName, decryptedUser?.lastName)
    }
}

That about wraps it up! (Excuse the pun).

With just those few functions, I can now safely cache all my data with the added dependency of the Secure Enclave hardware capabilities. If the data in the keychain was ever compromised, we can now safely know, it will be encrypted with the Secure Enclave. If you want to read more about the secure enclave, you can find some links to more information below. You can also find the entire EnclaveWrapper code along with a KeychainWrapper that I used with it.

import Foundation
import CryptoKit
import LocalAuthentication
import KeychainAccess

//MARK: - EnclaveWrapperKey

public struct EnclaveWrapperKey: Codable {
    let privateKeyDataRepresentation: Data
    let salt: Data
}

extension EnclaveWrapperKey {
    private func privateKey() throws -> SecureEnclave.P256.KeyAgreement.PrivateKey {
        let context = LAContext()
        context.localizedReason = "Authenticate to retrieve the private key"
        let privateKey = try SecureEnclave.P256.KeyAgreement.PrivateKey(dataRepresentation: self.privateKeyDataRepresentation, authenticationContext: context)
        return privateKey
    }
    
    func symmetricKey() throws -> SymmetricKey {
        //generate symmetric key from shared secret
        let privateKey = try privateKey()
        let sharedSecret = try privateKey.sharedSecretFromKeyAgreement(with: privateKey.publicKey)
        
        // Perform HKDF to derive the symmetric key from the shared secret
        return sharedSecret.hkdfDerivedSymmetricKey(using: SHA256.self,
                                                    salt: self.salt,
                                                    sharedInfo: Data(),
                                                    outputByteCount: 32)
    }
}

//MARK: - EnclaveWrapper

public enum EnclaveWrapperError: Error {
    case secureEnclaveUnavailable
    case saltNotFound
    case keyGenerationFailed
    case keysNotFound
}

public class EnclaveWrapper {
    let keyAccess: EnclaveWrapperKeyAccess
    
    public init(keyAccess: EnclaveWrapperKeyAccess) {
        self.keyAccess = keyAccess
    }
    
    //MARK: - Public functions
    
    //seals data in AES.GCM seal and returns combined Data
    public func sealData<T: Encodable>(data: T) throws -> Data {
        guard SecureEnclave.isAvailable else {
            throw EnclaveWrapperError.secureEnclaveUnavailable
        }
        
        //fetch cached key if we have one
        var enclaveKey: EnclaveWrapperKey? = keyAccess.enclaveKey
        
        //if we don't have a cached key, generate a new one
        if enclaveKey == nil {
            let newPrivateKey = try SecureEnclave.P256.KeyAgreement.PrivateKey()
            let newKey = EnclaveWrapperKey(privateKeyDataRepresentation: newPrivateKey.dataRepresentation,
                                           salt: generateRandomSalt())
            //cache it for future use
            try keyAccess.updateKey(newKey)
            enclaveKey = newKey
        }
        
        //check we have keys at this point, either from cache or generated
        guard let validKey = enclaveKey  else {
            throw EnclaveWrapperError.keyGenerationFailed
        }
        
        let symmetricKey = try validKey.symmetricKey()
        let encodedData = try JSONEncoder().encode(data)
        return try AES.GCM.seal(encodedData, using: symmetricKey).combined!
    }
    
    public func openSealedData<T: Decodable>(type: T.Type, data: Data) throws -> T {
        guard SecureEnclave.isAvailable else {
            throw EnclaveWrapperError.secureEnclaveUnavailable
        }
        
        guard let enclaveKey = keyAccess.enclaveKey else {
            //if we dont have a cached key, opening sealed data will not work
            throw EnclaveWrapperError.keysNotFound
        }
        
        let symmetricKey = try enclaveKey.symmetricKey()
        
        let sealedBox = try AES.GCM.SealedBox(combined: data)
        let data = try AES.GCM.open(sealedBox, using: symmetricKey)
        return try JSONDecoder().decode(T.self, from: data)
    }
    
    //MARK: - Private functions

    private func generateRandomSalt() -> Data {
        var data = Data(count: 32)
        _ = data.withUnsafeMutableBytes { buffer in
            SecRandomCopyBytes(kSecRandomDefault, 32, buffer.baseAddress!)
        }
        return data
    }
}

// MARK: - EnclaveKeychainAccess

public protocol EnclaveWrapperKeyAccess {
    var enclaveKey: EnclaveWrapperKey? { get }
    func updateKey(_ key: EnclaveWrapperKey) throws
}

public class EnclaveKeychainAccess: EnclaveWrapperKeyAccess {
    private let privateKeyTag = "com.demoApp.secureEnclaveDemo.secureEnclavePrivateKey"
    
    public init() {}
    
    public var enclaveKey: EnclaveWrapperKey? {
        return try? Keychain.shared.get(key: privateKeyTag, ofType: EnclaveWrapperKey.self)
    }
    
    public func updateKey(_ key: EnclaveWrapperKey) throws {
        try Keychain.shared.set(value: key, key: privateKeyTag)
    }
}

// MARK: - KeychainWrapper

public class Keychain {
    private let keychain: KeychainAccess.Keychain
    
    public init(service: String) {
        self.keychain = .init(service: service)
    }
    
    public func get<T: Codable>(key: String, ofType type: T.Type) throws -> T? {
        guard let data = try keychain.getData(key) else {
            return nil
        }
        
        return try JSONDecoder().decode(T.self, from: data)
    }
    
    public func set<T: Codable>(value: T?, key: String) throws {
        guard let value = value else {
            try keychain.remove(key)
            return
        }

        let data = try JSONEncoder().encode(value)
        
        try keychain.set(data, key: key)
    }
    
    public func remove(key: String) throws {
        try keychain.remove(key)
    }
}

public extension Keychain {
    static let shared: Keychain = Keychain(service: "com.demoApp.secureEnclaveDemo")
}

Related Articles:

• Apple: Secure Enclave
• Apple: Shared Secret
• Andy Ibanez: CryptoKit & Secure Enclave

Article Photo by Getty Images

I can recall many times when I deployed services with Terraform only to find out it didn’t work as intended. I would then sometimes spend hours pecking through the cloud UI looking for a missing configuration or what could possibly cause the issue. Sometimes it would even be the same missing configuration for a previous Terraform deployment. You know, like those elusive Security Group's egress/ingress rules. Only if I had an automated way to test for configurations that should be set or services that should be active. Let me introduce unit and integration tests using Terratest from Gruntwork.

Unit Tests

Unit tests are automated tests to ensure that a section of an application (known as the "unit") meets its design and functions as intended. With Terraform this can really only be done by deploying to a real environment. When deploying infrastructure, it is very rarely just a single unit, however more an integration test of a few to many services. For that reason, there is no true unit testing with IaC.

So how can this be done? How can we test our infrastructure in the smallest unit possible? Let’s take this simple infrastructure example.

Testing this entire infrastructure would take too long and would have a high risk of failure. This is why relying on end to end testing for infrastructure is not advised, and should be broken up into smaller unit and integration tests. In this example, we can do this by making sure each service is broken up into modules.

By testing each module we can deploy our infrastructure, validate it works, and then destroy it. Here is a snippet of code on performing this test on an AWS VPC with Terratest.

  defer terraform.Destroy(t, terraformOptions)
  terraform.InitAndApply(t, terraformOptions)
  outputs, err := TerraformOutputAll(t, terraformOptions)
  CheckIfError(err)
  validateVPC(t, outputs)

Let's walk through each line. Most of these lines have a terraformOptions these are options that are passed. These include the Terraform directory, and environment variables, or any Terraform variables to pass. They also may have a t, which is the testing package in Go.

1. In Golang, the defer keyword is used to delay the execution of a function or a statement until the nearby functions return. Meaning that the Terraform will destroy the deployed infrastructure once the deployment and validation is completed, successfully, or not.
2. The Terraform init and apply commands are run, which will deploy the infrastructure.
3. Outputs are injected into a variable to be used to validate our tests. Some examples are VPC ID, public subnets, and private subnets.
4. A wrapper to check for any errors.
5. Run Go tests to validate that the VPC is working as intended.

Here is an example of two tests. One that will verify that the public subnets are indeed public, and the other verifying that the privates are in fact private. Don't focus too much on the code, what I want to point out is that we can take the Terraform outputs, and assert that they are public, or private by using the assert.True or assert.False. Meaning if a private subnet that was created was public, it would fail the test.

  // Verify if the network that is supposed to be public is really public
  for _, v := range outputs.AWSVPCPublicSubnets {
    assert.True(
      t,
      aws.IsPublicSubnet(
        t,
        string(v),
        awsRegion,
      ),
    )
  }

  // Verify if the network that is supposed to be private is really private
  for _, v := range outputs.AWSVPCPrivateSubnets {
    assert.False(
      t,
      aws.IsPublicSubnet(
        t,
        string(v),
        awsRegion,
      ),
    )
  }

When I run the tests on the VPC and they all pass, I would get the results like so.

=== RUN   TestVPC/Subnets
--- PASS: TestVPC (7.11s)
    --- PASS: TestVPC/Subnets (0.93s)
PASS
ok      example 7.474s

The above example is testing the subnets within a VPC. Which, I would call an integration test. since our VPC module includes an option to create subnets, the unit in question is the module itself. Before we go into what integration tests would look like, let's go over the stages in Terratest.

Terratest Stages

Terratest has a helper function RunTestStage. This help function is used as a variable as stage. The quoted text is the name of the stage, which we can now skip by passing an environment variable. If we wanted to deploy this VPC, however, didn't want to destroy it after the run is completed, we can export the env var SKIP_destroy_vpc. This is important to include in the integration tests to avoid destroying infrastructure that was created in a separate module and needed for other services.

	stage := test_structure.RunTestStage

	defer stage(t, "destroy_vpc", func() {
		DestroyTerraform(t, terraformOptions)
	})

	stage(t, "apply_vpc", func() {
		outputs = ApplyTerraform(t, terraformOptions)
	})

	stage(t, "validate_vpc", func() {
		validateVPC(t, outputs)
	})

The output now shows that the destroy_vpc was skipped:

The 'SKIP_apply_vpc' environment variable is not set, so executing stage 'apply_vpc'.
applying...
The 'SKIP_validate_vpc' environment variable is not set, so executing stage 'validate_vpc'.
=== RUN   TestVPC/Subnets
The 'SKIP_destroy_vpc' environment variable is set, so skipping stage 'destroy_vpc'.
--- PASS: TestVPC (7.59s)
    --- PASS: TestVPC/Subnets (0.96s)
PASS
ok      example 7.925s

Integrations Tests

Here is a real-world example of how integration tests would work. We want to deploy an ECS cluster, and we want to make sure that the container on the service is running, and that the ingress/egress security group rules for the ALB and ECS are set correctly. We already deployed other modules needed and left them deployed by using the SKIP_destroy_foo environment variables. Now running the tests will skip any stage name that matches the environment variables, in this example SKIP_destroy_ecs.

	stage := test_structure.RunTestStage

	var outputs TerraformOutputs

	defer stage(t, "destroy_ecs", func() {
		DestroyTerraform(t, terraformOptions)
	})

	stage(t, "apply_ecs", func() {
		outputs = ApplyTerraform(t, terraformOptions)
	})

	stage(t, "validate_ecs", func() {
		validateECS(t, outputs)
	})

	stage(t, "integration_ecs", func() {
		integrationECS(t, outputs)
	})

Within the integration_ecs we will be testing:

• VPC and subnets
• Security group ingress/egress rules
• ECS service running

The 'SKIP_apply_ecs' environment variable is not set, so executing stage 'apply_ecs'.
apply...
The 'SKIP_validate_ecs' environment variable is set, so skipping stage 'validate_ecs'.
The 'SKIP_integration_ecs' environment variable is not set, so executing stage 'integration_ecs'.
=== RUN   TestECS/ECS_Cluster
=== RUN   TestECS/VPC
=== RUN   TestECS/Security_Groups
The 'SKIP_destroy_ecs' environment variable is set, so skipping stage 'destroy_ecs'.
--- PASS: TestECS (12.30s)
    --- PASS: TestECS/ECS_Cluster (0.28s)
    --- PASS: TestECS/VPC (0.99s)
    --- PASS: TestECS/Security_Groups (0.21s)
PASS
ok      example 12.718s

Conclusion

I can now with confidence test deployed infrastructure using Terraform by using Terratest. I can have Terratest deploy in a separate sandbox environment, validate it works with set checks, and finally destroy the created infrastructure once done. A few takeaways I have from testing IaC with automation are to avoid end-to-end testing, make sure to set a bigger time out on the Go test command with -timeout, and spend time thinking of checks that you want to include in your tests.

Note

HashiCorp has switched from MPL v2 license to a “Business Source License” (BSL). This will have an impact on Terratest going forward past Terraform version 1.5.5, so just make sure for the time being not to upgrade past that version if using Terratest. If interested, look into the OpenTofu Linux Project, which has a forked Terraform, and is dedicated to open source.

Article Photo by James von Hagel

Responsive web design is a term frequently used in web design and development. But what exactly does it entail? This blog post will introduce the multitude of facets of responsive web design, as it covers more ground than one would initially think.

Definition

"Responsive web design (RWD) is a web development approach that creates dynamic changes to the appearance of a website, depending on the screen size and orientation of the device being used to view it."
- Nielsen Norman Group

While this definition emphasizes screen size, a broader perspective considers the diverse contexts in which a website is accessed:

Responsive web design is an approach that factors in all the different environments in which a website is displayed

Benefits

In modern web development, focused on metrics, responsive web design is somewhat overlooked. Due to its complex and interpretive nature and the countless possible environments it can be challenging to implement, test, and measure.
It is clear however that, every user having their own unique environment, good responsiveness will have a positive impact. Each aspect of this guide will feature a brief introduction, methods, guidelines, and potential pitfalls for an effective implementation.

Display & Window Parameters

When it comes to responsive web design, the most crucial parameters are the display and window parameters. These include:

Width

Width is the primary consideration in responsive web design. It involves several key elements:

• Design: Different base designs for various width categories.
• Breakpoints: Define the points at which each design is applied based on screen width.
• Adaptation: Specify how the design should adjust between these breakpoints. For example, if you have a mobile design at 375px with a breakpoint at 768px, it's essential to define how this design adapts between 375px and 768px.

Grid Layout

One approach to handling these adaptations is by using a grid layout to structure your web page effectively.

Height

Height impacts what is visible "above the fold" - the content users see without scrolling. When using tools like Figma, it's not always clear what users will initially see on the website. However, this is essential for user retention. Additionally, from a graphic design perspective, the balance of the page can change when only a portion is visible.

⚠ Keep in mind that on smartphones, the height can change once users start scrolling or when a keyboard appears.

Vertical Rhythm

Much like the grid layout, another typography concept to consider is vertical rhythm. However, implementing this concept can be challenging due to the presence of numerous non-text elements intertwined with text. Additionally, text elements at the same height may be placed within different containers, causing them to flow vertically independently of one another.

Pixel Density

CSS

Screens have varying pixel densities. For example, some screens can render a 0.5px border, while others will round it up to 1px. While this difference may seem minor, it can add up and affect the alignment of elements on the screen.

Images

The srcset attribute is a useful tool for serving different images based on factors like screen size, pixel density, and accepted formats. This ensures that the image displayed is of the right size and quality for the device.

<picture>
  <source
    media="(max-width: 767px)"
    type="image/webp"
    srcset="image-mobile.webp, image-mobile@2x.webp 2x"
  />
  <source
    media="(min-width: 768px)"
    type="image/webp"
    srcset="image-desktop.webp, image-desktop@2x.webp 2x"
  />
  <source
    media="(max-width: 767px)"
    type="image/webp"
    srcset="image-mobile.webp, image-mobile@2x.png 2x"
  />
  <source
    media="(min-width: 768px)"
    type="image/webp"
    srcset="image-desktop.webp, image-desktop@2x.png 2x"
  />
  <img src="fallback.png" alt="image" />
</picture>

The above code has a different image on a viewport below 768px and serves webp and retina images where appropriate.

Contrast

Consider contrast when designing for various screens. Cheaper screens tend to have lower contrast ratios, making it difficult to distinguish between certain colors, especially when looking at a screen at an angle. Tools like Lighthouse in Chrome can help identify contrast issues.

⚠ It's possible that backgrounds, like alternating rows, may not contrast sufficiently with their surroundings. This doesn't impact text legibility, so might not be flagged by these tools.

Text Over a Background Image

When combining responsive web design and text over images, it may happen that at certain widths, the text appears over a part of the image with low contrast. A little trick to improve readability is to use a text shadow.

Subtle shadow:

color: #fff;
text-shadow: 1px 1px 0 rgba(0, 0, 0, 0.5);

More pronounced shadow:

text-shadow: 1px 1px 0 rgba(0, 0, 0, 0.5), -1px 1px 0 rgba(0, 0, 0, 0.5),
  -1px -1px 0 rgba(0, 0, 0, 0.5), 1px -1px 0 rgba(0, 0, 0, 0.5);

Orientation / Aspect Ratio

The user might prefer to view certain types of content, such as graphs, tables, and videos, in landscape mode. In this case, it is crucial to consider that fixed position elements, like the header, leave enough space for the rest of the content.

⚠ If the text size adapts to the view width, using landscape will create a zooming-in effect, which is unlikely to be the intended result of using landscape mode.

Sizing Units

In addition to using pixels for sizing, viewport units offer a dynamic way to define element sizes. Viewport units are percentages of the viewport's width or height and adapt to various window sizes. For cases where the viewport changes dynamically, such as when a user scrolls on a smartphone, small, large, or dynamic units can be used.

⚠ Be cautious when using 100vw, as a vertical scrollbar can reduce the available width, causing content overflow and the appearance of a horizontal scrollbar.
The following is a workaround:

width: calc(100vw - (100vw - 100%));

REM

The rem unit is a flexible way to define sizes based on the font size of the root element. For example, if the root font size is 16px, you can set a paragraph's font size as 1rem, which equals 16px.

html {
  font-size: 16px;
}

p {
  font-size: 1rem;
}

For a fixed design across screen sizes, a combination of viewport width (vw) and rem units can be used. For instance:

html {
  font-size: calc(100vw / 375);
}

p {
  font-size: 16rem;
}

In this case, the paragraph font size will be 16px at a viewport width of 375px and adjust proportionally with changes in the window width. This way it is easy to follow the design and use rem instead of px after all values.

⚠ However, when using rem units, the 100vw workaround unfortunately won't work.

font-size: calc((100vw - (100vw - 100%)) / 375);

Scrolling

When the content exceeds its container's dimensions, it's essential to define the scroll containers and specify their scrolling behavior, such as "normal," "fixed," or "sticky."

Mobile browsers have a scrollbar with the overlay property, a feature not widely available on non-mobile browsers. An alternative solution is the use of scrollbar-gutter; however, as of now, it is not universally supported.

On desktop, vertical scrolling is preferred, since it is easily accessible via the scroll wheel. This allows to have a narrower, more discrete vertical scrollbar. A horizontal scrollbar will need to be larger and more obtrusive, since the scrollbar thumb will need to be used.
On mobile on the other hand, is is advised to limit vertical scrolling to the whole page. When the content exceeds its container, horizontal scrolling can be easily done by swiping.

⚠ In both cases, avoid using infinite scrolling if a footer is present, since it will become unreachable.

⚠ Avoid using two nested vertical scrollbars as it can lead to a confusing user experience. Using the mouse wheel inside such an element will cause an inconsistent scrolling behavior depending on the position of both scroll bars.

Different Scroll Patterns Based on the Same Design

The following example shows how one design can have a wide array of possible scroll patterns.

Scroll Position

The Intersection Observer API has better performance than checking for the scroll position on scroll and resize. With responsive web design there's the additional benefit that the point at which a certain event is triggered can easily be controlled with an element that is variable in size.

Browser & OS

Fonts

When choosing fonts, keep in mind that every OS has its own system fonts and a font stack is recommended.

To ensure uniformity across different platforms, web fonts can also be used. The fonts can be hosted with the rest of the code. Alternatively, services like Google Fonts provide a convenient way to access a wide variety of fonts. These services often split font files into smaller parts, ensuring that only the necessary font subsets are downloaded. This approach is particularly recommended for extensive font sets, such as Japanese characters, to optimize loading performance.

However, it's worth noting that slight variations in rendering may still occur depending on the browser being used.

The following examples use Google Fonts with a fixed css, but still have slight differences across browsers.

font-size: 24px;
line-height: 1.5;

Graceful Degradation & Progressive Enhancement

Graceful degradation and progressive enhancement are design principles that utilize the latest browser features while maintaining functionality on legacy browsers, at the cost of performance and design variations. It can be applied to both javascript and css features.
For instance, consider the flex box gap property, which is unavailable on browsers earlier than iOS 14.5. This will result in layout differences, but won't disrupt the user experience, and takes less time to implement than the old method.

CanIUse is the reference for assessing browser compatibility.
Tools like Modernizr offer an extensive list of checks to implement in your website, enabling you to handle exceptions in your styling and code effectively.

This approach isn't just limited to browser functionalities; it can also be applied to screen sizes to ensure the layout remains functional on less common sizes, even if it's not pixel-perfect in all cases.

Touch Screens

Designing for different screen types involves considering how users interact with the content.

On conventional screens, hover states can convey information, they are, however, unavailable on touch screens. This distinction must be taken into account when designing user interfaces.

Touch screens on the other hand have other events like swipes and pinches, which aren't available on conventional screens. Also since swiping is used for scrolling, the presence or not of scrollbars doesn't have to be accounted for on mobile designs.

Additionally, touch screen devices offer unique input behaviors, such as keyboard types and select lists. When designing for mobile screens, it's often beneficial to prioritize using native select elements rather than custom-styled ones.

Media Queries

While they are commonly used to adjust styles based on screen width and print, properties such as orientation, resolution, pointer type, hover capability, and many more are available.

When to use media queries and when other solutions are preferred is discussed further in the Design System section of this article.

Component Drive Design

Component Driven Design is a methodology similar to atomic design, where applications are built starting from the smallest components. This approach emphasizes that a component should adapt to its container. Consequently, a web page primarily serves as a container for arranging these components, simplifying the process of achieving responsiveness.

Design Systems

Whether you're using a component library like Material or Ant Design, or creating your own there are 2 possible approaches to responsive web design.

1. Use the same style on all sizes and differentiate with properties. e.g. a large button on mobile and a medium one on desktop.
2. Different styles separated by media queries. e.g. a large button on all screen sizes, but with a different style.

The second method is preferred since it doesn't rely on javascript and isn't impacted by server side rendering.

As a general rule use media queries when components have a different base design, bug avoid using them for design differences based on the available space.

Consider the following case, an article component has an image and text. The image is to the left and the text to the right, but when the width is limited, the image takes the full width and the text comes below it. Since we don't know how much space will be available we can't rely on media queries (its container can have several columns, or there can be a side menu).
Below is an example how this can be solved, using css values like min

<article class="article">
  <div class="article_image_container">
    <img class="article_image" />
  </div>
  <div class="article_text">
    <h1 class="article_title">Lorem ipsum dolor sit amet</h1>
    <p class="article_description">
      Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
      tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim
      veniam, quis nostrud exercitation
    </p>
  </div>
</article>

.article {
  display: flex;
  align-content: flex-start;
  /* Wrap allows the text to be next or under the image */
  flex-wrap: wrap;
  padding: 20px;
  width: 100%;
  border: 1px solid #ccc;
  gap: 16px;
}

.article_image_container {
  /* 
    With flex the image will grow vertically with the text
    With block there is an extra space below the image
  */
  display: grid;
  /* Width of the image in horizontal layout */
  min-width: 200px;
  flex: 1;
}
.article_image {
  aspect-ratio: 4/3;
  width: 100%;
  background: #ccc;
  border-radius: 16px;
}

.article_text {
  display: flex;
  flex-direction: column;
  flex: 1;
  /*
    If the container is smaller than the image min-width and gap (216px) + 240px
    the text container will be under the image
    If the container is wider, the text will take its full width - (216px)
  */
  min-width: max(240px, calc(100% - 216px));
  gap: 16px;
}

.article_title {
  font-size: 24px;
  line-height: 1.3;
}

.article_description {
  display: flex;
  flex-direction: column;
  text-align: justify;
  font-size: 16px;
  line-height: 1.5;
}

Libraries

Libraries like Material (React, Vue, Angular) and Ant Design form a good basis for medium-sized projects. (They can be a bit heavy for smaller projects and larger projects might benefit more from a tailor-made in-house library).
They've been extensively tested across diverse environments, making them reliable choices for responsive web design.
They can be customized to fit most designs, but the more they're modified, the higher the risk of introducing bugs. It is therefore important to align with the design team at the start of the project, what library to use, and review together what can be customized at what cost.
They have the additional benefit that the components already use best practices. There also is less need for the UI and UX team to define all states and use cases, nor for the developers to implement them from scratch.

Storybook

Storybook, which is a tool for constructing and testing UI components and pages in isolation, and is ideal for Component Driven Design. Additionally, Storybook enables visual testing for different browsers and various viewport sizes.

Accessibility

Accessibility deserves its own in-depth article, but here are some points to consider when implementing responsive web design.

Screen Readers

In the process of implementing responsive layouts, it's sometimes necessary to adjust the order of elements. It's crucial to remember that these changes can also affect the experience of users relying on screen readers as well as SEO.

Font Size

When using font sizes adapting to screen width, make sure that the minimum possible font size is still readable.

p {
  font-size: max(16rem, 12px);
}

Testing Environment

Tools like browserstack can be used for cross-browser and cross-device testing. They're helpful for ensuring web applications function correctly and appear consistent on various browsers, devices, and OS. Additionally, they provide debugging and inspection tools to streamline issue resolution and quality assurance in web development.

Google Analytics

It can be costly and time-consuming to achieve a perfect result across all environments, it is therefore important to prioritize. Google Analytics provides valuable data on the devices, operating systems, and screen sizes that your audience uses. When making decisions about design parameters and testing, analyze this data to help achieve the most efficient coverage.

SEO

Google takes into account mobile-friendliness to rank websites. Since performance is also an important factor, it is crucial to strike the right balance between serving the right content quickly for the right environment, while limiting the burden of having content served for other environments or too many javascript checks when loading the page.

Text Ellipsis

Text ellipsis can be used when text overflows, but scrolling is not desired (small containers like chips or table cells). CSS (ellipsis and line clamp) offers several advantages over a fixed javascript string length, since it adapts to different character and container sizes.

⚠ Make sure that, when using an ellipsis, there is a way for users to visualize the full text. In some cases, the capability to select and copy the full text should also be maintained. This can be an issue with tooltips.

Collaborating With UI & UX Designers

The earlier in the project there is an alignment between the developers and designers, the better. Understanding each other's viewpoints and issues will help the implementation. A common view on the design philosophy and clear initial guidelines will also make it possible for developers to implement the designs without having to burden the designers with creating every possible state and use case for every page and component.

References

• Responsive Web Design (RWD) and User Experience
• Responsive layout grid
• Scrolling and Attention
• 8-Point Grid: Vertical Rhythm
• Mozilla Resources for Developers, by Developers
• An image format for the Web
• Can I use
• Make your website more readable
• The large, small, and dynamic viewport units
• System Font Stack
• Modernizr
• Component Driven User Interfaces
• Storybook

Header, grid and responsive design images by Hanaka Misono

Theory walkthrough

TCA

If you are reading this article I would assume that you already have some knowledge about what is TCA. Anyway here below you can find a quick brief with a diagram about this architecture. ![TCA Diagram](/assets/img/articles/2023-10-26-The-Composable-architecture-and-TDD/TCA-diagram.webp The Composable Architecture is a framework that can be used to build state-driven, testable applications. In TCA, each feature is represented by a Store that holds the feature's State and Reducer function. Feature State is updated by sending an Action to the Reducer function. The Reducer function takes the current state and action as parameters and mutates the current state to produce a new state. The View observes the state and updates itself when the state changes.
Additionally, the Store manages feature Environment dependencies (API client, database client, location manager, etc.) that can be invoked by the Reducer function, potentially producing Effects. Effects can, in turn, trigger another action to be sent to the Reducer. This simple flow ensures that feature data flows in a unidirectional manner and decouples the View from the business logic, which is crucial when developing a mobile application.
Keeping this in mind, we can recognise one of the key advantages of TCA - Testability. TCA strictly separates the view from business logic, enabling it to be tested without the need to create any views or provide any runtime dependencies.
If you want to dive into the TCA basics, here is a list of available resources that you can explore:

• TCA Documentation
• GitHub repo with CaseStudies
• Pointfree videos

TDD

I think that the best definition of what is TDD is given in Test-Driven iOS Development with Swift book by Dr. Dominik Hauser:

As the name suggests, in TDD, tests drive the development. This means that the developer writes code only because there is a test that fails. The tests dictate whether the code has to be written, and they also provide a measure when a feature is "done" - it is done when all tests for the feature pass.

The TDD flow is simple and consists of only three steps:

• Red - you start by writing a failing test. It needs to test a required feature of the software product that is not already implemented or an edge case that you want to make sure is covered.
• Green - in the green step, you write the simplest code that makes the test pass.
• Refactor - in the refactor step you improve the code. You remove duplication, extract common values, and so on. You should never skip this step and always think how you can improve your code. When you came up with an improvement, first make sure it's reflected in your test and then refactor an implementation. Not vice-versa!

Practice

Testing State changes

Let's say I want to have a TextField where user will input a name of their friend. There also will be a Discover button that should be enabled only when input name is not empty.

❌ Red:

First, let's start with implementing a failing test. Here, in the screenshot below, you can see how the TCA library conveniently displays failing tests. It highlights all state changes and fields that differ from the expected output. ![Failing test](/assets/img/articles/2023-10-26-The-Composable-architecture-and-TDD/failing-test-screenshot.webp

✅ Green:

Now let's make this test pass by implementing the logic:

public struct HomeFeature: Reducer {
    public struct State: Equatable {
        @BindingState var name: String = ""
        var discoverButtonDisabled: Bool = true
    }
    
    public enum Action: Equatable, BindableAction {
        case binding(BindingAction<State>)
    }
    
    public var body: some ReducerOf<Self> {
        BindingReducer()
        Reduce { state, action in
            switch action {
            case .binding(\.$name):
                state.discoverButtonDisabled = state.name.isEmpty
                return .none
            case .binding(_):
                return .none
            }
        }
    }
}

All good, now the test passes: ![Passing test](/assets/img/articles/2023-10-26-The-Composable-architecture-and-TDD/passing-test-screenshot.webp

📈 Refactor:

I want to refactor the discoverButtonDisabled boolean in this section. I anticipate that in the future, this Discover button will trigger an API call and be replaced by a ProgressView. To prepare for this, let's convert the boolean into an enum value. Again, first start by changing your test implementation and then move to the Feature.
Test:

    func testNameInput() async {
        let store = TestStore(initialState: HomeFeature.State()) { // Create test store for HomeFeature
            HomeFeature()
        }
        
        // Send action to imulate user input
        await store.send(.binding(.set(\.$name, "Steave"))) {
            // Assert name was changed to 'Steave'
            $0.name = "Steave"
            // Assert that Discover button was enabled
            $0.discoverButtonState = .enabled // Compiler error will be raised.
        }
    }

Implementation:

public struct HomeFeature: Reducer {
    enum ButtonState {
        case enabled
        case disabled
        case loading
    }
    
    public struct State: Equatable {
        @BindingState var name: String = ""
        var discoverButtonState: ButtonState = .disabled
    }

    public enum Action: Equatable, BindableAction {
        case binding(BindingAction<State>)
    }

    public var body: some ReducerOf<Self> {
        BindingReducer()
        Reduce { state, action in
            switch action {
            case .binding(\.$name):
                state.discoverButtonState = state.name.isEmpty ? .disabled : .enabled
                return .none
            case .binding(_):
                return .none
            }
        }
    }
}

Testing effects

In TCA, Effects play a big role, and it is crucial to be able to test them. Let's take a look how this is done. The logic is that when user presses the Discover button, an API request is fired in order to get the data to be shown. If the API request is successful, then a sheet showing the data should be opened. Otherwise, it should show an alert with an error message.

Successfull response

    func testDiscoverWishesTapped() async {
        let store = TestStore(initialState: HomeFeature.State()) {
            HomeFeature()
        } withDependencies: {
            $0.apiClient = .testValue // Use mock API client
        }
        store.exhaustivity = .off(showSkippedAssertions: true) // Test only assertions
        await store.send(.discoverWishesTapped) // Nothing happens
        
        // User inputs name
        await store.send(.binding(.set(\.$name, "Steave"))) 
        
        // expected wishes
        let expected = try! await APIClient.mock.requestWishes()
        
        // User taps discover
        await store.send(.discoverWishesTapped) {
            $0.discoverButtonState = .loading
        }
        // Receive wishes and open sheet
        await store.receive(.wishesResponse(.success(expected)), timeout: .seconds(0.1)) {
            $0.discoverButtonState = .enabled
            $0.destination = .submitWish(SubmitWishFeature.State(name: "Steave", wishes: expected.wishes))
        }
        
        // Check if we actually created scoping between two features
        // by sending an action to child store through the parant
        await store.send(.destination(.presented(.submitWish(.selectWish(0))))) {
            $0.$destination[case: /HomeFeature.Destination.State.submitWish]?
                .selectedWish = .init(id: "mock1", title: "Health", iconName: "heart.square")
        }
    }

Failing response

    func testFailingDiscoverWishes() async {
        let store = TestStore(initialState: HomeFeature.State(name: "Steave")) {
            HomeFeature()
        } withDependencies: {
            $0.apiClient = .failing // set API client to failing
        }
        
        // Request is triggered
        await store.send(.discoverWishesTapped) {
            $0.discoverButtonState = .loading
        }
        
        // Failed response, expect that apiError will be shown
        await store.receive(.wishesResponse(.failure(APIError.callUnimplemented)), timeout: .seconds(0.1)) { 
            $0.discoverButtonState = .enabled
            $0.destination = .alert(.apiError)
        }
        
        // Test that user can close the alert
        await store.send(.destination(.presented(.alert(.ok)))) {
            $0.destination = nil
        }
    }

Now let's focus on key TCA features that are used here:

• .receive() - API response produces an effect with the result, wich has to be received in the test. This method will do it for you and you will be able to make assertions agains the new State. On the other hand if there is some Effect that was not received, the test will fail.
• withDependency - Setting this parameter is a great way to control dependencies you provide to a test. In this example I use different API client implementations. One for mock another one for failing.
• exhaustivity - When this parameter is set to .on, the test framework checks whether all possible code paths and conditions in your application have been tested. If you skip an assertion for some state value, the test will fail. In cases where it's set to .off, only the assertions you make will be tested.
• destination - This is a @PresentationState value which determines a state of presented sheet/alert. Nothing is presented when it's set to nil. In addition, it tests whether parent and child domains can exchange actions between each other by sending the .selectWish action from the parent to the child. In TCA, this is basically called scoping.

Testing Navigation stack

As you might know, the TCA framework provides an out-of-the-box solution for stack navigation with the help of StackState. Essentially, it holds the states of all child domains and allows you to push and pop them, as well as send actions between them. Because it's state-driven, you can easily test it.
In the following example, the user is presented with a list of wishes. The user is able to select only one wish and after pressing the Proceed button, they are taken to results screen. Let's take a look how this behaviour can be tested.

    func testProceedButtonTapped() async {
        let store = TestStore(initialState: SubmitWishFeature.State(name: "Steave", wishes: wishes)) {
            SubmitWishFeature()
        }
        // Nothing happens if nothing is selected
        await store.send(.proceedButtonTapped)
        
        // User selects a wish
        await store.send(.selectWish(0)) {
            $0.buttonState = .enabled
            $0.selectedWish = .init(id: "mock1", title: "Health", iconName: "heart.square")
        }
        
        // Request is sent to save selection on backend side
        await store.send(.proceedButtonTapped) {
            $0.buttonState = .loading
        }
        
        // Receive successfull response and open Result screen.
        await store.receive(.saveWishResponse(.success(.init())), timeout: .seconds(0.1)) {
            $0.buttonState = .enabled
            $0.path[id: 0] = .result(.init(name: "Steave", wish: .init(id: "mock1", title: "Health", iconName: "heart.square")))
        }
    }

Once response is received a new screen is pushed to navigation stack. We can test this by getting the first element from the path and asserting against its state.

Moreover, with TCA you can test the navigation by sending push/pop as well as child domain actions directly. Check out the following example:

    func testResultNavigation() async {
        let store = TestStore(initialState: SubmitWishFeature.State(name: "Steave", wishes: wishes)) {
            SubmitWishFeature()
        }
      
        // Directly send push action to the store
        await store.send(.path(.push(id: 0, state: .result(ResultWishFeature.State(name: "Steave", wish: .init(id: "mock1", title: "Health", iconName: "heart.square")))))) {
            $0.path[id: 0] = .result(ResultWishFeature.State(name: "Steave", wish: .init(id: "mock1", title: "Health", iconName: "heart.square")))
        }
        
        // Send action into child domain and assert State changes.
        await store.send(.path(.element(id: 0, action: .result(.shareInMyProfileTapped)))) {
            $0.path[id: 0, case: /SubmitWishFeature.Path.State.result]?.isSharedInMyProfile = true
        }
        
        // Pop from child domain and assert navigation stack is empty
        await store.send(.path(.popFrom(id: 0))) {
            $0.path = StackState()
        }
    }

Conclusion

Here we've covered testing of the most esential parts of any iOS application:

• State changes by user input or user interaction
• State changes by Effects
• Dependency management inside tests
• Presentation of sheets and alerts
• Navigation stacks testing

All of these aspects are covered by the TCA framework out-of-the-box. Combined with a strong separation of business logic from the UI, it ensures the full testability of iOS apps. Moreover, TCA employs scoping, enabling the exchange of data and actions between different parts of the application. Essentially, it transforms the entire app into a global state, consisting of states from standalone features. This allows developers to test comprehensive scenarios involving different features rather than small, isolated parts of the app. This capability of TCA is crucial when employing Test-Driven Development for application development.

We have the data, we have three ChartTypes, which show temperature (line chart), precipitation (bar chart) and wind (double line chart) and we have customized the axes.

Data pre-processing

Looking at the wind chart, with the double line, it is a bit difficult to make sense of it, there's just too much data. So let's see what we can do about it. We have one measurement every ten minutes for 24 hours, so in total, we have 144 measurements. Plotting 144 data points for the average wind speed and for the maximum wind speed on the same chart makes it a bit crowded. But maybe we don't need all those data points. Let's have a look at how we can turn a data set with 144 points into one with only 24, as we want one measurement every hour for the wind.

One way to do it could be to only pick every 6th measurement, and just ignore the rest. But this new dataset could end up showing a different trend than the initial one and is not a good downsampling representation of the original data set. We need to do this in such a way that for every 6 data points in the original data set, we have only one data point, and this new one is an accurate representation or summarization of the original 6. This process is called downsampling.

So let's create a function that takes the original data set and returns the downsampled one:

func downsample(originalArray: [WeatherDataPoint]) -> [WeatherDataPoint] {
  var downsampledArray: [WeatherDataPoint] = []
  // .. We'll add some stuff here
  return downsampledArray
}

We'll use Swift's stride function because it's perfect for this use case. The stride function takes a start, an end and a stride, and returns a sequence from start to, but not including, end, stepping by stride. So for example, stride(from: 0, to: 10, by: 2) will be [0, 2, 4, 6, 8], and stride(from: 0, to: 10, by: 3) will be [0, 3, 6, 9]. We want to create slices of the original data source, of size 6.

for i in stride(from: 0, to: downsampledArray.count, by: 6) {
      let endIndex = min(i + 6, downsampledArray.count)
      let slice = Array(self[i..<endIndex])
      let aggregatedValue: WeatherDataPoint
      // .. Figure out how to calculate the aggregated value
      downsampledArray.append(aggregatedValue)
  }

But what is the aggregated value? That's interesting, right? Because it depends. Our data set measures wind average speed, and wind maximum speed. What's a good aggregated value for 6 average speed data points? Well, it's the average of those. What about the maximum speed? It's the maximum of those. Because the average wind speed in an hour is the average of the wind speed in the first 10 minutes, the one in the following 10 minutes, and so on (6 times). And similarly for the maximum speed.

So let's pass the aggregation function as a parameter, and redefine our method like this:

func downsample(originalArray: [WeatherDataPoint], aggregationFunction: ([WeatherDataPoint]) -> WeatherDataPoint) -> [WeatherDataPoint] {
    var downsampledArray: [WeatherDataPoint] = []

    for i in stride(from: 0, to: originalArray.count, by: 6) {
        let endIndex = min(i + 6, originalArray.count)
        let slice = Array(originalArray[i..<endIndex])
        let aggregatedValue = aggregationFunction(slice)
        downsampledArray.append(aggregatedValue)
    }
    return downsampledArray
}

To make it even nicer, we can make this method an extension on Array and pass the step (the length of the slice) as a parameter too:

extension Array where Element == WeatherDataPoint {
    func downsample(step: Int, aggregationFunction: ([WeatherDataPoint]) -> WeatherDataPoint) -> [WeatherDataPoint] {
        var downsampledArray: [WeatherDataPoint] = []

        for i in stride(from: 0, to: self.count, by: step) {
            let endIndex = Swift.min(i + step, self.count)
            let slice = Array(self[i..<endIndex])
            let aggregatedValue = aggregationFunction(slice)
            downsampledArray.append(aggregatedValue)
        }

        return downsampledArray
    }
}

We can now use this method to downsample the wind data series, so we add to the StationMeasurements the following:

var downsampledWindSeries: [WeatherDataPoint] {
  return windSeries.downsample(step: 6) { slice in
    let count = slice.count
    let values = slice.map { $0.measuredValue }
    let secondaryValues = slice.compactMap { $0.secondaryMeasuredValue }
    return WeatherDataPoint(dateTime: slice.safelyAccessElement(at: count/2)?.dateTime ?? slice[0].dateTime,
                            measuredValue: values.reduce(0, +) / Double(count),
                            secondaryMeasuredValue: secondaryValues.max())
        }
    }

Where safelyAccessElement is a helper method which gives us the element at one index, or nil if the index is out of bounds.

private extension Array {

    func safelyAccessElement(at index: Int) -> Element? {
        guard indices.contains(index) else {
            return nil
        }

        return self[index]
    }

}

Now we have managed to turn our data set from 144 points to 24 points. We can use this one to draw the wind chart instead, and the chart immediately looks better.

Interactive charts

Now that the charts look better, let's make them even more useful: let's show the value for a precise measurement when the user taps on a point in the chart.

We want to detect when the user taps on the chart, show an indicator of the point they tapped, and show the value at that point in a label above the chart. So we add a .chartOverlay which detects the taps and drags to our chart like this:

.chartOverlay { proxy in
    GeometryReader { geo in
        ZStack(alignment: .topLeading) {
            Rectangle().fill(.clear).contentShape(Rectangle())
                .gesture(
                    DragGesture(minimumDistance: 0.0)
                        .onChanged { value in
                            switch chartType {
                            case .temperature(let data), .precipitation(let data), .wind(let data):
                                selectedElement = findElement(in: data, location: value.location, proxy: proxy, geometry: geo)
                            }
                        }
                        .onEnded { _ in
                            selectedElement = nil
                        }

                )
            if let selectedElement {
                LollipopView(chartType: chartType, selectedElement: selectedElement, chartProxy: proxy, geometryProxy: geo)
            }
        }
    }
}

We want to keep the currently selected element in the selectedElement @State property, which can be nil if no element is currently selected (if the user isn't currently tapping or dragging their finger on the chart). If the user taps on the chart and an element is selected, we show an indicator of that (vertical line from the X axis to the chart point where the user tapped, and a rounded circle at its top; we call this LollipopView)

For finding the element on which the user tapped, we have to translate the touch point from screen coordinates to chart coordinates and find the closest data point to that. Here is the findElement method:

private func findElement(in data: [WeatherDataPoint], location: CGPoint, proxy: ChartProxy, geometry: GeometryProxy) -> WeatherDataPoint? {
    let relativeXPosition = location.x - geometry[proxy.plotAreaFrame].origin.x
    if let date = proxy.value(atX: relativeXPosition) as Date? {

        // Find the closest element.
        var minDistance: TimeInterval = .infinity
        var index: Int?
        for dataIndex in data.indices {
            let nthDataDistance = data[dataIndex].dateTime.distance(to: date)
            if abs(nthDataDistance) < minDistance {
                minDistance = abs(nthDataDistance)
                index = dataIndex
            }
        }
        if let index {
            return data[index]
        }
    }
    return nil
}

When we have a selected element, we want to show a visual indicator of the tapped point, so here's our LollipopView:

struct LollipopView: View {
    let chartType: ChartType
    let selectedElement: WeatherDataPoint
    let chartProxy: ChartProxy
    let geometryProxy: GeometryProxy

    var body: some View {

        let dateInterval = Calendar.current.dateInterval(of: .minute, for: selectedElement.dateTime)!
        let startPositionX1 = chartProxy.position(forX: dateInterval.start) ?? 0

        let lineX = startPositionX1 + geometryProxy[chartProxy.plotAreaFrame].origin.x
        let chartAreaHeight = geometryProxy[chartProxy.plotAreaFrame].maxY
        let plotLineY = chartAreaHeight - (chartProxy.position(forY: selectedElement.secondaryMeasuredValue ?? selectedElement.measuredValue) ?? 0)

        // The vertical line
        Rectangle()
            .fill(.gray)
            .frame(width: 2, height: plotLineY)
            .position(x: lineX, y: chartAreaHeight - plotLineY / 2)

        // The symbol shown at the point we've tapped on the graph
        Circle()
            .foregroundColor(.blue)
            .frame(width: 16, height: 16)
            .overlay(
                Circle()
                    .stroke(Color.white, lineWidth: 4)
            )
            .position(x: lineX, y: chartProxy.position(forY: selectedElement.measuredValue) ?? 0)

        // If we're on the wind data chart, show the diamond on the gust line too
        if case .wind = chartType {
            Rectangle()
                .rotation(Angle(degrees: 45))
                .aspectRatio(1.0, contentMode: .fit)
                .foregroundColor(.red)
                .frame(width: 13, height: 13)
                .overlay(
                    Rectangle()
                        .rotation(Angle(degrees: 45))
                        .stroke(.white, lineWidth: 4)
                        .aspectRatio(1.0, contentMode: .fit)
                )
                .position(x: lineX, y: chartProxy.position(forY: selectedElement.secondaryMeasuredValue ?? 0) ?? 0)
        }
    }
}

On top of what we mentioned, for the wind chart which has 2 line charts, we want to show different symbols on each line: a circle on the average speed and a diamond on the gust. That's why we added the last block of code in the LollipopView.

Printing the latest measurement or the currently selected measurements above the chart should be pretty straightforward, as long as you make sure you format the data correctly.

And here's the LollipopView in action. We just made our charts interactive!

Resources:

1. Swift-Charts-Examples

Article photo generated with Midjourney

Apple introduced Swift Charts in iOS 16, and improved it in iOS 17, adding, among others, new chart types. If your app still needs to support iOS 15 or lower versions, you can use a 3rd party library, like Daniel Gindi's Charts. In this demo project, we'll focus on the initial release of Swift Charts and well target iOS 16.

Data

First, let's find some data to plot. We'll be looking at weather data, and, for the sake of simplicity, we'll just load locally some JSON from this file. The JSON looks like this:

{
    "name": "Københavns Lufthavn",
    "latitude": 55.6139,
    "longitude": 12.6455,
    "temperatureSeries": [{
        "dateTime": "2023-09-21T13:10:00Z",
        "measuredValue": 21.9
    }, ...
    ],
    "precipitationSeries": [{
        "dateTime": "2023-09-21T13:00:00Z",
        "measuredValue": 1.4
    }, ...
    ],
    "windSeries": [{
        "dateTime": "2023-09-21T13:10:00Z",
        "measuredValue": 6.2,
        "secondaryMeasuredValue": 8.2
    },
    ]
}

We have the details of a station, with its name, and position, and then we have 3 series of data, one for temperature (measured in Celsius degrees), one for precipitation (measured in millimetres) and one for wind, which contains the value of the average wind and the value of the gust (max wind speed) for that particular time. All entries in the data series have a dateTime. For temperature and wind, we have 144 data points, corresponding to one measurement every 10 minutes, and for precipitation we have 24 data points, corresponding to a measurement every hour.

Let's start a new Xcode project, using SwiftUI, and let's add the JSON file to the project and the target.

We want to parse this data, so we're creating our model. The purpose of this article is to play with Swift Charts; there are other, possibly better-suited ways of modelling the data, but for now, let's create a WeatherDataPoint struct.

struct WeatherDataPoint: Codable, Identifiable {
    let dateTime: Date
    let measuredValue: Double
    let secondaryMeasuredValue: Double?

    var id: Date {
        dateTime
    }
}

And a StationMeasurements struct:

struct StationMeasurements: Codable {
    let name: String
    let latitude: Double
    let longitude: Double
    let temperatureSeries: [WeatherDataPoint]
    let precipitationSeries: [WeatherDataPoint]
    let windSeries: [WeatherDataPoint]
}

And let's parse the data from the JSON file into a mock:

extension StationMeasurements {
    static var mock: StationMeasurements {
        let url = Bundle.main.url(forResource: "StationData", withExtension: "json")!
        let data = try! Data(contentsOf: url)
        let decoder = JSONDecoder()
        decoder.dateDecodingStrategy = .iso8601
        let jsonData = try! decoder.decode(StationMeasurements.self, from: data)
        return jsonData
    }
}

Basic Charts

We now have the data in our project, we can go ahead and start playing with it. We want to have three types of charts, a line chart to show the temperature, a bar chart to show precipitations, and another one with two lines on the same chart, so we can see the average and maximum wind on the same graph.

Let's describe that in code, by making an enum with a case for each chart type, and the data as an associated value.

enum ChartType {
    case temperature(temperatureData: [WeatherDataPoint])
    case precipitation(precipitationData: [WeatherDataPoint])
    case wind(windData: [WeatherDataPoint])
}

Let's start by creating the temperature chart. We'll make a new SwiftUI View, called ChartView. It will have a ChartType, which we'll also use to pass the data in through the associated value of the enum cases. The temperature chart will be a line chart, so inside the Chart we'll create a LineMark. We can then configure the line style, color and interpolation method, to make it prettier.

struct ChartView: View {
    let chartType: ChartType

    var body: some View {
        switch chartType {
        case .temperature(let temperatureData):
            Chart {
                ForEach(temperatureData) { element in
                    LineMark(
                        x: .value("dateTime", element.dateTime),
                        y: .value("temperature", element.measuredValue)
                    )
                    .lineStyle(StrokeStyle(lineWidth: 4))
                    .foregroundStyle(Color.blue)
                    .interpolationMethod(.cardinal)
                }
            }
        case .precipitation(let precipitationData):
            EmptyView()
        case .wind(let windData):
            EmptyView()
        }

    }
}

For the other two chart types, we can return EmptyView() for now. Here is the line chart showing the temperature data:

Let's continue and, similarly, create the bar chart. This time, we have to use the BarMark, but mostly everything else remains the same; we can remove the interpolationMethod, which doesn't make sense for a bar chart, and just change the identifiers.

case .precipitation(let precipitationData):
    Chart {
        ForEach(precipitationData) { element in
            BarMark(
                x: .value("dateTime", element.dateTime),
                y: .value("precipitation", element.measuredValue)
             )
              .lineStyle(StrokeStyle(lineWidth: 4))
              .foregroundStyle(Color.blue)
         }
    }

The wind chart will be a bit more interesting, it will consist of two line charts. So on the same Chart, we will have two LineMarks. The main difference from the temperature chart is that, for each LineMark, we will have to specify the series argument as well, so they will be plotted as individual series. We want the max wind speed to be plotted as a dashed line, so we also specify the dash parameter in the lineStyle.

case .wind(let windData):
    Chart {
        ForEach(windData) { element in
            LineMark(
                x: .value("dateTime", element.dateTime),
                y: .value("average wind", element.measuredValue),
                series: .value("average", "average wind")
            )
            .lineStyle(StrokeStyle(lineWidth: 4))
            .foregroundStyle(Color.blue)
            .interpolationMethod(.cardinal)

            if let secondaryMeasuredValue = element.secondaryMeasuredValue {
                LineMark(
                    x: .value("dateTime", element.dateTime),
                    y: .value("max wind", secondaryMeasuredValue),
                    series: .value("gust", "gust")
                )
                .lineStyle(StrokeStyle(lineWidth: 4, dash: [6]))
                .foregroundStyle(Color.red)
                .interpolationMethod(.cardinal)
            }
        }
    }

Now, we have the data plotted, but we would like to configure the axes. We want the vertical axis to be to the left of the chart, and we want to be able to change the font of the label on the vertical axes. Adding this code after the Chart {...} does the trick.

.chartYAxis {
    AxisMarks(position: .leading) {
        AxisValueLabel()
        AxisGridLine(stroke: StrokeStyle(lineWidth: 0.5))
    }
}

But that only customizes one of the three charts, and we don't like to repeat code.

So what we want now is to extract the Chart content, and use only one Chart, to be able to customize the axes the same way for all types of charts without repeating code.

Let's create the chart contents for our three charts.

struct LineChartViewContent: ChartContent {
    let data: [WeatherDataPoint]

    var body: some ChartContent {
        ForEach(data) { element in
            LineMark(
                x: .value("dateTime", element.dateTime),
                y: .value("temperature", element.measuredValue)
            )
            .lineStyle(StrokeStyle(lineWidth: 4))
            .foregroundStyle(Color.blue)
            .interpolationMethod(.cardinal)
        }
    }
}

struct BarChartViewContent: ChartContent {
    let data: [WeatherDataPoint]

    var body: some ChartContent {
        ForEach(data) { element in
            BarMark(
                x: .value("dateTime", element.dateTime),
                y: .value("precipitation", element.measuredValue)
            )
            .lineStyle(StrokeStyle(lineWidth: 4))
            .foregroundStyle(Color.blue)
        }
    }
}

struct DoubleLineChartViewContent: ChartContent {
    let data: [WeatherDataPoint]

    var body: some ChartContent {
        ForEach(data) { element in
            LineMark(
                x: .value("dateTime", element.dateTime),
                y: .value("average wind", element.measuredValue),
                series: .value("average", "average wind")
            )
            .lineStyle(StrokeStyle(lineWidth: 4))
            .foregroundStyle(Color.blue)
            .interpolationMethod(.cardinal)

            if let secondaryMeasuredValue = element.secondaryMeasuredValue {
                LineMark(
                    x: .value("dateTime", element.dateTime),
                    y: .value("max wind", secondaryMeasuredValue),
                    series: .value("gust", "gust")
                )
                .lineStyle(StrokeStyle(lineWidth: 4, dash: [6]))
                .foregroundStyle(Color.red)
                .interpolationMethod(.cardinal)
            }
        }
    }
}

Customizing the axes

And now we can modify the ChartView to use the newly created ChartContents, and then we can customize the axes once for all charts.

struct ChartView: View {
    let chartType: ChartType

    var body: some View {
        Chart {
            switch chartType {
            case .temperature(let temperatureData):
                LineChartViewContent(data: temperatureData)
            case .precipitation(let precipitationData):
                BarChartViewContent(data: precipitationData)
            case .wind(let windData):
                DoubleLineChartViewContent(data: windData)
            }
        }
        .chartYAxis {
            AxisMarks(position: .leading) {
                AxisValueLabel()
                AxisGridLine(stroke: StrokeStyle(lineWidth: 0.5))
            }
        }
    }
}

We also want to customize the horizontal axes. On this one, we display dates. Since the measurements are for 24 hours, and the space on the axis is quite limited, we're only interested in the time. We print one label every 4 hours and only print the time. We also show a grid line every 4 hours.

.chartXAxis {
    AxisMarks(values: .stride(by: .hour, count: 4)) { value in
        if let date = value.as(Date.self) {
            let hour = Calendar.current.component(.hour, from: date)
            AxisValueLabel {
                Text(date, style: .time)
            }
            AxisGridLine()
        }
    }
}

So here's how far we've come:

Where can we go from here? It would be nice if we could tap on the chart to see the exact measurement value, right? And maybe we can do something about that wind chart, which looks a bit crowded. We're looking at that in part 2.

Article photo generated with Midjourney

We (the MLJP mobile team members) wanted to catch up with AI and be ready to make better proposals for our clients. Therefore, we created a team to research this technology applied to mobile development and come up with an easy idea to get a nurturing experience.

What is Artificial Intelligence?

AI is a field of computer science that aims to create intelligent machines capable of performing tasks that typically require human intelligence. AI algorithms are generally based on models trained using specific data sets.

The goal of AI modelling is to generalize instructed knowledge during the inference phase, using data sets provided during the training phase. Training data sets generally consist of pairs of data points and their corresponding ground truth values, enabling the model to learn the relationship between the two and attempt to reproduce it.

Various types of data, such as images, text, or tabular data, can be used in this regard. The input/output data do not necessarily have to be the same. For instance, you can ask a model to classify an image into a specific category (e.g., whether an image contains a cat).

With the project developed in this article, we aim to detect face landmarks in an image. Thus, we feed the model an image and receive tabular data representing the coordinates of these landmarks as an output. The technical details are as follows:

• Data to be analyzed: An RGB image.
• Model Input data: A 192 x 192 pixel image, framed and resized around the face.
• Model Output data: 468 face landmarks, each having X, Y, and Z coordinates.
• Model type: A deep learning model that uses artificial neural networks based on the TensorFlow Lite framework.

Choose your lipstick

The app developed is a face makeup, starting with the lipstick color. This application aims to provide users with the ability to change the color of their lips in real-time using advanced pose estimation technology. Here is a basic overview of how this functionality works:

• Real-time Lip Detection: Using a pre-trained pose estimation model from the AI team, we track the position and shape of the user's lips in real-time. The model analyzes the facial landmarks and provides accurate coordinates of the lip contour.

• Color Palette Selection: To offer a diverse range of lip colors and shades, we provide users with RGB sliders. Users can interact with the sliders and select any desired lip color.

• Real-time Color Application: Once the user selects a lip color from the palette, the application applies the chosen color to the detected lip region in real-time. This is achieved by mapping the color to the specific lip contour coordinates obtained from the AI model.

We picked up this idea, after a long brainstorming, basically because it required low resources (developer’s face and a smartphone frontal camera) and an easy scenario to work with (one existing AI model). In addition, face makeup applications have gained significant popularity in recent years due to the rise of social media platforms and the desire for users to enhance their appearance in a fun and creative way.

In the following sections, we are going to explain the detailed process we followed to achieve the idea proposed and what challenges and issues we found along the way.

Image Processing Flow Overview

First of all, to take a video image from a mobile device camera and get some face landmarks, we needed to follow a series of steps. Here's a high-level overview of the main processes involved:

1. Capture the Video: We use the device’s built-in camera functionality to capture the video. We make sure the video includes the face we want to detect and extract landmarks from whilst being careful to ensure that the face is neither too close nor too far away. We should note that most face detection models also have difficulty matching a face that is looking too far (left, right, up or down), or tilting too far (either side).

2. Extract Frames from the Video: We convert the video into individual frames, as a video is essentially a sequence of images. We use the native frameworks in both Android and iOS to extract frames from the video, iterate through each frame from the video and perform the subsequent steps on each frame as a static image.

3. Convert Images to TensorFlow Lite Input: Before using these images in TensorFlow Lite, we need to preprocess them to match the input requirements of our face landmark detection model. This typically involves resizing the images to the model's expected input size and normalizing pixel values. Each model will have its own specific requirements for the images we pass to it, and failing to correctly preprocess our images will either give wildly inaccurate results or be unable to match a face. A more detailed explanation of this step is explained in the following sections (Cropping the Images and Image Processing Techniques).

4. Pass Images to TensorFlow Lite Model: We load the preprocessed images into our TensorFlow Lite face landmark detection model and run inference on each image. The model will output the predicted landmarks for the face detected in the frame. TensorFlow Lite provides APIs for loading the model and running inferences.

5. Overlay Images on the Video Frames: Once we have the face landmarks for each frame, we overlay an image or add visual effects using the landmarks as reference points. We use Android and iOS advanced graphic frameworks to do so.

Now that we have a good overview of the steps involved in detecting face landmarks in an image, let’s look at some of the image processing techniques we use to convert our images.

Cropping the Images

To improve the accuracy of the face landmark model, we avoid using the entire frame image when we only need the face. Thus, one of the first image-processing tasks we need is to crop the face, so that we are only processing the area we need to send to the model.

It's important to note that face landmark models are typically trained on datasets where the faces are already cropped or centered. Therefore, using cropped face images during inference helps align the input with the data distribution the model was trained on, resulting in better performance.

However, when cropping a face from an image to use in a face landmark model, there are several challenges and issues that need to be addressed. Here are some of the key considerations:

• Face Detection Accuracy: The accuracy of the face landmark model heavily relies on the quality and correctness of the cropped face. If the cropping process is not performed accurately, it can result in incomplete or inaccurate face regions, leading to degraded detection performance. Ensuring precise and comprehensive cropping is crucial to obtain reliable results.

• Face Localization: The face landmark model expects the face to be properly centered and aligned. However, variations in pose, scale, rotation, and occlusions in the original image can make face localization challenging. Addressing these factors and implementing robust techniques for face localization is essential to obtain accurate and properly cropped faces.

• Correct Aspect Ratio: Maintaining the correct aspect ratio of the face during cropping is important to avoid distortion or stretching. Faces in images can have different orientations and aspect ratios. It's necessary to handle these variations appropriately, ensuring that the cropped face maintains its correct proportions for effective face detection.

• Background and Context: Removing the background and irrelevant context from the cropped face will improve the performance of the face landmark model. Eliminating distractions, such as cluttered scenes or irrelevant patterns, that may interfere with the face landmark process, will help improve the model's accuracy.

• Head Cropping: Although we talk about face landmarks most models will require the whole head. Just cropping the face area will result in inaccurate results so the crop area should aim to contain the whole head, including the chin, ears and top of the head.

• Image and Lighting Quality: The quality of the camera photo image, including factors like resolution, noise, blur, and lighting conditions, can impact the performance of the face detection model. Noise or blur can obscure facial features, while poor lighting conditions may affect the visibility of the face. Properly handling image enhancement techniques, such as de-noising, sharpening, or adjusting brightness/contrast, can help improve the quality of the cropped face for more accurate detection.

• Occlusions and Partial Faces: Addressing occlusions, such as hair, glasses, or hands covering parts of the face, is a significant challenge when either trying to crop a face from an image or detect face landmarks. Detecting and accounting for these occlusions is crucial to ensure the cropped face includes the complete facial features required for accurate detection. Advanced techniques like occlusion detection and additional preprocessing steps can assist in handling these situations but it is usually more efficient to simply ensure that the user is aware of the face detection limitations and should avoid covering their face when possible.

With the above in mind, we tried two ways of cropping.

1. Center Cropping: This is the easiest to implement. We simply crop the largest possible square from the center of the image with its height equal to its width (when the images are in portrait orientation). To prevent the user from accidentally moving their face outside of this area we can overlay guidelines on the video image to indicate to the user the ideal position for their face. This method relies on the user to accurately position their face in the optimal location at the correct distance. If the user has difficulty with this, then we can change to Center Padding. This method adds a buffer area to the left and right of the image to turn the rectangle into a square. In this way, the user’s face will never be accidentally cropped but, by increasing the size of the image we will be reducing its accuracy.

2. Face Detection Cropping: This uses a face detection model to determine the precise position of the whole face (without individual face landmarks) and will return a bounding rectangle with an origin: (x, y) and a size: (width, height). This will usually result in a rectangle that will probably need to be padded out on the right and left sides to make a square. We should avoid cropping the top and bottom of the rectangle to get a square, as this will remove the chin, etc, that may confuse the face landmark model. The advantage of this method is that the cropped area will only contain the face area giving the face landmark model the most accurate image possible.

In our project, we decided to use the face detection model. After face cropping, the image is resized smaller for the model. As the images we input to the model are only a few hundred pixels wide, having the face fill as much of the image as possible will give a wider range for the landmark positions to be assigned to. For example, the lips alone can have over 50 landmarks. Another advantage of using a face detection model is that the user is free to move around, and the model will still accurately locate their face. Even when the user is far away from the camera the app will still find and expand the image appropriately. On the other hand, adding another layer of intensive image processing will increase CPU usage resulting in faster battery depletion and an increase in temperature for mobile devices.

Examples of Center Padding vs Center Cropping vs Face Detection Cropping

In deciding which cropping method is most suitable for your use case, please look at the below examples. They show exactly how the images will appear depending on how far the person is away from the camera. Note that center cropping can be effective only if the user actively positions themselves within the center cropping area, otherwise the cropped image may be unusable.

The top line shows Center Padding, the middle line shows Center Cropping and the bottom line shows Face Detection Cropping. ✅ is for images suitable for face landmark detection. ⚠️ is for images that may not be usable for face landmark detection (very small or partially cropped faces etc). ❌ is for images that are not usable for face landmark detection (faces that are badly cropped or obscured etc).

Image Processing Techniques

Even with the face cropped from the image, we will still need to perform some image processing to ensure the image is appropriately prepared for the model to be recognized. Here is a general guide on how to process the cropped image for a TensorFlow Lite face detection model:

1. Resize the Image: TensorFlow face detection models usually have specific input size requirements, such as a fixed width and height. Resizing the image to these dimensions ensures compatibility with the model. We can use resizing functions provided by the chosen library or process the images with the graphics frameworks provided by our mobile OS. We need to trim the image to the correct shape (usually from a rectangle to a square) and then scale the image down to the correct size.

2. Convert Image to the Required Color Format: Verify if the model expects the image in a specific color format, such as RGB or BGR. We should convert the image to the appropriate color format if necessary. Model Libraries will usually provide such conversion functions to convert the color format of an image for us. If not, then we would need to find an alternative library or write the necessary color format conversion code ourselves.

3. Normalize Pixel Values: We normalize the pixel values of the image to a specific range suitable for the model. Commonly used normalization techniques include scaling the pixel values to [0, 1]. Normalizing the image is done using mathematical operations or built-in library functions.

4. Convert Image to the Required Data Format: TensorFlow models typically expect input data in the form of tensors. We convert the preprocessed image to the appropriate data format using functions provided by the library we're using.

We should note that some (if not most) models usually don’t tell you that they have been unable to locate a face, and instead return their default face landmarks. This is when the model gives us a collection of face landmarks, but they usually do not match the user’s face and will not change over time. This can be a cause of confusion and is another good reason to preprocess the video frame image with a face detection model before anything else. This way, it is possible to parse the results and filter out images that have no faces. Also, we can discard faces that are positioned or orientated in ways that we know the face landmark model will be unable to use.

Face Landmark Position Conversion

After successfully obtaining the predicted face landmarks from our model, we now have to take the landmark coordinate values that we need, in this case, the positions of the lips, and use these to draw an image on the detected face. To draw the lips on top of the original face, we did the following steps:

1. Obtain Face Landmarks: After running the face landmark detection model with TensorFlow Lite, we obtain the predicted face landmarks. These landmarks represent specific facial features, such as the corners of the eyes, nose, and lips.

2. Identify Lip Landmark Points: Determine the specific landmark points that correspond to the lips. The exact landmarks can vary depending on the face landmark model we use, but typically, they include the points representing the outer edges of the lips, such as the corners or key points along the lip contours. Note that both the upper lip and lower lip are separate, so we have edge and corner points for both the top and bottom of the upper and lower lips.

3. Extract Lip Landmark Coordinates: Retrieve the coordinates of the lip landmark points from the face landmarks output. These coordinates typically consist of an array of (x, y) pairs representing the position of each lip landmark point on the image. Some models can predict 3D landmarks as a (x, y, z) coordinate but in this case, we are only overlaying a simple image onto a video so we will ignore the depth value (z).

4. Convert the Landmark Coordinates to Screen Coordinates: After obtaining the results of a TensorFlow Lite face landmark detection model, the coordinates for each landmark still need to be reverted to the correct size and scale of the original image. The reason for this is that if we pass an image to the model that is, for example, 192 x 192 then all of our landmark coordinates are also going to be from 1 to 192. The model has no knowledge of the size of the video frame image or screen size, so we must revert these values to the correct size with the following steps:

• Retrieve the width and height of the original image. These dimensions represent the size of the image before any resizing or preprocessing steps are applied.
• Compute the scaling factors for both the width and height by dividing the dimensions of the output from the TensorFlow Lite model by the dimensions of the original image. This step helps determine how much the coordinates need to be scaled to match the original image's size.
• Multiply each coordinate (x, y) obtained from the TensorFlow Lite model by the corresponding scaling factor calculated in the previous step. This scaling operation reverts the coordinates to the correct size and scale of the original image.

5. Overlay Lips on the Original Video Image: Once we have the lip landmark coordinates, we draw and overlay the lips on the image with the following steps:

• Create a new empty image view to use for drawing the lips.
• Iterate over the lip landmark coordinates and connect them to form the lip contours for the top and bottom lips. We use standard line draw functions to draw the outlines of the lips on our new image.
• Fill the lips using the contour points. We use standard shape-fill functions to fill the area enclosed by the lip contours.
• Adjust the color and opacity of the drawn lips to achieve the desired visual effect.
• Once we have drawn the lips on the image view, we overlay it over the video as necessary.

Although specific implementation details will vary based on the library you are using and the format of the face landmarks provided by your TensorFlow Lite model by following these general steps, you can utilize the face landmarks obtained from TensorFlow Lite to draw lips on the original image.

Technological background: Android

This is the development explanation of our Android app.

Implementation

• LiveCameraView: This composable function is responsible for displaying the live camera preview and UI components for adjusting the lip's color using RGB sliders. It uses Jetpack Compose, a modern UI toolkit, to build the UI. It receives data from the LiveCameraViewModel and updates the UI accordingly.

• LiveCameraViewModel: This class is responsible for managing the camera preview and applying AR makeup effects. It follows the ViewModel pattern and interacts with the Model to retrieve data and update the UI. It uses the MLKitFaceDetector for face detection and the FaceLandmarkAIInterpreter for face landmark detection.

  • processCameraImage(fullImageBitmap: Bitmap): Processes the camera image by detecting faces, extracting the face image, performing face landmark detection, and applying AR makeup effects. It takes the full camera image as a parameter.
  • arProcessedOutputBitmap: StateFlow<Bitmap?>: Represents the AR processed output bitmap, which can be observed by the UI.

• MLKitFaceDetector: This package contains functions related to face detection using the ML Kit Face Detection API.

  • detectFace(fullImageBitmap: Bitmap, faces: (result: List) -> Unit): Detects faces in the given fullImageBitmap using the ML Kit Face Detection API. It takes the full image bitmap and a callback function faces that will be invoked with the list of detected Face objects.
  • cropBitmapFromFace(fullImageBitmap: Bitmap, face: Face): Bitmap: Crops the fullImageBitmap based on the bounding box of the provided face object. It returns a new bitmap that contains the cropped face image.
  • mergeBitmaps(fullImageBitmap: Bitmap, croppedFaceImageBitmap: Bitmap, face: Face): Bitmap: Merges the croppedFaceImageBitmap into the fullImageBitmap at the bounding box of the provided face

• FaceLandmarkAIInterpreter: This class provides the functionality to process face images and detect facial landmarks using an AI model. It encapsulates the logic for interpreting the AI model's output and extracting the facial landmark positions. Load the FaceLandmark AI model by specifying the model file (face_landmark_model.tflite):

  faceLandmarkInterpreter.loadModel("face_landmark_model.tflite")
  

  An Interpreter object is created to handle model inference. The input tensor's shape is retrieved, and a Bitmap with dimensions 192x192 pixels is created to match the input size required by the model. The output tensor's shape is retrieved, and a TensorBuffer is created to store the model's output predictions. Make sure the model file is placed in the assets directory of your Android project.

  • processFaceImage(faceImage: Bitmap): List: Processes the face image and returns a list of facial landmark positions. It takes the face image as a parameter and returns a list of PointF objects representing the positions of the facial landmarks. Process an input image using the loaded model:

  val inputBitmap: Bitmap = // Load or create a 192x192 size bitmap
  val landmarks: List<PointF> = faceLandmarkInterpreter.processImage(inputBitmap)
  

• FaceLandmarkARProcessor: This package contains functions related to processing face landmarks and applying AR makeup effects to the lips in the input bitmap image.

  • processImageByAR(bitmap: Bitmap, facePos: List, color: Int = Color.RED, radius: Float = 10f, isDrawLandmarks: Boolean = false): Bitmap: Processes the input bitmap image by detecting facial landmarks, applying AR makeup effects to the lips, and optionally drawing facial landmarks. It takes the bitmap image, a list of face landmark positions (facePos), the desired color for the AR makeup effects (color), the radius for drawing facial landmarks (radius), and a flag indicating whether to draw facial landmarks (isDrawLandmarks). Returns the processed bitmap image.
  • drawLandmarks(bitmap: Bitmap, landmarks: List, radius: Float, isDrawLandmarks: Boolean): Bitmap: Draws facial landmarks on the input bitmap image. It takes the bitmap image, a list of facial landmark positions (landmarks), the radius for drawing facial landmarks (radius), and a flag indicating whether to draw facial landmarks (isDrawLandmarks). Returns the bitmap image with facial landmarks drawn.
  • drawRedColorfulMarks(bitmap: Bitmap, facePos: List, color: Int): Applies a red colorful mark effect to the lips in the input bitmap image. It takes the bitmap image, a list of face landmark positions (facePos), and the desired color for the effect (color). This function draws polygons on the lip region with the specified color.

Libraries

• TensorFlow Lite: A library for running machine learning models on mobile devices. It enables the app to perform real-time inference using the AI model for face landmark detection.

• TensorFlow Lite Task Vision: A library for performing vision-related tasks with TensorFlow Lite. It provides the necessary tools and utilities for working with vision models.

• ML Kit Face Detection: A library provided by Google for face detection. It offers pre-trained models and APIs for detecting faces in images and videos.

• CameraX: A Jetpack library that simplifies the camera development process on Android. It provides a consistent API across different Android devices and versions. CameraX consists of multiple modules, including:

  • Camera2 API: A lower-level camera API introduced in Android 5.0 (API level 21) that provides more advanced features and greater control over camera hardware.
    • CameraX CameraView: A pre-built UI component for displaying a camera preview and capturing images or videos. It is built on top of the Camera2 API and provides a higher-level and consistent API.
    • CameraX Lifecycle: A module that provides integration with the Android lifecycle components, allowing for easy management of camera operations during different lifecycle states.

Technological background: iOS

This is the development explanation of our iOS app.

Implementation

1. Set Up the Camera for Video Capture:

   1. Create a capture session to manage video input from the front camera using AVCaptureSession.
   2. Set up the input device by creating an instance of AVCaptureDeviceInput using the camera device and add the input device to the capture session.
   3. Using AVCaptureVideoDataOutput, configure the capture session to receive video frames from the capture session.

2. Set Up Preview Layer:

   1. Create an instance of AVCaptureVideoPreviewLayer to display the camera feed on the screen.
   2. Start the capture session to begin receiving video frames.

3. Set Up Face Detection with Vision Framework:

   1. Adopt the AVCaptureVideoDataOutputSampleBufferDelegate protocol in your view controller.
   2. Implement the captureOutput(_:didOutput:from:) function to receive and process video frames.
   3. Inside captureOutput create a face detection request with an instance of VNDetectFaceRectanglesRequest to detect the face in the captured video frames.
   4. Create a VNImageRequestHandler to handle the captured video frames and perform the face detection request.
   5. Within the completion handler of the face detection request, process the detected face rectangle results to obtain a bounding box (CGRect) that matches the face area.
   6. Implement a cropFace(from:boundingBox:) function to extract the face region from the captured video frame.

4. Set Up TensorFlow Lite To Obtain Face Landmarks:

   1. Prepare a machine learning model that is specifically designed for face landmark detection. This model should be compatible with TensorFlow Lite and trained to detect landmarks on cropped face images.
   2. Load the TensorFlow Lite model (with a .tflite extension) into your Xcode project and ensure it is included in the app target.
   3. Create an instance of Interpreter from TensorFlow Lite using the loaded model file.
   4. Before feeding the cropped face image into the TensorFlow Lite model, you need to preprocess it. This typically involves resizing, normalizing, and converting the image to a suitable format expected by the model. The specific implementation details will depend on the image settings (size, orientation etc) and the input tensor specifications.
   5. Pass the preprocessed cropped face image to the TensorFlow Lite model for inference using the interpreter.
   6. Extract the face landmarks from the output tensor obtained from the TensorFlow Lite model. The specific implementation will depend on the structure of your model and the format of the output tensor.

5. Identify Facial Features and Add Images to Their Positions:

   1. Identify the specific facial features that you want to draw on by using the face landmarks. In this case, the lips.
   2. The face landmarks obtained from the TensorFlow Lite interpreter are typically in normalized coordinates [0, 1] and relative to the cropped face image. To draw the facial features on the video preview layer, convert these landmarks to screen coordinates by using the dimensions of the video preview layer and the position of the cropped face within the video frame to perform the coordinate transformation.
   3. Determine the appropriate visual representation for the lips image (e.g., border lines and filled shapes) by using path.addLine etc and create a corresponding overlay view. Configure the appearance of this image, such as color, line thickness, and transparency, as required.
   4. Add the overlay view as a subview of the video preview layer and adjust the position and size of the view to align with the specific facial features accordingly.

Libraries

• Vision Library Framework: Apply computer vision algorithms to perform a variety of tasks on input images and video.

  • iOS 11.0+, iPadOS 11.0+,macOS 10.13+, Mac Catalyst 13.0+, tvOS 11.0+, visionOS 1.0+ Beta.
  • The Vision framework performs face detection, face landmark detection, text detection, barcode recognition, image registration, and general feature tracking. Vision also allows the use of custom Core ML models for tasks like classification or object detection.

• AVCaptureVideoDataOutput (AVFoundation Framework): A capture output that records video and provides access to video frames for processing.

  • iOS 4.0+, iPadOS 4.0+, macOS 10.7+, Mac Catalyst 14.0+, tvOS 17.0+ Beta.
  • class AVCaptureVideoDataOutput : AVCaptureOutput

• TensorFlow Lite Framework: TensorFlow Lite is a mobile library for deploying machine learning models on mobile, microcontrollers and other edge devices.

  • pod 'TensorFlowLiteSwift', '~> 0.0.1-nightly', :subspecs => ['Metal']

Wrapping up

The integration of AI undoubtedly brings a novel dimension to the capabilities of mobile apps. However, our experiences with this current project, particularly in the domain of image processing, underscore the complexity that can be involved. The power of AI is apparent, yet the challenges it poses require meticulous handling, involving extensive pre-processing and post-processing to attain the desired outcomes.

Truth be told, we embarked on this venture with the choice of a Flutter app development. However, we encountered several impediments with the TensorFlow library's compatibility with Flutter, therefore it led us to pivot towards more established platforms: iOS and Android. While this transition may have led us away from Flutter at this moment, we hold firm in the belief that the rapid advancement of cross-platform capabilities will soon catch up in this field as well.

Looking ahead, our project's trajectory points towards embracing augmented reality (AR) libraries to introduce more complicated elements into the implementation. Specifically, we are currently working on overlaying sunglasses onto the detected faces. As we move forward, the challenges encountered and lessons learned have undoubtedly fueled our determination to strengthen our AI knowledge in mobile app development, propelling us to craft even more captivating and innovative experiences for our clients.

Before we get started:

• Create an account with Spatial
• Install Unity version 2021.3.21 and install the “WebGL Build Support” module with it. If you don’t yet have Unity Hub installed, you will need to install it first
• Download the Spatial Unity Starter Template
• Download this 3D golf ball in .obj file format
• Download this Monstarlab Badge

Start by opening the Spatial Starter template project in Unity. You will be prompted to update to the latest Spatial SDK. Click 'yes' and allow the project to load. We will need to configure our Spatial credentials first. Click on the 'Spatial SDK' tab in the header > Account. This will open the Spatial Portal window (which we will keep coming back to).

Click on "Get Login Token", which will redirect you to your browser where you can copy your token directly to your clipboard. Once you copy it, come back to Unity and click on "Paste Login Token". Now you should be ready and set up to start working!

![Account Config](/assets/img/articles/2023-09-06-How-to-create-an-Interactive-Experience-using-the-Spatial-Creator-Toolkit/AccountConfig.webp

In the Project window, search for "GolfCourseDriving" and open this scene.

In the Spatial Portal window, under the 'Config' tab, set 'Active Package' to 'Space - Golf Course'. Now that we've set this golf course environment as our active scene, we can go ahead and test this scene in Spatial by clicking on 'Test Active Scene' at the top of the window.

![Active Package](/assets/img/articles/2023-09-06-How-to-create-an-Interactive-Experience-using-the-Spatial-Creator-Toolkit/Config.webp

If this is your first time testing a scene on Spatial, it will take some time to load, but subsequent tests will be much quicker. Once it's done, your sandbox environment will open on your browser.

You can walk around using your arrow keys. Walk up to the golf cart and click [F] to drive it. Now your arrow keys will drive the golf cart around the scene! This is already pretty fun, but let's go back to Unity and see how we can add more interactivity to this scene.

Back in Unity, let's change out the default blue sky. Open the lighting window from 'Window' > 'Rendering' > 'Lighting'. Under the 'Environment' tab, click on the small circle next to 'Skybox Material'. This will let you search through your project to replace the material. Search for 'Epic_GloriousPink' and select it.

![Lighting](/assets/img/articles/2023-09-06-How-to-create-an-Interactive-Experience-using-the-Spatial-Creator-Toolkit/Lighting.webp

You will notice instantly that this also changes the ambient light of the scene to a more pink-ish hue. That's because the source of the environment lighting is currently being pulled from the sky, which is what happens in real life. However, if you'd like to control the mood of the scene yourself, you can choose a color from 'Lighting' > 'Environment lighting' > 'Source' > 'Color'. Personally, I like the pink sunset!

Now, let's import the 3D model of the golf ball into the project. You can do this by opening the root of your Unity project in your File explorer/Finder and pasting the 3D model folder inside the 'Assets' folder. For more information on importing 3D objects and how to deal with materials and textures, you can read more here.

From the Project window, search for "Trigger (Collectable)" and drag this prefab directly into your scene. Set its position in the Transform component to (X: 522, Y: 4.48, Z: 654). If this is your first time using Unity, you can read here to learn how to navigate the scene view.

If you look at the 'Spatial Trigger Event' component, you can see a few things that will happen when the player walks into the radius of this object.

![Trigger Collectable](/assets/img/articles/2023-09-06-How-to-create-an-Interactive-Experience-using-the-Spatial-Creator-Toolkit/Trigger1.webp

It's listening for 'Local Avatar', which is any player in Spatial. Once a player enters this trigger, 'On Enter Event' is called and the following happens:

• A coin sound effect is played
• The coin game object is disabled
• A particle effect plays
• The 'Spatial Trigger Component' is disabled

All of these happening in unison gives the impression of the player "collecting" the coin when they walk over it.

Let's switch out the coin model to the golf ball we imported earlier. Do this by disabling the child object "Coin" and dragging in the golf ball model as a child to the 'Trigger (Collectable)' game object. Scale and position it so it's roughly at the same size and position as the coin.

Make sure to switch out the coin for the golf ball in the 'On Trigger Event', so the new 3D model is disabled when collected.

![On Trigger Event](/assets/img/articles/2023-09-06-How-to-create-an-Interactive-Experience-using-the-Spatial-Creator-Toolkit/Trigger2.webp

Now, let's set up a Quest. A Quest is a set of tasks the player needs to complete in order to fulfil a certain goal or objective. We can also reward players when they finish a quest, but more on that later.

Create an empty game object in the scene hierarchy and name it 'Golf Quest'. Click 'Add Component' > 'Spatial Quest'. Give your Quest a name and description. Enable the options as shown below:

![Spatial Quest](/assets/img/articles/2023-09-06-How-to-create-an-Interactive-Experience-using-the-Spatial-Creator-Toolkit/Quest1.webp

Let's add a task under the Tasks list.

![Task List](/assets/img/articles/2023-09-06-How-to-create-an-Interactive-Experience-using-the-Spatial-Creator-Toolkit/Task1.webp

This task shows us that we would need to collect 5 golf balls to complete this task, but we need some way of tracking that a golf ball has been collected. Back in the 'Golf Ball' game object, in the 'Spatial Trigger Event Component' > 'On Enter Event', add a 'Quest Event'. Select the Quest you created, and choose 'Add Task Progress' and 'Golf balls collected'.

![Quest Event](/assets/img/articles/2023-09-06-How-to-create-an-Interactive-Experience-using-the-Spatial-Creator-Toolkit/QuestEvent.webp

Now, when a player walks over this golf ball, it will update the task progress on the Quest. On the Quest game object, we said they would need to collect 5 golf balls, so go ahead and duplicate the golf ball 4 more times and place them around the environment as you like. Once you're done, go ahead and test the active scene to try it out yourself on Spatial!

If you run around and collect the balls, the quest gets updated on the bottom left of the screen. However, if you try driving a golf cart to collect it, you'll notice it's unable to pick it up. Let's fix this.

Back in Unity, select a GolfCart game object. You'll see it's on the 'Vehicle' layer. If you check any of the golf ball triggers, it's listening for a 'LocalAvatar', which would be the player. On the golf cart, switch out its layer to 'AvatarLocal'. The editor will ask if you want to change the layer of all its children objects too. Select 'No, this object only'.

![Game object layers](/assets/img/articles/2023-09-06-How-to-create-an-Interactive-Experience-using-the-Spatial-Creator-Toolkit/Layer.webp

Now, we need to add a box collider around the cart to define the boundary of what needs to collide with the golf ball to trigger its event. On the cart object, click 'Add Component' > 'Box Collider'. Set 'IsTrigger' to true. Edit the box collider to be bigger than the cart.

Go ahead and repeat these steps for all the carts. Test the scene again in your sandbox environment and you'll find you can drive the carts around to collect the golf balls now!

Finally, to reward the players for finishing the quest, we need to publish our space first (it can be set to private). In the Spatial Portal > Config tab, select 'Create New World'and then select 'Publish'. You'll receive an email once your space is successfully published. After that, we can set up a badge in Spatial Studio.

Select your world, and then select Badges > + New Badge. Here, you can upload the Monstarlab badge that you downloaded at the start of this blog (or feel free to create your own!). Give your badge a name and description.

![Badge Description](/assets/img/articles/2023-09-06-How-to-create-an-Interactive-Experience-using-the-Spatial-Creator-Toolkit/Badge.webp

Now you can return to Unity, and go back to your Quest object. Add a 'Quest Reward' and select 'Badge'. Copy and paste the 'Id' from your badge in Spatial Studio.

![Badge ID](/assets/img/articles/2023-09-06-How-to-create-an-Interactive-Experience-using-the-Spatial-Creator-Toolkit/BadgeID.webp

And there you go! Run the scene again, collect all the golf balls, and you should be rewarded with a badge when you're done! 🙂

![Reward](/assets/img/articles/2023-09-06-How-to-create-an-Interactive-Experience-using-the-Spatial-Creator-Toolkit/End.webp

Useful resources

• https://docs.spatial.io/
• https://docs.spatial.io/quests

Article Photo by Spatial

Spatial Unity Starter Template by Spatial

By default, all unit tests written with JUnit and Mockito will be categorised as Unit tests whereas all other types of tests integrate one or more pieces or units of the system.

Here are for example some common context type annotations provided by SpringBoot for integration tests, which can be used to test various components in a backend application:

@DataJpaTest

@DataMongoTest

@WebMvcTest

@SpringBootTest

If you are interested in how to set up an isolated test with SpringBoot and TestContainers, then please refer to our previous blog post provided with Gradle and Kotlin.

The purpose of this article is to share our own experience with a few arguments about how to structure the common type of unit and integration tests (also shortly named as *IT) in Spring Boot with Maven and Java. This type of setup can be very individual and might be considerable only by some engineers. This knowledge share will just demonstrate how to achieve it and how to organise them best, as well as which arguments support this idea.

Below are some reasons which might serve as an argument for setting up your build profiles in this way:

The complexity of unit and integration tests can vary from each other by setup and the amount of components involved. Unit tests are normally short, they use very little dependencies or no dependencies at all and are fast in their execution. The Integration tests on the other hand could have the size from small to huge, depending on the part of the system they will be executed against and often instability might come into question. If the setup requires multiple dependencies which are not just Beans, but an entire system or containers to be provided, they will also run longer. If a problem occurs in a unit test, it will cause the pipeline to fail. On the other hand, some other integration test may have already been executed before this failure, which could have been avoided and postponed for a later phase. Parallel execution of these 2 different types of tests could alert about a failing unit test way earlier, which means putting the shorter and simpler execution beforehand will cost less time and effort by providing early feedback about failures.

Success is measured, and in this scenario, the measurement scale is the code coverage report, which is normally generated and provided with specific tools such as Jacoco with SonarQube visualisation and code quality checks. After unit test execution, Jacoco could help collect reports from different parts of the execution in order to visualise it. Unit tests could for example provide line coverage for the methods that they are bound to. This is not true for the integration tests, because integration tests might aim to test or connect a few units of an application, but in reality, the execution can start or execute many other parts of it: as a result the report will include all parts executed during the test, even though some units weren't intended to be part of the testing at all.

As unit tests are small, they also run fast. On the other hand the integration tests are not only different in size, but also in execution time. This can of course depend on several aspects such as how many beans need to be created, or which context is necessary for the application to prepare, or how many containers or systems should be started in order to get ready for the test completion. This leads the test execution to take much longer and even with some intentional delays put in the code, it might prolong to some minutes.

The more "parties" are involved in the "business" the more problems and blockers are possible. As unit tests normally do not have many services or components as dependencies, they always run stable and are important to fix, because this would be the first quality control. Of course, it is always a good practice to achieve maximum stability for the integration tests, but this is not always possible or should be postponed for internal specific reasons. As many different typesIf the arguments above were convincing, then the example below
of technologies might be used to prepare the test system, it could also be preferred to fix them at a later time.

Technical Setup

If the arguments above were convincing, then the example below should provide a very simple technical setup to create different build profiles which can later be used in CI/CD pipeline configuration.

In the current example a standard SpringBoot application is used with the following components

• Controllers for endpoint calls
• Service layer for repository access
• Repository layer to access PostgreSQL Database

The unit tests mock all dependencies with Mockito and JUnit5 as following:

@ExtendWith(SpringExtension.class)
class UserServiceTest {

    @Mock
    UserRepository userRepository;

    @InjectMocks
    UserService userService;

    @Test
    void findAll() {
        //GIVEN
        User user = new User();
        user.setFirstname("TestName");
        user.setLastname("TestLastname");
        doReturn(new ArrayList<>(){{
            add(user);
        }}).when(userRepository).findAll();

        //WHEN
        List<User> users = userService.findAll();

        //THEN
        assertThat(users.get(0).getFirstname()).isEqualTo("TestName");
        verify(userRepository).findAll();
    }
}

In order to demonstrate the separation we also provided a full isolation test with TestContainers, which tests the following integration in an isolated context: Endpoint Call >> Service Layer >> Repository Layer >> PSQL Instance.

@Testcontainers
@SpringBootTest(classes = TestProfileApplication.class)
@ActiveProfiles("test")
@AutoConfigureMockMvc(addFilters = false)
class UserServiceIT {

  @Autowired
  UserRepository userRepository;

  @Autowired
  MockMvc mockMvc;

  @Test
  void retrieveSavedUser() throws Exception {
    //GIVEN
    var user = new User();
    user.setFirstname("Name");
    user.setLastname("Lastname");
    userRepository.save(user);

    //WHEN
    mockMvc.perform(MockMvcRequestBuilders.get("/user/{id}", 1))

    //THEN
            .andExpect((MockMvcResultMatchers.status().isOk()))
            .andExpect(result -> {
              var content = result.getResponse().getContentAsString();
              Assertions.assertThat(content).contains("Lastname");
            });
  }
}

For simplicity in the structure, both tests follow the famous Given-When-Then pattern for writing the tests.

And finally, to configure it, the following 2 plugin configurations for Surefire and Failsafe are added in pom.xml.

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.0</version>
                <configuration>
                    <skipTests>${skip.unit.tests}</skipTests>
                </configuration>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

If we execute the usual mvn clean isntall in CLI, then both tests would execute after each other without an order.

In order to split the execution we would need to be more specific about which plugin we would like to make part of the maven lifecycle. By default, in SpringBoot and Maven using the following command would skip the Integration Test by scanning the files which end with *IT.java, which of course can also be configured with a different naming convention pattern if needed.

mvn clean -DskipITs=true verify

This will then let the Surefire plugin execute all tests by ignoring *IT tests.

In order to do the opposite, a flag configuration will be held for Surefire to let it ignore the unit test execution phase. In the end, running the following command for Failsafe, will run the IT tests accordingly.

mvn clean -Dskip.unit.tests=true verify

As a result of this setup the pipeline can be configured to run separately to block the follow-up checks based on the results off both execution phases.

Below is a sample from GitHub Actions.

Tools & References used here...

• Monstarlab:: template with Kotlin and Gradle
• Maven as build tool
• Java and OpenJDK as a programming and runtime environment
• Hibernate as an ORM layer for JPA
• PostgreSQL as a Datasource
• FlyWay for DB script migrations
• Spring Boot for Backend Application development
• Test Containers for running PSQL Docker based Container
• Unsplash Images for writing the blog post
• Brain and Mind from Monstarlab::

As AI is taking over the world, call centers will probably be one of the sectors that may be impacted and where humans could be replaced by AI bots (at least, for simple inquiries).

In this article, we will be implementing a prototype of such an application. The technology stack is as follows:

• NodeJS API & Websocket server
• Twilio API (via the official SDK)
• OpenAI API (via the official SDK)
• AmiVoice websocket API

Twilio will be used as our entrypoint for receiving phone calls. We will provision a phone number for Twilio platform and connect it to our NodeJS server on incoming calls.

Then, a voice recognition service (for Japanese language only) called AmiVoice will be used for transcribing the caller voice stream into text by the means of websocket API.

OpenAI API will be used for sending requests with the transcribed texts and getting back the responses. The responses will be relayed back verbally to the caller via Twilio’s internal text-to-speech features.

Architecture

The architecture is quite straight-forward and easy to understand.

1. The caller calls the phone number.
2. Upon receiving the call, Twilio will start a stream and continously provide voice data.
3. That data will be relayed to AmiVoice for speech-to-text analysis.
4. Transcribed texts are received from AmiVoice when a transcription is complete. AmiVoice also takes care of detecting pauses in speech so it will return transcribed texts of complete sentences.
5. The text data is provided to ChatGPT.
6. ChatGPT returns a response.
7. That response is relayed to Twilio.
8. The response (originally in text format) will be spoken via Twilio’s text-to-speech feature to the caller.

Implementation

⚠️ Please note that I didn't include all the code as I want to focus on the main parts of the implementation. The full code can be found here: https://github.com/tonystrawberry/callbot.twilio.amivoice.

Setting ngrok for local testing

Our NodeJS HTTP/websocket server will need to be accessed from the internet. In order to achieve that, let’s setup ngrok and get our ngrok domain.

• Install ngrok on your machine. Installation instructions can be found on the official website.

• We will be using port 3000. Run the command below to provide a tunnel for the internet to access our local server.

  ngrok http 3000
  

• It will display your ngrok domain. In my case below, it is https://df2b-39-110-216-51.ngrok-free.app. Please note that everytime your restart ngrok (on the free version), your domain will change and you will need to reconfigure the Twilio settings accordingly.

![ngrok](/assets/img/articles/2023-08-30-Building-a-conversational-phone-callbot-Twilio-ChatGPT-AmiVoice/ngrok.webp

Setting up Twilio

First, let’s setup Twilio and get our phone number.

• Sign up and log in into your account.
• Buy a phone number. More information can be found here: https://support.twilio.com/hc/en-us/articles/223135247-How-to-Search-for-and-Buy-a-Twilio-Phone-Number-from-Console
• Go to the Voice Configuration for your purchased phone number and set the following configuration:
  • Configure with → Webhook
  • A call comes in → Webhook
  • URL → The ngrok domain appended with /twiml
  • HTTP → HTTP POST

With this, we are done with our Twilio configuration.

Building our HTTP server

Now, let’s dive into the implementation of the application. First, we need to setup a HTTP server for receiving the HTTP POST request from Twilio on incoming calls.

// server.js

const Http = require("http");

const HttpDispatcher = require("httpdispatcher");
const dispatcher = new HttpDispatcher();

const httpServer = Http.createServer(handleRequest);
const HTTP_SERVER_PORT = 3000;

/* POST /twiml */
/* This endpoint is called when the call is first connected */
/* This endpoint needs to be set on Twilio as the Voice Request URL */
/* Reference: https://www.twilio.com/docs/voice/tutorials/how-to-respond-to-incoming-phone-calls/node */
dispatcher.onPost("/twiml", (_request, response) => {
  const twilioVoiceResponse = new VoiceResponse();

  /* Setup the media stream from Twilio to our own websocket */
  twilioVoiceResponse.start().stream({
    url: `wss://${env.NGROK_DOMAIN}`,
  });

  /* Say the first message to the caller */
  twilioVoiceResponse.say(
    {
      voice: "Polly.Takumi-Neural",
      language: "ja-JP",
    },
    "こんにちは。匠です。何でも聞いてください。"
  );

  /* Do not hangup the call for 40 seconds */
  twilioVoiceResponse.pause({ length: 40 });

  response.writeHead(200, { "Content-Type": "text/xml" });

  /* Send the TwiML twilioVoiceResponse back to Twilio */
  /* Example:
  <?xml version="1.0" encoding="UTF-8"?>
  <Response>
    <Start>
      <Stream name="VoiceStream from Twilio" url="wss://df2b-39-110-216-51.ngrok-free.app"/>
    </Start>
    <Say language="ja-JP" voice="Polly.Takumi-Neural">こんにちは。かずはです。何でも聞いてください。</Say>
    <Pause length="40"/>
  </Response>
  */
  log(`Twilio WS: TwiML response sent back: ${twilioVoiceResponse.toString()}`);
  response.end(twilioVoiceResponse.toString());
});

/* Start the HTTP server on localhost with port HTTP_SERVER_PORT */
httpServer.listen(HTTP_SERVER_PORT, () => {
  log(`Server listening on: http://localhost:${HTTP_SERVER_PORT}`);
});

The code above sets the POST /twiml endpoint. A response has to be sent back to Twilio to provide it with instructions on what to do. In our case, we want to Twilio to:

• start a stream to our websocket (wss://54de-126-36-198-149.ngrok-free.app in my case). Documentation about Twilio Voice Stream can be found here.
• make Twilio say an introductory message to the caller. We specified the language ja-JP and the used voice Polly.Kazuha-Neural. A list of usable voices for text-to-speech can be found here.
• pause the call for 40 seconds. Otherwise, the call would end immediately.

That is it for our HTTP server.

Using Twilio VoiceStream

In the previous step, we provided Twilio with a websocket URL. We will now implement that websocket server.

// server.js

const WebSocketServer = require("websocket").server;
const httpServer = Http.createServer(handleRequest);

/* Setup the websocket server */
const websocketServer = new WebSocketServer({
  httpServer: httpServer,
  autoAcceptConnections: true,
});

websocketServer.on("connect", connection => {
  log("Twilio WS: Connection accepted");
  new VoiceStream(connection);
});

When Twilio connects to websocket server, we will instantiate a VoiceStream object that will manage that connection. Implementation is as follows.

// server.js

/* VoiceStream class that will store the Websocket connection [Twilio] <-> [Websocket server] */
/* This class will handle the media stream from Twilio */
/* It will also handle the transcription from AmiVoice */
class VoiceStream {
  constructor(connection) {
    // processMessage is called when a message is received from Twilio
    connection.on("message", message => {
      this.processMessage(message);
    });

    // close is called when the connection is closed by Twilio
    connection.on("close", () => {
      log("Twilio WS: Connection closed by Twillio");
      this.close();
    });

    this.messageCount = 0;
    this.amiVoiceConnection = null;
    this.callSid = ""; // unique call identifier from Twilio
  }

  /* This function is called when a message is received from Twilio */
  /* The message will be a JSON object */
  /* Reference: https://www.twilio.com/docs/voice/tutorials/consume-real-time-media-stream-using-websockets-python-and-flask */
  processMessage(message) {
    if (message.type === "utf8") {
      const data = JSON.parse(message.utf8Data);
      switch (data.event) {
        case "connected":
          // This event is received when the connection is first established
          // Example: { event: "connected", protocol: "Call", version: "0.2.0" }
          log("Twilio WS: Connected event received: ", data);
          break;
        case "start":
          // This event is received when the stream is started
          // We will store the callSid and create an AmiVoice connection
          // Example: {
          //   event: "start",
          //   sequenceNumber: "1",
          //   start: {
          //     accountSid: "AC0d7016bb1842a6b3e80d1d6d56036784",
          //     streamSid: "MZ639f5aaf9b0c0fe84f01b5e8478c7d52",
          //     callSid: "CAea9962c5d642db0576ffd21fa1b9d6ad",
          //     tracks: [ "inbound" ],
          //     mediaFormat: { encoding: "audio/x-mulaw", sampleRate: 8000, channels: 1 }
          //   },
          //   streamSid: "MZ639f5aaf9b0c0fe84f01b5e8478c7d52"
          // }}
          log("Twilio WS: Start event received: ", data);

          this.callSid = data.start.callSid;
          this.amiVoiceConnection = new AmiVoice(env.AMIVOICE_API_KEY);

          // When the AmiVoice connection receives a transcription, we will process it with ChatGPT and send it back to Twilio
          this.amiVoiceConnection.on("transcription", transcriptionData => {
            this.receivedTranscription(transcriptionData, this.callSid);
          });
          break;
        case "media":
          // This event is received continuously while the stream is active
          // It contains the audio data from Twilio that we will send to AmiVoice
          // Example: {
          //   event: 'media',
          //   sequenceNumber: '256',
          //   media: {
          //     track: 'inbound',
          //     chunk: '255',
          //     timestamp: '5180',
          //     payload: 'fv//fv///37///////////////////////////9+//9+////fv///37/////fv////////////////9+//////////9+////fn7/////////////////////fv////9+//9+/////////////////37//////////37//////////37//////////////////37/////////////fv9+/w=='
          //   },
          //   streamSid: 'MZ1476190b98d7e720c314ecc9cde50b73'
          // }
          this.amiVoiceConnection.send(data.media.payload);
          break;
        case "closed":
          // This event is received when the stream is closed
          // Example: { event: "close", streamSid: "MZ639f5aaf9b0c0fe84f01b5e8478c7d52" }
          log("Twilio WS: Close event received: ", data);
          this.close();
          break;
      }

      this.messageCount++;
    } else if (message.type === "binary") {
      log("Twilio WS: binary message received (not supported)");
    }
  }

  // This function is called when the connection is closed
  // We will close the AmiVoice connection as well
  close() {
    log(
      "Twilio WS: Closed. Received a total of [" +
        this.messageCount +
        "] messages"
    );
    this.amiVoiceConnection.close();
  }

  // This function is called when a transcription is received from AmiVoice
  // We will send the transcription to ChatGPT and send the response back to Twilio via API
  // We can identify the call using the callSid that we stored earlier and pass the ChatGPT response to Twilio
  // Reference: https://www.twilio.com/docs/voice/tutorials/how-to-modify-calls-in-progress/node
  async receivedTranscription(data, callSid) {
    log(`Twilio WS: Received message from AmiVoice: ${data.body.text}`);

    // Send a <Say> message to Twilio that will be read back to the caller
    const completion = await openai.createChatCompletion({
      model: "gpt-3.5-turbo",
      messages: [{ role: "user", content: data.body.text }],
    });

    // Update the call with the <Say> message
    const chatGptMessage = completion.data.choices[0].message.content;
    const twiml = new VoiceResponse();
    twiml.say(
      {
        voice: TWILIO_VOICE,
        language: TWILIO_LANGUAGE,
      },
      chatGptMessage
    );
    twiml.pause({ length: 40 });

    twilioClient
      .calls(callSid)
      .update({ twiml: twiml.toString() })
      .then(_call =>
        log(
          `Twilio WS: ChatGPT response sent back with TwiML: ${chatGptMessage}`
        )
      );
  }
}

Throughout the call, Twilio will send to the websocket server four types of message:

• connected : event notifying that Twilio connects to our websocket for the first time.
• start : event notifying that the stream has started. Useful informations such as callSid (unique identifier for the call) are provided in the body. We will use that callSid later to send back verbal messages from ChatGPT to the call. Besides, we will setup the connection with AmiVoice websocket API and a callback method for processing transcribed messages received.
• media : event containing the voice data of the caller. That data will be sent to AmiVoice websocket API for transcription.
• closed : event notifying the end of the connection. Received when the user hangs up or if an error occured.

Transcribing with AmiVoice WebSockets API

// amivoice.js

/*
  This class will store the connection to the AmiVoice API.
  It will also handle the connection and disconnection of the service.
  It will also handle the sending of audio to the service and the sending of the transcription to the server via EventEmitter.
*/

const EventEmitter = require("events");
const WebSocketClient = require("websocket").client;

const AMIVOICE_WEBSOCKET_URL = "wss://acp-api.amivoice.com/v1/";

class AmiVoice extends EventEmitter {
  constructor(apiKey) {
    super();
    this.apiKey = apiKey;
    this.isReady = false;
    this.connect();
  }

  /* connect() will create a new connection to the AmiVoice API. */
  /* All the listeners are set up here. */
  /* Reference: https://docs.amivoice.com/amivoice-api/manual/user-guide/request/websocket-interface */
  connect() {
    if (!this.isReady) {
      const client = new WebSocketClient();

      client.on("connectFailed", error => {
        log("AmiVoice: " + error.toString());
        this.isReady = false;
      });

      client.on("connect", connection => {
        this.connection = connection;
        log("AmiVoice: websocket client connected");

        connection.on("message", message => {
          if (message.type === "utf8") {
            // code is the first character of the message and is used to determine what the message is
            // Reference: https://docs.amivoice.com/amivoice-api/manual/reference/websocket/packet/packet-state-transition
            // A: recognition processing completed and recognition result accepted
            const code = message.utf8Data.charAt(0);

            switch (code) {
              case "A":
                const data = JSON.parse(message.utf8Data.substring(2));
                log(`AmiVoice: [A] ${data.text}`);
                this.emit("transcription", { body: data });
                break;
            }
          }
        });

        connection.on("close", () => {
          this.isStarted = false;
          log("AmiVoice: connection closed");
        });

        this.isReady = true;
        this.start();
      });

      client.connect(AMIVOICE_WEBSOCKET_URL);
    }
  }

  // send() will send the audio to the AmiVoice API
  // The audio is sent as a base64 encoded string in the payload so it must be decoded first
  // Then the payload is converted to a Uint8Array and the first byte is set to 0x70
  // Then the Uint8Array is converted to a Buffer and sent to the AmiVoice API
  // Reference: https://docs.amivoice.com/amivoice-api/manual/reference/websocket/command/p-command-packet
  // Format: p<audio_data>
  // According to the documentation:
  //   `<audio_data>は、セッション開始時の s コマンドで指定した音声フォーマットの音声データです。
  //   この音声データの先頭に、0x70（ 'p' のアスキーコード）を付け、バイナリフレームで送信します。`
  send(payload) {
    if (this.isStarted) {
      const buff = Buffer.from(payload, "base64");

      const outData = new Uint8Array(buff.length + 1);
      outData[0] = 0x70; // "p"
      for (let i = 0; i < buff.length; i++) {
        outData[1 + i] = buff[i];
      }

      this.connection.send(Buffer.from(outData));
    }
  }

  // close() will send the close command to the AmiVoice API
  // Reference: https://docs.amivoice.com/amivoice-api/manual/reference/websocket/command/e-command-packet
  // Format: e
  close() {
    if (this.isStarted && this.isReady) {
      log("AmiVoice: send close command");
      const endCommand = "e";
      this.connection.send(endCommand);
      this.isStarted = false;
    }
  }
}

module.exports = {
  AmiVoice,
};

The AmiVoice class will manage the AmiVoice websocket connection and provide methods for sending data (send()) and receiving the transcribed data (A message inside the connection.on("message") block).

• send(payload) function is used for relaying the data from Twilio to AmiVoice and is called in server.js.
• When AmiVoice is done with a transcription task, it will send back to us a message with a code A accompanied with the transcribed text. Upon receiving it, we will emit (we used EventEmitter library for this) that transcription back to our VoiceStream so that we can process it.

I will skip the detailed explanations of the other parts of the code as they are very specific to AmiVoice. More information about AmiVoice (in Japanese) can be found here.

Getting & relaying back ChatGPT answers

Finally, let’s talk to Mr. ChatGPT. We got the transcribed text of the caller voice and will provide ChatGPT with it.

// amivoice.js

case "A":
  const data = JSON.parse(message.utf8Data.substring(2));　// to exclude the A code from the body
  this.emit("transcription", { body: data });
  break;

// server.js

// When the AmiVoice connection receives a transcription, we will process it with ChatGPT and send it back to Twilio
this.amiVoiceConnection.on("transcription", (transcriptionData) => {
  this.receivedTranscription(transcriptionData, this.callSid);
});

...

// This function is called when a transcription is received from AmiVoice
// We will send the transcription to ChatGPT and send the response back to Twilio via API
// We can identify the call using the callSid that we stored earlier and pass the ChatGPT response to Twilio
// Reference: https://www.twilio.com/docs/voice/tutorials/how-to-modify-calls-in-progress/node
async receivedTranscription(data, callSid) {
  log(`Twilio WS: Received message from AmiVoice: ${data.body.text}`);

  // Send a <Say> message to Twilio that will be read back to the caller
  const completion = await openai.createChatCompletion({
    model: "gpt-3.5-turbo",
    messages: [{role: "user", content: data.body.text}],
  });

  // Update the call with the <Say> message
  const chatGptMessage = completion.data.choices[0].message.content;
  const twiml = new VoiceResponse();
  twiml.say({
    voice: TWILIO_VOICE,
    language: TWILIO_LANGUAGE,
  }, chatGptMessage);
  twiml.pause({ length: 40 });

  twilioClient.calls(callSid)
    .update({twiml: twiml.toString()})
    .then(_call => log(`Twilio WS: ChatGPT response sent back with TwiML: ${chatGptMessage}`));
}

The receivedTranscription called upon receiving the transcription is quite compact and straight-forward:

• It will send the transcription to text to ChatGPT using openai library and get back the the answer.
• A VoiceResponse will be built with the ChatGPT message and other settings such as voice, language , pause and update the current call identified by the callSid. More information here.

Now, the application’s implementation is done. We are now able to call a phone number and talk to ChatGPT 🎉 Here is a demonstration video about the prototype we just built.

Demo

Conclusion

Thank you for reading! With simple libraries and services, we managed to make a pretty awesome AI callbot for talking about anything. Full code with installation instructions can be found here: https://github.com/tonystrawberry/callbot.twilio.amivoice.

Of course, we could have considered other services such as Amazon Connect coupled with Lambda instead of Twilio or Amazon Transcribe instead of AmiVoice API. I chose Twilio over Amazon Connect for this prototype for the simplicity of installation and usage and AmiVoice seems to be better at recognizing and transcribing Japanese speech than Amazon Transcribe.

Feel free to contact me for any questions: https://github.com/tonystrawberry/

Reference: https://cloudapi.kddi-web.com/magazine/twilio-media-stream/twilio-amivoice-speech-recognition-method

Article Photo by Andy Kelly

• The code is based on the Connect-go tutorial.
• We'll build a container development environment using Dev Container,
• Introduce Hot Reload using tools like Air
• Demonstrate the use of protoc-gen-validate for validation.

To focus on setting up the development environment, I'll exclude technical explanations or architecture design related to gRPC and Connect, as well as topics like CI/CD, from this article. You can find the repository for the sample code here.

Operating Environment

I have conducted operational tests in the following environment:

• M1 MacBook Pro (Ventura 13.4.1)
• Docker Desktop for Mac Version 4.20.1
• VSCode Version 1.79.2
• Dev Containers v0.295.0

Technology Stack

My technology stack includes:

• VSCode Dev Containers
• Golang
• Air
• Connect-go
• protoc-gen-validate(PGV)

Setting Up the Environment

Personally, I believe there are three key criteria for creating the optimal development environment:

1. Maintaining a certain level of code quality without relying solely on human attention.
2. Compiling written code quickly to receive immediate feedback.
3. Allowing anyone to start development promptly.

Taking these criteria into account for gRPC development environment, I have selected the following tools and libraries: VSCode Dev Container as the IDE, Golang as the programming language, Connect as the gRPC library, and libraries like Air for achieving Hot Reload. The tools chosen for this setup are considered industry standards and were also selected based on personal preferences after testing and evaluation.

Setting Up VSCode Dev Container

I added configurations based on the Go template created with the "Add Dev Container Configuration Files" command from the Dev Containers extension. To install Go configurations in the container, I set up the container build using a Dockerfile. The other main configuration items are primarily these three:

• Introducing Features.
• Configuring VSCode settings and introducing extensions.
• Installing necessary tools for Go development in the Dockerfile.

While there are other things I might want to do, such as setting up git pre-commit configurations, I prioritized starting gRPC development and kept the setup minimal.

Introducing Features

Configuring Oh My Zsh

To establish my shell environment, I have adopted Oh My Zsh. Since people often have their own preferences when it comes to shells, it's best to choose something you like. Here are the steps to introduce Oh My Zsh to the Dev Container:

Installing Zsh and Oh My Zsh

By installing the common-utils feature, you can incorporate both Zsh and Oh My Zsh. Configure it within the "features" section in the devcontainer.json file like this:

{
  // // Features to add to the dev container. More info: https://containers.dev/features.
  "features": {
    "ghcr.io/devcontainers/features/common-utils:2": {
      "configureZshAsDefaultShell": true
    }
  }
}

Setting Zsh as the Default Shell

You can change the default shell to Zsh by adding the following script to the Dev Container's dockerfile. Reference: stackoverflow

RUN echo "if [ -t 1 ]; then" >> /root/.bashrc
RUN echo "exec zsh" >> /root/.bashrc
RUN echo "fi" >> /root/.bashrc

Configuring Oh My Zsh in .zshrc

Include the configuration for Oh My Zsh in the .zshrc file within the .devcontainer directory. You can use the zsh-template from Oh My Zsh for this purpose.

Copying .zshrc to the Container

Finally, add the following line to your Dockerfile to copy the .zshrc file to the home directory within the container:

ADD .zshrc $HOME

Configuring VSCode Settings and Introducing Extensions

Now let's dive into configuring VSCode. We'll start with the settings.json configuration, where I will use the official default settings. You can customize these settings to match your personal preferences. When it comes to introducing extensions, this is the part where personal preferences shine through the most. In my case, I went with the following setup. Since these are all well-known extensions, I'll skip detailed descriptions for each. Just remember to add the necessary configurations to your settings files as needed.

In your .devcontainer.json:

{
  "customizations": {
    "vscode": {
      "extensions": [
        "EditorConfig.EditorConfig",
        "streetsidesoftware.code-spell-checker",
        "wayou.vscode-todo-highlight",
        "ms-azuretools.vscode-docker",
        "shardulm94.trailing-spaces",
        "bungcip.better-toml"
      ]
    }
  }
}

Configuring Dockerfile for Golang Development

An essential part of Golang development is the Go extension from Extensions. Installing this extension is enough to get started with development.

In your .devcontainer.json:

{
  // Configure tool-specific properties.
  "customizations": {
    "vscode": {
      "extensions": ["golang.go"]
    }
  }
}

In the Dockerfile, you'll want to install dependencies required by the Go extension, such as gopls and dlv, along with any other necessary tools to complete your Golang development environment.

Dockerfile

RUN go install github.com/cweill/gotests/gotests@latest
RUN go install github.com/josharian/impl@latest
RUN go install github.com/go-delve/delve/cmd/dlv@latest
RUN go install honnef.co/go/tools/cmd/staticcheck@latest
RUN go install golang.org/x/tools/gopls@latest

Setting Up Connect-go Environment

With the Golang environment set up as described in the previous sections, we can now proceed to set up Connect-go. Essentially, by following the tutorial, you can easily get Connect-go up and running.

Introducing Extensions

For the Dockerfile, add the following commands based on the tutorial's instructions:

Dockerfile

RUN go install github.com/bufbuild/buf/cmd/buf@latest
RUN go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest
RUN go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
RUN go install github.com/bufbuild/connect-go/cmd/protoc-gen-connect-go@latest

Writing Sample Code

Following the tutorial, create go.mod, buf.yaml, and buf.gen.yaml files, write your main code, and execute main.go to complete the Connect-go setup.

Introducing Hot Reload

To enhance the development environment, we will introduce Hot Reload. Hot Reload requires two steps:

1. Automatic code generation.
2. Hot Reload for Golang code.

Automatic Code Generation

For automatic code generation, utilize the buf generate command upon saving proto files. You can achieve this using the "Run on Save" extension. Add "emeraldwalk.RunOnSave" to your devcontainer.json and configure it in .vscode/settings.json:

{
  // Run on Save
  // proto-gen
  "emeraldwalk.runonsave": {
    "commands": [
      {
        "match": "\\.proto$",
        "cmd": "rm -rf gen | buf generate"
      }
    ]
  }
}

This cmd is used to remove all files within the gen directory. This precaution is taken because the buf generate command might not remove inadvertently generated files.

As the scale of your project increases, I think I will need to reconsider it and potentially implement a more refined approach.

Introducing Golang's Hot Reload with Air

For Golang's Hot Reload, we've introduced the use of Air. There are various ways to install it, but by adding RUN go install github.com/cosmtrek/air@latest to your Dockerfile, the setup is complete. The configuration file needs to be set up as well. Running the air init command within the container will generate the .air.toml file, which you can modify based on the .air_sample.toml, adapting it for your project's structure.

In this setup, we've set delay=500. This delay is important because without it, when RunOnSave operates, it might not be able to read files under the gen directory.

Once the Hot Reload configuration is in place, make sure Air automatically starts after the container is up. Use the postStartCommand in your devcontainer.json:

{
  "postStartCommand": "air -c .air.toml"
}

With these steps, your Hot Reload configuration is now fully set up.

Adding Request Validation Feature

Now, let's move on to adding the request validation feature, something that would be useful in practical usage. We'll use protoc-gen-validate (PGV) for Connect's validation.

Add RUN go install github.com/envoyproxy/protoc-gen-validate@latest to your Dockerfile. Then, as documented, add the necessary configurations to buf.gen.yaml and buf.yaml. Be careful not to forget adding opt according to your environment's needs.

buf.gen.yaml

plugins:
  - plugin: buf.build/bufbuild/validate-go
    out: gen
    opt: paths=source_relative

buf.yaml

deps:
  - buf.build/envoyproxy/protoc-gen-validate

After this, run the buf mod update command to generate the buf.gen.yaml file. Be aware that automatically running bef generate might prevent you from receiving error messages. If things don't work smoothly, manually running the commands is recommended.

Now, let's implement validation. Import validate in your greet.proto file and modify the validation rules for GreetRequest, like so:

greet/v1/greet.proto