Edge infrastructure decisions tend to stay invisible until scale, security, or compliance constraints start affecting day-to-day operations. At that stage, the trade-offs between managed platforms and self-assembled architectures become impossible to ignore.

As edge logic grows in volume and operational importance, the manner in which it is expressed and governed starts to matter. Auditing changes, enforcing consistency across environments, and treating edge behavior as deployable code become harder when edge configuration is managed through a web console instead of being versioned and reviewed as code.

In this context, companies often choose to migrate their edge infrastructure from Cloudflare to Amazon CloudFront and AWS Web Application Firewall (WAF) to regain control and align edge behavior with infrastructure-as-code (IaC) practices. However, such migrations are rarely simple lift-and-shift exercises. While Cloudflare abstracts most edge concerns behind a unified interface, AWS exposes them as discrete building blocks that must be explicitly designed, wired, and maintained.

Rebuilding Edge Capabilities on AWS

This article documents a real-world migration from Cloudflare to a CloudFront and WAF based architecture, driven by the need for stronger control, auditability, and infrastructure-as-code (IaC) enforcement.

Several capabilities that were implicit on Cloudflare had to be rebuilt explicitly, including redirects, rate limiting, persistent blocking, image optimization, and testing. The objective was not to copy Cloudflare’s features one configuration block at a time, but to arrive at a serverless edge architecture defined entirely in Terraform, with predictable behavior and clear operational ownership.

Routing & Redirects

The migration challenge becomes clearer by comparing how routing and redirects are handled on each platform:

• Cloudflare Page Rule: Follow a straightforwardIf URL matches X, Then do Y logic. They are used for redirects, forcing HTTPS, bypassing cache, and similar edge logic. The model is configuration-driven and flexible, with little friction as rules accumulate.
• AWS Ordered Cache Behavior: In Amazon CloudFront, ordered cache behaviors are heavyweight configuration objects designed to route traffic to different origins, such as /api/* to Lambda or /static/* to S3. They are not intended for simple conditional logic. From an infrastructure perspective, they are architectural primitives, verbose to express in Terraform, and limited to 75 behaviors per distribution.

The Classic AWS Mistake: Overusing ordered_cache_behavior

A common reflex during an AWS migration is to translate every Cloudflare Page Rule into an ordered_cache_behavior block in Terraform. Even though AWS has raised the default limit to 75 behaviors per distribution, pushing the configuration in that direction does not scale in practice.

This approach breaks down for two reasons.

1. The Maintenance Cost: Terraform is verbose. Managing 20 redirects via cache behaviors generates a lot of lines of HCL code. Changing a single security header policy requires updating 20 blocks. It is error-prone and tedious. While Terraform modules can mitigate code verbosity (DRY), they do not solve the underlying architectural inefficiency or the deployment latency caused by managing dozens of distinct behaviors.
2. The Deployment Fatigue: CloudFront distribution updates are inherently slow. Each terraform apply involving many cache behaviors forces the CloudFront API to validate and reprocess the entire behavior chain. As the configuration grows, CI/CD pipelines become noticeably slower and less predictable.

The Solution: One Global Edge Router

Rather than defining a separate infrastructure block for every redirect, a single global redirect function is used.

The function is attached to the default cache behavior, allowing it to intercept all incoming requests. It effectively acts as a lightweight edge router. The execution flow is simple:

1. Inspect the requested URL
2. Look up the URL in a centralized mapping
3. If a match exists, return a 301 Moved Permanently response immediately
4. If no match is found, forward the request to the origin without modification

function handler(event) {
   const request = event.request;
   const uri = request.uri;

   const redirectMap = {
       '/a/nice/path':   'https://to/the/new/path',
       '/another/path': 'https://to/another/new/path',
   };

   if (redirectMap[uri]) {
       return {
           statusCode: 301,
           statusDescription: 'Moved Permanently',
           headers: {
               'location': { value: redirectMap[uri] }
           }
       };
   }
   return request;
}

Security: Persistent Blocking on AWS WAF

This is where the behavioral gap between platforms becomes most visible.

On Cloudflare, an IP repeatedly hitting a sensitive endpoint can be handled with a simple rate-limiting rule and a fixed block duration. The problem is effectively solved at the configuration level.

AWS WAF does not offer an equivalent mechanism. It provides rate-based rules, but their behavior is transient and threshold-driven, closer to a revolving door than a sustained block:

1. An attacker IP exceeds the request threshold (ex: 100 requests per 5min)
2. AWS WAF blocks the IP
3. Attacker stops (or gets blocked) for 5 minutes
4. The rate drops below the threshold
5. AWS releases the IP immediately
6. The attacker starts again

Achieving persistent blocking exposes a core limitation of AWS WAF. The service is stateless and does not retain historical context beyond the evaluation window.

The Solution: A Stateful Penalty Box

AWS WAF does not provide a built-in mechanism for persistent blocking, which makes it necessary to introduce State on top of an otherwise stateless system.

The solution is composed of four parts:

• Sensor: A WAF rate-based rule used to detect abusive traffic
• Logic: A Lambda function triggered every minute via EventBridge
• Memory: An S3 JSON file storing IP addresses and their expiration timestamps
• Blocker: A WAF IP set used as a blocklist

How it works: Every minute, the Lambda function runs and queries AWS WAF for IPs currently violating the configured rate-based rules. For each IP, a release time is calculated based on the configured blocking duration, for example current time plus 24 hours. This information is written to S3.

On each execution, the Lambda also rebuilds the list of IPs whose release time has not yet expired and pushes this consolidated list to the WAF IP set attached to the Web ACL.

Existing Solutions vs. Our Needs

This problem is not unique. AWS provides a reference implementation on GitHub that demonstrates persistent blocking with AWS WAF. While useful as a proof of concept, it does not meet production requirements for two reasons.

1. The Terraform Gap: The AWS-provided solution is implemented entirely in CloudFormation. For environments standardized on Terraform, this introduces an inconsistency in tooling and workflow.
2. The Scalability Issue: The reference implementation maps a single WAF rule to a single blocking duration. Handling different threat levels requires deploying the stack multiple times. For example, aggressive login brute-forcing might require a 24-hour ban, while low-intensity scraping might only justify a 10-minute block. Each case would require a separate deployment.

The Upgrade: A Multi-Rule Engine

The logic was rewritten from scratch in Node.js and Terraform to support vector-based configuration. Instead of relying on a single variable, the Lambda function receives a JSON configuration that maps WAF rule names to ban durations:

{
  "RateLimit_Login_High": 1440,  // Ban 24h
  "RateLimit_Search_Low": 10     // Ban 10m
}

On each execution, the Lambda iterates over this configuration. When an IP violates a given rule, the corresponding ban duration is applied. This transforms a simple script into a mechanism capable of handling multiple threat levels within a single execution flow.

The IPv6 Trap

One detail that Cloudflare abstracts away but AWS exposes directly is IP address handling. AWS WAF requires separate IP sets for IPv4 and IPv6 addresses. Mixing both formats in a single list is not supported.

As a result, the implementation maintains separate blocklists (PenaltyBox-IPv4 and PenaltyBox-IPv6) independently. Failing to do so leaves roughly 40 percent of modern mobile traffic unprotected, as many mobile networks default to IPv6.

Image Optimization: Where is the “Polish” button?

On Cloudflare, image optimization is enabled by toggling a feature called Polish. Large JPEGs are automatically delivered as optimized WebP or AVIF images without any additional configuration.

On AWS, CloudFront behaves differently. It delivers exactly what is stored in the origin, with no built-in image transformation or format negotiation.

The Solution: Dynamic Image Transformation

To address this, the official AWS Solution: Dynamic Image Transformation was deployed.

The architecture places a Lambda function behind CloudFront that processes images on the fly using the Sharp library. A Lambda-based implementation was chosen, although an ECS-based alternative is also available.

Just like Cloudflare, the configured Lambda handles format conversion automatically. When a user requests <img src=”/hero.jpg”>, the Lambda checks the browser’s Accept header. If the browser supports it, the Lambda converts the image to WebP or AVIF and optimizes the quality.

For this to work correctly, the Accept header must be explicitly whitelisted in the CloudFront cache policy. By default, CloudFront removes most headers to maximize cache hit ratios. Without this configuration, the Lambda function cannot detect format support and will serve the original JPEG to all clients.

Handling this explicitly restores the simplicity of automatic compression while also enabling dynamic resizing for responsive image delivery.

Testing Edge Behavior Reliably

Configuring complex WAF rules and geo-blocking is one challenge. Verifying that those rules behave as expected, without relying on VPNs or location-based workarounds, is another.

Manual curl commands and ad-hoc shell scripts do not scale for this purpose. Edge behavior needed to be tested in a way that was repeatable, portable, and as codified as the infrastructure itself.

Choosing the Right Tool: Japa vs. Hurl

A JavaScript test runner such as Japa was an initial option, given its existing use for backend testing. It offers full language expressiveness and familiar syntax.

For infrastructure and edge testing, this approach introduced unnecessary overhead. Two constraints stood out:

• Overhead: Running a Node.js test runner requires bootstrapping a runtime, managing npm dependencies, and writing asynchronous test code, even when the goal is to validate a simple HTTP response.
• Portability: These tests needed to run in lightweight CI environments and be usable by operations teams, without requiring a full development setup.

Hurl was selected instead. It is a lightweight command-line tool written in Rust that focuses exclusively on HTTP integration testing. Requests and assertions are defined in a plain text format, making tests easy to read, version, and execute. Its narrow scope matches the problem space well.

Testing a caching rule with Hurl:

HEAD {{host}}/
HTTP 200


HEAD {{host}}/
HTTP 200
[Asserts]
header "X-Cache" contains "Hit from cloudfront"


header "Age" exists
header "Age" toInt >= 0

Caching validation is performed using two consecutive requests. The first request populates the cache. The second request asserts that the response is served from CloudFront, indicated by a cache hit and the presence of an Age header.

More complex behavior can be validated just as directly. For example, redirect validation is reduced to asserting a 301 status code and an exact match on the Location header.

GET {{host}}/oldpath
HTTP 301
[Asserts]
header "Location" == "https://mywebsite.com/newpath"

Testing the Geo-Blocking

Geographic restrictions are difficult to validate without relying on VPNs or physical location changes. This is addressed by introducing a controlled testing mode based on the X-Forwarded-For header.

By default, AWS WAF evaluates the source IP of the TCP connection. Using a Terraform dynamic block, WAF can be instructed to trust the X-Forwarded-For header only when testing is explicitly enabled.

dynamic "forwarded_ip_config" {
  for_each = var.enable_testing_mode ? [1] : []
  content {
    header_name       = "X-Forwarded-For"
    fallback_behavior = "MATCH"
  }
}

When enable_testing_mode is set to true in a staging environment, AWS WAF evaluates the injected IP address instead of the physical source IP. This makes it possible to use Hurl to simulate requests from specific geographic locations and validate geo-blocking rules deterministically.

Conclusion

Migrating from Cloudflare to Amazon CloudFront and AWS WAF is not without cost. It requires engineering effort, debugging time, and a deep understanding of AWS primitives. Convenience is traded for ownership, and implicit platform features are replaced with explicit design decisions.

However, over time, the ROI becomes clear:

1. Immutable Infrastructure: Edge behavior is no longer modified through ad-hoc console changes. Redirects, security rules, and routing logic are fully defined in Terraform. Rolling back a WAF rule becomes a predictable deployment rather than a manual intervention.
2. Granular Control: Behavior is no longer constrained by predefined platform features. Blocking durations, routing logic, and image optimization strategies are defined explicitly and can be adapted as requirements evolve.
3. Observability: Logs and metrics are fully owned. Traffic and security events can be streamed directly to CloudWatch and queried with Logs Insights, without reliance on vendor-specific dashboards or plan limitations.

Companies considering a similar migration, or facing comparable edge architecture constraints, can reach out to TrackIt for guidance and hands-on support.

Rebuilding Edge Infrastructure on AWS: Lessons from a Cloudflare to CloudFront Migration was originally published in TrackIt on Medium, where people are continuing the conversation by highlighting and responding to this story.

For me, this project was the logical next step in my automation journey. I had already streamlined my infrastructure by automating the installation of Debian using Proxmox templates. I could spin up a fresh, secure OS in seconds, but the automation stopped there. I still found myself manually configuring the Bitcoin software on top of those fresh VMs. It was a friction point that undermined the speed of my Proxmox setup.

I wanted a better way. I wanted a solution that adhered to the principles of Infrastructure as Code to bridge that final gap. I wanted a reproducible, secure, and automated deployment process that I could audit and version control. That is why I built ansible-bitcoin-node.

But this automation is more than just a convenience, it is a strategic building block. This role serves as a foundational layer for a much larger personal project: I am currently developing a “Bitcoin Node as a Service” platform. If you want to be the first to know when it launches, check it out here.

In this article, I will share why I chose Ansible for this task, how the role is architected, and how you can use it to spin up your own node in minutes.

Why Ansible ?

For me, the choice was dictated by the reality of my current infrastructure. My homelab is currently a hybrid environment, services are split about 50/70 between virtual machines and Kubernetes. While I might eventually migrate fully to a pure Kubernetes cluster, managing VMs remains a core part of my operations. I needed a tool that fit this transitional hybrid phase perfectly, allowing me to treat my traditional VMs with the same rigor and reproducibility as my containerized workloads.

Project Overview & Architecture

The goal of this role is simple: take a fresh Linux server (Debian/Ubuntu) and turn it into a fully functioning, secure Bitcoin node with zero manual intervention.

Don’t hesitate to open PR on github if my Ansible role work with other version of Debian or Ubuntu not listed here, or if you want to add some distributions like Fedora for example.

Tech Stack

• Target OS: Debian / Ubuntu (tested on Debian 13)
• Automation: Ansible
• Software: Bitcoin Knots (Default) or Bitcoin Core (Optional)
• Process Management: Systemd

Key Features

I designed this role to follow security best practices strictly while offering flexibility in the choice of implementation.

• Knots by Default, Core Available: By default, the role installs Bitcoin Knots. I chose Knots as the default implementation because of its advanced features and enhanced mempool policies, which give node runners more control over the transactions they relay. However, if you prefer the standard implementation, you can switch to Bitcoin Core simply by changing a variable.
• Security & User Management: The role creates a dedicated system user (default: bitcoin): the Bitcoin daemon never runs as root.
• Trustless Installation: It doesn’t just download the binary. It downloads the checksums and verifies the GPG signatures against the developers’ keys (Luke Dashjr for Knots, or the Core maintainers). If the signature doesn’t match, the deployment fails immediately.
• Systemd Integration: It installs a robust systemd service file. This ensures that bitcoind starts automatically on boot and restarts in case of a crash.

Deploy your Node

The role is available on Ansible Galaxy so no need to clone the repository and install it manually.

The tutorial is basically the same as the project readme file.

Install the role

Pull the role directly from Galaxy to your machine:

ansible-galaxy role install MaximeMRF.ansible-bitcoin-node

Create your playbook

Create a file named deploy_node.yml. This is where you call the installed role and define your configuration.

---
- name: Deploy Bitcoin Node
  hosts: bitcoin_nodes
  become: yes

  roles:
    - role: MaximeMRF.ansible-bitcoin-node

Define your Hosts

Create an inventory file hosts.yaml :

all:
  children:
    bitcoin_nodes:
      vars:
        ansible_user: "debian"
      hosts:
        node-btc-01:
          ansible_host: 192.168.0.10
          bitcoin_variant: "knots"
          bitcoin_version: "29.2.knots20251110"
          # optional, default is "x86_64"
          bitcoin_architecture: "x86_64"
          bitcoin_enable_indexes: false
          bitcoin_config:
            uacomment: "MyKnotsNode"
            server: 1
            listen: 1
            logips: 1
            bind: "0.0.0.0"
            rpcbind: "0.0.0.0"
            rpcallowip: "0.0.0.0/0"
            rpcuser: "bitcoinrpc"
            rpcpassword: "myverysecurepassword"
            prune: 4096
            dbcache: 4096
            maxmempool: 300
            zmqpubrawblock: "tcp://0.0.0.0:28332"
            zmqpubrawtx: "tcp://0.0.0.0:28333"
        node-btc-02:
          ansible_host: 192.168.0.20
          bitcoin_variant: "core"
          bitcoin_version: "29.2"
          bitcoin_enable_indexes: false
          bitcoin_config:
            uacomment: "MyCoreNode"
            server: 1
            listen: 1
            logips: 1
            bind: "0.0.0.0"
            rpcbind: "0.0.0.0"
            rpcallowip: "0.0.0.0/0"
            rpcuser: "bitcoinrpc"
            rpcpassword: "myverysecurepassword"
            prune: 4096
            dbcache: 4096
            maxmempool: 300
            zmqpubrawblock: "tcp://0.0.0.0:28332"
            zmqpubrawtx: "tcp://0.0.0.0:28333"

Understanding the Key Parameters:

• bitcoin_variant: This is the main switch. Set it to knots for the enhanced version (recommended) or core for the standard reference implementation. You can switch for variants and re-apply the playbook to take effect, same for the versions.
• bitcoin_version: Set the full version name to take effect. Don’t hesitate to update or downgrade the version and re-apply the playbook it will work fine.
• bitcoin_config: This dictionary maps directly to the bitcoin.conf file. Every key-value pair here is rendered into the final configuration file on the server.
• bitcoin_architecture: Defaults to x86_64 (standard Intel/AMD servers). You must change this to aarch64 if you are deploying on a Raspberry Pi or an ARM-based VPS. This ensures Ansible downloads the correct binary for your hardware.

Run it

Launch the deployment and watch Ansible handle the heavy lifting.

ansible-playbook -i hosts.yaml playbook.yaml

Conclusion & What’s Next

Automating the deployment of a Bitcoin node changes the game. It turns a tedious, error-prone manual process into a reliable, reproducible asset. Whether you are running a single node at home or managing a fleet of validators, ansible-bitcoin-node ensures your infrastructure is secure and standardized from day one.

The Next Step: Bitcoin Node as a Service As I hinted, this role is the foundation for something much bigger. I’m currently building a platform to make Bitcoin infrastructure even more accessible.

Are you interested in a “Bitcoin Node as a Service” solution? I’m looking for early beta testers and feedback. See the project here.

Open source lives on community feedback. I built this tool to be robust, but there is always room for improvement.

• Star the Repo: If you found this useful, please give the project a ⭐ on GitHub. It helps with visibility.
• Contribute: Found a bug? Want to add support for a Linux Distro? Don’t hesitate to open an Issue or submit a Pull Request.

Happy hosting, and don’t trust, verify.

As game development scales in complexity, Continuous Integration and Continuous Delivery (CI/CD) pipelines have become central to maintaining productivity. Conventional platforms such as Jenkins, Github CI, GitLab CI, or CircleCI offer flexibility and maturity for typical software delivery. However, their architectures are not optimized for the specific computational and data demands of Unreal Engine projects. Compiling millions of lines of C++ code and processing terabytes of binary assets requires an orchestration layer purpose-built for this environment.

Unreal Horde, developed by Epic Games, extends CI/CD beyond the general-purpose model. It introduces Granular Parallelization, Native Caching, and Deep Build Graph integration, three design principles that align with the technical and operational realities of Unreal Engine development. The following sections examine each of these principles in detail, highlighting how they address the performance, scalability, and maintenance challenges that traditional CI/CD tools face in large Unreal Engine environments.

Granular Parallelization for Build Speed

The primary technical challenge in Unreal Engine development is the sheer computational intensity of the build process, particularly the enormous C++ compilation phase. The rate at which the pipeline completes this process directly governs iteration speed and developer efficiency. Traditional CI tools eventually reach a hard performance ceiling, while Horde achieves a level of scalability that translates into exponential speed gains.

The Bottleneck of Job-Level Parallelization

Traditional CI/CD systems such as Jenkins are built around Job-Level Parallelization. They can effectively distribute independent, self-contained tasks (for example, Run Unit Tests or Build iOS Client) to separate Build Agents. However, when confronted with a single, monolithic process like compiling the engine and game code, a generic tool can only assign the job to one high-spec machine. That machine quickly becomes a bottleneck, leaving hundreds of available CPU cores idle and developers waiting for builds to finish.

Horde’s Breakthrough: Task-Level Distribution

Horde eliminates this inefficiency by operating at a Task-Level granularity, made possible through its native understanding of Unreal Engine’s Build Graph system.

• Build Graph Decomposition: When a build begins, Horde reads the Build Graph script and decomposes it into thousands of small, non-sequential, interdependent tasks.
• Distributed Execution: Instead of assigning an entire job to a single agent, Horde distributes these atomic tasks across a scalable pool of Horde Agents, leveraging the elastic capacity of AWS. Independent C++ modules, for example, can be compiled simultaneously across hundreds of machines.

Rather than one server performing the entire build, a hundred servers can each process a fraction of the workload in parallel. This granular parallelization fully utilizes available compute power, transforming multi-hour builds into streamlined processes. For large Unreal Engine projects, it becomes a decisive architectural advantage by removing wait times, shortening feedback loops, and sustaining the development team’s momentum.

Managing Large Assets Efficiently (Caching and Monorepos)

A major challenge for general-purpose CI/CD systems in game development is managing data volume. Modern Unreal Engine projects operate with multi-terabyte monorepos containing extensive binary assets, textures, and cache files. In such environments, maintaining efficient data synchronization and artifact management is essential to sustain fast build times.

The Challenge of Redundant Processing

Traditional CI/CD systems such as Jenkins often process the same data multiple times across different build agents. Each agent independently downloads, compiles, and processes the same files, lacking a unified caching layer. This redundancy increases network traffic, storage I/O, and compute usage (particularly problematic when handling gigabytes of cooked game assets).

Horde’s Integrated Approach: Content Addressable Storage

Horde addresses this limitation through its native Content Addressable Storage (CAS), also known as Horde Storage or Zen, a system designed specifically for Unreal Engine’s scale and data characteristics.

• Centralized Artifact Repository: Each compiled code object, cooked asset, or cache file is assigned a unique cryptographic hash and stored in the central CAS repository.
• Elimination of Redundant Work: Once an artifact has been built by any Horde Agent, it becomes instantly available to all others. Subsequent builds simply fetch the cached version instead of reprocessing it.

This model minimizes unnecessary network and compute activity, ensuring consistent performance even as project size grows. When deployed on AWS, CAS benefits from services like Amazon S3 for durable, cost-effective storage, providing a native caching framework that general-purpose CI tools would require significant custom engineering to approximate.

Maintenance Overhead

Replicating Horde’s caching efficiency in other CI/CD platforms typically requires a complex stack of additional components: network file shares (such as NFS), custom caching proxy servers, or paid third-party plugins. These integrations add latency, introduce potential points of failure, and require ongoing maintenance.

Over time, the cumulative complexity and operational cost make it difficult for a general-purpose CI tool to manage the scale and data patterns of modern Unreal Engine projects effectively.

Native Integration for Reduced Maintenance and Greater Control

One of Horde’s most significant advantages is the reduction in ongoing maintenance effort required to support Unreal Engine development. By aligning natively with Unreal’s ecosystem, Horde simplifies operations and allows engineering teams to focus on building features rather than maintaining infrastructure.

The Build Graph Translation Overhead

CI/CD platforms such as Jenkins must interpret Unreal’s Build Graph scripts, which are the XML-based recipes that define how projects are compiled, packaged, and deployed. To do so, engineers often create intermediary wrapper scripts that translate between Jenkins’ general-purpose command model and Unreal Build Tool (UBT) operations. This middle layer is fragile, requires frequent updates when Unreal versions change, and introduces additional maintenance overhead.

Horde’s Native Execution Advantage

Horde eliminates the need for this translation entirely. Developed by Epic Games, it interprets and executes Build Graph scripts directly, understanding all dependencies, relationships, and execution rules without intermediary scripting.

• Direct Execution: Horde runs Build Graph instructions natively, ensuring full compatibility and minimal setup.
• Zero Maintenance Debt: Its tight integration with Unreal ensures stability across engine updates, removing the need for continuous adjustments.

This native alignment shifts engineering effort away from maintaining CI/CD plumbing toward optimizing build performance and scalability. The result is a cleaner, more predictable pipeline that supports faster iteration and consistent delivery across projects.

Summary Comparison Table

Traditional CI platforms remain highly effective for standard software delivery pipelines. However, the computational and data intensity of Unreal Engine development benefits from a system architected around the engine itself. Horde’s native integration, distributed architecture, and caching capabilities deliver measurable gains in build performance, maintainability, and scalability, particularly when deployed on AWS.

Conclusion

Selecting the right CI/CD platform depends on the nature of the workload. Jenkins offers proven versatility for multi-language, multi-stack projects. For Unreal Engine, however, where build complexity, asset volume, and iteration speed define competitiveness, Unreal Horde provides a specialized, scalable, and maintainable foundation.

When combined with the elasticity of AWS infrastructure, Horde enables game studios to maintain continuous delivery performance at scale, supporting larger teams, faster feedback loops, and smoother release cycles for ambitious Unreal Engine titles.

To explore the financial and operational advantages of running Horde on AWS, refer to the companion article Modernizing Unreal Horde CI/CD: Moving from On-Premise Infrastructure to AWS.

About TrackIt

TrackIt is an international AWS cloud consulting, systems integration, and software development firm headquartered in Marina del Rey, CA.

We have built our reputation on helping media companies architect and implement cost-effective, reliable, and scalable Media & Entertainment workflows in the cloud. These include streaming and on-demand video solutions, media asset management, and archiving, incorporating the latest AI technology to build bespoke media solutions tailored to customer requirements.

Cloud-native software development is at the foundation of what we do. We specialize in Application Modernization, Containerization, Infrastructure as Code and event-driven serverless architectures by leveraging the latest AWS services. Along with our Managed Services offerings which provide 24/7 cloud infrastructure maintenance and support, we are able to provide complete solutions for the media industry.

Unreal Horde vs. Traditional CI/CD: Optimizing for Unreal Engine Development was originally published in TrackIt on Medium, where people are continuing the conversation by highlighting and responding to this story.

Unreal Horde is a proprietary CI/CD system developed by Epic Games to meet the distinctive demands of Unreal Engine development. Designed as a high-performance orchestrator, it intelligently manages the most intensive build and asset-processing workloads compiling millions of lines of C++ code and transforming terabytes of content across distributed machines.

This orchestration drastically shortens build times, enabling faster iteration, testing, and innovation. For studios building ambitious Unreal Engine titles, Horde functions as the backbone of a modern production pipeline, turning what were once slow, fragmented processes into a unified, high-speed development engine.

Challenges of On-Premise Horde CI/CD

The Demands of Unreal Engine Development

Game development operates under extreme technical and creative pressure, where build speed directly determines how fast teams can innovate. Unreal Engine projects, in particular, push hardware to its limits compiling millions of lines of C++ code, processing massive textures, and cooking binary assets into optimized formats. A reliable Continuous Integration and Delivery system such as Unreal Horde becomes essential in managing these heavy workloads, ensuring stable, repeatable, and efficient development cycles.

Challenges of On-Premise Horde CI/CD

Running Horde on-premise often becomes a hidden constraint that slows development and consumes resources inefficiently. The challenge stems from highly uneven compute demand intensive build spikes followed by long periods of inactivity. To handle peak load, studios must purchase and maintain powerful servers that remain underutilized for much of the day, often as high as 80%. This static, hardware-bound model locks significant capital into infrastructure that delivers limited day-to-day value and constrains both scalability and financial flexibility.

Migrating Horde CI/CD to AWS

Transitioning Horde CI/CD to AWS represents a decisive move toward efficiency and scalability. The cloud addresses the core limitation of on-premise infrastructure idle capacity by turning compute into an on-demand resource.

With AWS, build environments scale dynamically to match actual workload requirements. When a large build begins, the necessary compute power can be provisioned within minutes; once complete, resources are released automatically. This model eliminates the waste of idle servers, removes capacity constraints, and converts what was once static infrastructure into a flexible, continuously optimized service.

Optimizing CI/CD Costs with Elastic Infrastructure

Migrating Horde CI/CD to AWS transforms cost management by aligning infrastructure spend directly with usage. Compute capacity scales in real time to match workload requirements, converting what was once fixed infrastructure into a flexible, consumption-based model. The result is a system that adapts to production needs while eliminating the financial drag of idle servers.

Elasticity for Build-Driven Workloads

Elasticity remains the defining strength of AWS, particularly suited to the burst-heavy nature of game development. Horde consumes compute resources only when builds are active. During large-scale or overnight builds, the system automatically scales to launch hundreds of virtual Horde Agents across Amazon EC2. When the build completes, the instances terminate and billing stops. This dynamic allocation ensures that costs reflect actual activity rather than fixed capacity planning.

The Spot Strategy

Cost efficiency is further enhanced through Amazon EC2 Spot Instances. Because Horde Agents are stateless and easily replaceable, workloads can be redistributed if an instance is interrupted making them ideal for Spot usage. By tapping into AWS’s surplus compute capacity, up to 90% of build agents can run on Spot Instances, delivering substantial savings without compromising reliability or throughput. This approach significantly reduces the per-build cost while preserving the performance and scale expected from a production-grade CI/CD pipeline.

Staying Current Without Reinvestment

On-premise infrastructure inevitably ages, tying studios to outdated hardware until the next budget cycle. AWS removes this limitation by providing immediate access to the latest CPU and GPU generations as they become available. This continuous modernization keeps Horde CI/CD running on optimal infrastructure without reinvestment, maintenance overhead, or the depreciation associated with static assets.

Accelerating Builds with On-Demand Scalability

Beyond cost efficiency, the most tangible advantage of running Horde CI/CD on AWS lies in the speed and responsiveness of the build process. In game development, build velocity directly determines how quickly teams can test, iterate, and innovate.

Eliminating Wait Times with Instant Compute

The most significant obstacle to developer productivity is waiting whether for a build to complete, a test to run, or an iteration to deploy. When Horde operates on AWS, this friction effectively disappears.

The system can mobilize hundreds of CPU cores within minutes, distributing compilation and cooking workloads across scalable compute resources. During high-activity periods, such as milestone check-ins, this capability removes bottlenecks entirely. Developers receive faster feedback, spend less time context-switching, and can validate smaller, more frequent changes without delay.

Scaling Seamlessly with Project Growth

As game projects evolve, the demands on build infrastructure expand new features, additional platforms, larger teams. On-premise environments struggle to keep pace, requiring months of budgeting, procurement, and configuration to add capacity. AWS eliminates these constraints by providing near-unlimited scalability.

Build Agents can be deployed horizontally and instantly, ensuring that compute capacity grows in step with the project. Whether scaling for multiple console launches or a sudden increase in developer headcount, the pipeline remains responsive and future-ready, ensuring that infrastructure never becomes the limiting factor in a project’s timeline.

Operational Agility and Resilience

Beyond cost optimization and speed, running Horde CI/CD on AWS transforms operational management. The platform introduces built-in resilience, automation, and observability that shift focus away from infrastructure maintenance toward continuous delivery and innovation.

Infrastructure as Code for Repeatability and Control

Migrating to AWS enables full Infrastructure as Code (IaC) implementation through tools such as Terraform or AWS CloudFormation a strategic leap from traditional, manually managed setups.

• Versioned Infrastructure: Every component of the Horde environment servers, storage, and networking is defined as code. These definitions are version-controlled, auditable, and easily replicated across environments.
• Rapid Disaster Recovery: In the event of a failure, the entire Horde pipeline can be redeployed in a different AWS Region or Availability Zone within minutes, ensuring continuity that physical infrastructure cannot match.
• Environmental Consistency: IaC guarantees that development, testing, and production environments remain identical, eliminating discrepancies and minimizing integration issues.

Built-in Resilience and Durable Storage

Physical data centers inherently carry risks hardware failures, network interruptions, or localized outages. AWS mitigates these risks through architecture designed for fault tolerance and data protection.

• High Availability: AWS services operate across multiple Availability Zones (AZs), each a distinct, isolated facility. If one zone experiences disruption, Horde operations continue unaffected in another.
• Data Durability: Build artifacts, logs, and assets stored in Amazon S3 benefit from industry-leading durability guarantees, ensuring essential data remains secure and recoverable. This level of reliability would be costly and complex to replicate on-premise.

Refocusing Engineering Effort on Core Development

Delegating infrastructure management to AWS allows engineering teams to focus exclusively on the areas that add creative and technical value. Responsibilities such as power management, hardware replacement, and network patching are handled by AWS, freeing teams to refine the Horde pipeline, optimize build performance, and integrate custom development tools. The outcome is a leaner, more efficient operation that accelerates delivery and innovation without the operational weight of maintaining physical servers.

Conclusion

Adopting AWS for Horde CI/CD establishes a foundation that is elastic, cost-optimized, and resilient designed to support studios pursuing ambitious Unreal Engine projects at any scale.

The cloud model directly addresses the three key constraints that limit studio performance today:

• Financial Rigidity: Transitioning from fixed capital investments to a flexible, usage-based model unlocks financial agility. Leveraging services such as Amazon EC2 Spot Instances delivers significant cost efficiency and virtually unlimited scalability without long-term capital lock-in.
• Innovation Bottlenecks: With on-demand compute capacity, build queues and developer idle time are eliminated. Teams can iterate continuously, maintaining creative momentum without being constrained by infrastructure capacity.
• Operational Risk: Through Infrastructure as Code and resilient services such as Amazon S3 and multi-AZ architectures, the environment achieves high availability, rapid recoverability, and long-term data durability.

This transformation releases engineering teams from the burden of managing hardware, power, and cooling, allowing them to focus entirely on improving build efficiency and advancing the game itself.

About TrackIt

TrackIt is an international AWS cloud consulting, systems integration, and software development firm headquartered in Marina del Rey, CA.

We have built our reputation on helping media companies architect and implement cost-effective, reliable, and scalable Media & Entertainment workflows in the cloud. These include streaming and on-demand video solutions, media asset management, and archiving, incorporating the latest AI technology to build bespoke media solutions tailored to customer requirements.

Cloud-native software development is at the foundation of what we do. We specialize in Application Modernization, Containerization, Infrastructure as Code and event-driven serverless architectures by leveraging the latest AWS services. Along with our Managed Services offerings which provide 24/7 cloud infrastructure maintenance and support, we are able to provide complete solutions for the media industry.

Modernizing Unreal Horde CI/CD: Moving from On-Premise Infrastructure to AWS was originally published in TrackIt on Medium, where people are continuing the conversation by highlighting and responding to this story.

The adoption of cloud-native technologies has transformed the way media workflows are designed and deployed. As organizations face growing volumes of digital assets and increasingly complex distribution needs, scalable and resilient infrastructure becomes essential. Kubernetes has emerged as a popular choice for orchestrating such workloads, but managing stateful services within clusters can introduce operational challenges.

Below is a detailed overview of deploying a Media Asset Management (MAM) solution — Phraseanet — in a production environment using Amazon EKS (Elastic Kubernetes Service). While the application is containerized and orchestrated with Kubernetes, critical stateful components such as the database, Redis, Elasticsearch, and RabbitMQ are offloaded to AWS managed services. This hybrid approach reduces operational complexity and enhances reliability, allowing the MAM to meet production-grade performance and scalability requirements.

Note: While this article focuses primarily on Phraseanet, the concepts discussed — such as auto-scaling and managed services — are broadly applicable and can be adapted for other MAM systems.

Why Use Amazon Managed Services Instead of Pods?

Leveraging AWS managed services enhances performance, reliability, and scalability while minimizing the operational burden associated with managing these components within a Kubernetes cluster. Services such as Amazon RDS and Amazon ElastiCache for Redis deliver optimized performance and reliability through continuous monitoring and maintenance by AWS, offering features like high availability and automatic failover.

Built-in scalability allows seamless capacity adjustments based on demand, a critical feature for managing unpredictable traffic patterns in media asset workflows. These services also integrate efficiently with the broader AWS ecosystem, contributing to a cohesive and secure cloud infrastructure. Compliance standards and robust security features are supported by default.

Phraseanet’s Infrastructure Recap

To determine which services are suitable for replacement with AWS managed solutions, below is a summary of the components used in the current Phraseanet infrastructure:

• Phraseanet Gateway: Serves as the main access point, routing traffic and handling frontend requests.
• Database (MySQL): Stores metadata and associated information for media assets, supporting search, retrieval, and scalability.
• Worker Service: Manages background tasks such as media processing, transcoding, and workflow automation.
• Elasticsearch: Indexes and searches large media libraries, improving metadata retrieval and content discovery.
• FPM (FastCGI Process Manager): Handles PHP processes for frontend and backend interfaces, ensuring responsive user interactions.
• RabbitMQ: Facilitates messaging between services, enabling reliable communication for components like the worker service.
• Redis: Provides caching and session management to reduce database load and improve performance.
• Phraseanet Setup: Initializes the environment by configuring databases and essential system settings.

These services are currently deployed as Kubernetes pods. The objective is to migrate selected components to AWS managed services.

Phraseanet Services Transitioning to AWS Managed Solutions

The key components suitable for AWS-managed replacements include Redis, RabbitMQ, Elasticsearch, and the MySQL database. These are mapped to the following AWS services:

• Amazon RDS: A managed database offering high availability, automated backups, and seamless scaling.
• Amazon ElastiCache: A managed Redis or Valkey service optimized for caching and real-time use cases, with built-in security and failover.
• Amazon OpenSearch: A managed Elasticsearch service providing scalable search and analytics capabilities.
• Amazon MQ: A managed RabbitMQ service that simplifies messaging and integrates well with other AWS services.

Valkey was selected over Redis for ElastiCache. As an open-source fork of Redis, Valkey offers enhanced multi-threading performance and improved memory efficiency, while maintaining full compatibility with Redis APIs. It is also supported by AWS with pricing up to 33% lower, making it a more cost-effective option.

Monitoring & Alerts Management

Amazon CloudWatch is used for monitoring and alert management. As a native AWS service, CloudWatch provides seamless integration, real-time monitoring, and automated alerts. It supports performance tracking, anomaly detection, and rapid incident response to ensure system reliability and operational efficiency.

Monitoring Managed Services

For Amazon RDS, an email alert notifies administrators when CPU usage exceeds 80% over a specified time period. This proactive measure helps maintain performance and prevent potential disruptions. Similar alerts can be configured for other managed services, monitoring critical metrics such as CPU utilization, memory usage, and disk space.

Monitoring Pods and the EKS Cluster

Amazon CloudWatch Container Insights is used to monitor the EKS cluster and associated pods. This service provides visibility into the performance and health of the Kubernetes environment by collecting and displaying critical metrics. It enables tracking of resource utilization, including CPU and memory usage, across nodes and individual pods.

Alarms can be configured for a wide range of components — nodes, namespaces, pods, and more — triggering notifications in the event of anomalies or threshold breaches.

Automating the Infrastructure with Auto Mode

EKS Auto Mode is used to deploy the MAM system due to its numerous benefits over the traditional EKS setup. Auto Mode simplifies cluster management by automating provisioning, scaling, and maintenance tasks for compute, storage, and networking infrastructure. This reduces the need for manual configuration and allows teams to concentrate on deploying and managing applications.

How Does EKS Auto Mode Scale Automatically?

EKS Auto Mode dynamically adjusts the number of EC2 instances based on workload demands. It integrates with Karpenter to provision and scale nodes efficiently, ensuring that clusters are resourced appropriately, reducing idle capacity, and improving responsiveness under load.

Difference from Managed Node Groups

Managed Node Groups in EKS offer automated management and scaling for EC2 worker nodes, but still require user-defined instance types, scaling policies, and update configurations. While this allows for greater customization, it also increases operational complexity compared to Auto Mode, which is fully automated.

Configuring Auto Mode

The use of Auto Mode is specified in the Terraform configuration, along with a general-purpose node pool. When deploying YAML files via HELM, EKS automatically selects the nodes best suited to the workload. This setup, managed through Terraform, offers a straightforward deployment experience.Additional details on EKS Auto Mode can be found in the full article available here.

Conclusion

Deploying a Media Asset Management system in a production environment becomes significantly more scalable and efficient with EKS Auto Mode and AWS managed services. Auto Mode reduces infrastructure management complexity, while managed services enhance performance, security, and reliability.

Automated scaling, real-time monitoring, and proactive alert systems contribute to a robust and responsive architecture, well-suited to dynamic production environments and the evolving demands of media asset workflows.

About TrackIt

TrackIt is an international AWS cloud consulting, systems integration, and software development firm headquartered in Marina del Rey, CA.

We have built our reputation on helping media companies architect and implement cost-effective, reliable, and scalable Media & Entertainment workflows in the cloud. These include streaming and on-demand video solutions, media asset management, and archiving, incorporating the latest AI technology to build bespoke media solutions tailored to customer requirements.

Cloud-native software development is at the foundation of what we do. We specialize in Application Modernization, Containerization, Infrastructure as Code and event-driven serverless architectures by leveraging the latest AWS services. Along with our Managed Services offerings which provide 24/7 cloud infrastructure maintenance and support, we are able to provide complete solutions for the media industry.

Scaling Media Asset Management with Kubernetes: Deploying a MAM for Production on Amazon EKS was originally published in TrackIt on Medium, where people are continuing the conversation by highlighting and responding to this story.

At AWS re:Invent 2024, Amazon introduced Auto Mode for Elastic Kubernetes Service (EKS), a new feature that simplifies Kubernetes cluster management. This deep dive into EKS Auto Mode explores its capabilities and benefits through a hands-on demonstration. Using Terraform, the guide below walks through each step of deploying an EKS Auto Mode cluster, showcasing how this new feature streamlines operations and enhances the cloud-native experience.

Understanding EKS Auto Mode: Features and Benefits

EKS Auto Mode simplifies running an EKS cluster by handling complex tasks like managing the Kubernetes control plane, maintaining controllers for load balancing and auto-scaling, and configuring IAM roles and policies. It also streamlines IAM setup for CSI (Container Storage Interface) drivers, enabling persistent storage while reducing operational overhead.

Key Features:

• Streamlined Management: Provides production-ready clusters with minimal operational overhead.
• Application Availability: Dynamically scales nodes based on application demands, reducing manual capacity planning (powered by Karpenter for autoscaling).
• Efficiency: Optimizes compute costs by terminating unused instances and consolidating workloads.
• Security: Uses immutable AMIs with locked-down software and regular node cycling for enhanced security.
• Automated Upgrades: Keeps clusters and components up-to-date with the latest patches.
• Managed Components: Includes built-in support for essential Kubernetes and AWS features.
• Customizable NodePools and NodeClasses: Allows for tailored configurations to meet specific workload requirements.

How EKS Auto Mode Updates the Cluster Automatically

EKS Auto Mode simplifies Kubernetes version upgrades by handling control plane updates and node replacements while maintaining workload availability through pod disruption budgets. Components such as the Amazon EBS CSI driver are managed as integrated services, eliminating the need for manual installation or updates.

This approach differs from standard EKS clusters, where components like the EBS CSI driver are typically installed and managed as add-ons. In EKS Auto Mode, AWS oversees the lifecycle of these components, ensuring they remain up to date and properly configured.

For example, when deploying an application with Auto Mode, the StorageClass references the provisioner ebs.csi.eks.amazonaws.com, which AWS manages as part of the service. In a standard EKS cluster, the provisioner ebs.csi.aws.com is used instead, requiring manual installation and management of the EBS CSI driver.

Automated Updates:

• Nodes are replaced with the new Kubernetes version.
• Components like CoreDNS, KubeProxy, AWS Load Balancer Controller, Karpenter, and AWS EBS CSI Driver are automatically updated.

User Responsibilities:

• Updating apps and workloads.
• Managing self-deployed add-ons and controllers.
• Updating Amazon EKS Add-ons.

Tutorial: EKS Auto Mode

Create the cluster

A closer look at cluster creation provides a better understanding of the EKS Auto Mode concept. This walkthrough covers deployment using Terraform.

Note: The provided configuration is simplified for demonstration purposes and is not intended for production use.

Configuration

The Terraform code used in this tutorial is available in this repository: https://github.com/MaximeMRF/eks-auto-mode-tutorial.

Before getting started, make sure the AWS CLI is properly configured and that both kubectl and Terraform are installed.

Begin by reviewing the terraform.tfvars file to ensure the variables align with project requirements. For instance, availability zones (AZs) may need to be adjusted from Europe to the US.

Next, open the eks.tf file. In the cluster_compute_config object, the enabled property is set to true, indicating that Auto Mode is activated for compute, network, and storage.

This configuration also creates a node pool named “general-purpose”. EKS automatically selects the nodes and instance sizes while scaling them using Karpenter, removing the need for manual setup.

Deployment of the cluster

To set up the project and install dependencies, run the following command:

terraform init

Next, generate the execution plan:

terraform plan

Finally, deploy the configuration:

terraform apply -auto-approve

Once the cluster is ready, update the kubectl configuration to access it. Adjust the cluster name and region based on the defined variables:

aws eks update-kubeconfig --name eks-auto-mode-cluster --region eu-north-1

Understanding the Auto-scaling

The cluster is now running. Listing the nodes with kubectl will show no active nodes yet, as no pods have been deployed.

To see how nodes are created when pods are scheduled, apply the deploy.yml file from the repository. This will launch a BusyBox container that runs indefinitely.

kubectl apply -f deploy.yml

Listing the nodes again will now show that EKS has created a node to run the container.

kubectl get nodes

As more pods are added, EKS will either launch new nodes or assign them to existing ones with available resources, all without the need for manual scaling configuration.

How EKS Auto Mode Manages Storage

Without Auto Mode, IAM permissions must be configured manually by retrieving and attaching the AmazonEBSCSIDriverPolicy to the cluster’s node role, allowing the EBS CSI driver to manage volumes. Auto Mode includes a built-in CSI driver with preconfigured permissions.

To test volume management with Auto Mode, use the preconfigured YAML file available in the kubernetes-objects folder of the repository.

How EKS Auto Mode Handles Load Balancing

EKS Auto Mode simplifies load balancing by removing the need for manual IAM policy creation and attachment, as these are preconfigured. It includes a built-in controller for provisioning load balancers with the necessary permissions.

Without Auto Mode, IAM policies must be manually created and attached to enable the AWS Load Balancer Controller (LBC). Auto Mode also requires defining an IngressClass and IngressClassParams, which are optional when using LBC. The IngressClass specifies the controller, with Auto Mode using eks.amazonaws.com/alb. This setup streamlines ALB management and reduces manual configuration.

Conclusion

EKS Auto Mode simplifies Kubernetes cluster management by automating tasks such as resource scaling, system patching, and security enforcement. This allows teams to focus on application development rather than infrastructure maintenance. It provides a production-ready environment that is efficient, secure, and continuously updated without the operational complexity.

About TrackIt

TrackIt is an international AWS cloud consulting, systems integration, and software development firm headquartered in Marina del Rey, CA.

We have built our reputation on helping media companies architect and implement cost-effective, reliable, and scalable Media & Entertainment workflows in the cloud. These include streaming and on-demand video solutions, media asset management, and archiving, incorporating the latest AI technology to build bespoke media solutions tailored to customer requirements.

Cloud-native software development is at the foundation of what we do. We specialize in Application Modernization, Containerization, Infrastructure as Code and event-driven serverless architectures by leveraging the latest AWS services. Along with our Managed Services offerings which provide 24/7 cloud infrastructure maintenance and support, we are able to provide complete solutions for the media industry.

Simplifying Kubernetes Management with EKS Auto Mode was originally published in TrackIt on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this article I will show you how to access financial datas in real time in your Javascript (Node.js) application thanks to the power of the Pyth Network, a oracle blockchain.

The Pyth Network

Pyth Network is a decentralized oracle solution designed to deliver high-fidelity, real-time financial market data directly to blockchain applications. By bridging the gap between traditional finance and decentralized ecosystems, Pyth enables smart contracts to access accurate and timely information essential for various decentralized finance (DeFi) applications.

Unlike traditional oracles that rely on aggregated third-party data, Pyth sources its information directly from first-party providers (like Revolut), including leading exchanges, trading firms, and market makers. This direct sourcing ensures the data’s authenticity, low latency, and high precision, making it particularly valuable for applications requiring real-time market insights.

The Hermes server

Hermes is an open-source service that continuously monitors Pythnet and the Wormhole Network for Pyth price updates, making them accessible through a user-friendly web API.

Hermes allows users to easily retrieve the latest price updates via a REST API or subscribe to SSE (Server Send Events) for real-time streaming data.

This infrastructure simplifies the integration of Pyth’s financial data into various on-chain and off-chain applications, providing developers with efficient access to real-time market information.

In our case we will not use directly the web API from scratch, Pyth makes available to us a Typescript client package to interact with the Hermes server.

Installing the Hermes client

Before installing the client make you sure that you have Node.js and npm (or another package manager) installed on your computer by doing this command:

node -v && npm -v

It should output the version of Node and the version of your packet manager.

Create a new folder a then write the command npm init -y to init a new Node.js project with all default parameters.

Then install the Hermes client with npm i @pythnetwork/hermes-client and create a new Javascript file called for example index.js .

Also, add "type": "module" at your package.json object because we will use imports instead of required.

In a first time I will show you how to get real time datas from Hermes and in a second time how to setup a simple alert program if a price go up or down.

You can find all the code of this tutorial here.

Retrieve datas from Hermes

The Pyth blockchain expose more than 600 price feeds and it will grow more in the time, here is the list of prices currently supported.

For this example we will retrieve only 4 different prices (BTC/USD, ETH/USD, AVAX/USD, SOL/USD) but feel free to adapt this program to get more prices.

So now, let’s code ! Open the index.js and copy this following program:

import { HermesClient } from '@pythnetwork/hermes-client'

// Initiate a connexion with the Hermes server hosted by the pyth fondation
const connection = new HermesClient('https://hermes.pyth.network')

// here is the list of prices: https://www.pyth.network/price-feeds
// This object contains the key that is the index and it correspondance in ids
const PriceIds = {
  BTC_USD: 'e62df6c8b4a85fe1a67db44dc12de5db330f7ac66b72dc658afedf0f4a415b43',
  ETH_USD: 'ff61491a931112ddf1bd8147cd1b641375f79f5825126d665480874634fd0ace',
  AVAX_USD: '93da3352f9f1d105fdfe4971cfa80e9dd777bfc5d0f683ebb6e1294b92137bb7',
  SOL_USD: 'ef0d8b6fda2ceba41da15d4095d1da392a0d2f8ed0c6c7bc0f4cfac8c280b56d',
}

// With the object PriceNames you can get the price index in a humain readable format
const PriceNames = {
  [PriceIds.BTC_USD]: 'BTC/USD',
  [PriceIds.ETH_USD]: 'ETH/USD',
  [PriceIds.AVAX_USD]: 'AVAX/USD',
  [PriceIds.SOL_USD]: 'SOL/USD',
}

// We have to transform our PriceIds object to a ids array
const priceIds = Object.values(PriceIds)

// Here we create our SSE connexion with the server
// thanks to EventSource: https://developer.mozilla.org/en-US/docs/Web/API/EventSource
const eventSource = await connection.getPriceUpdatesStream(priceIds, { parsed: true })

// We will get datas in json format
eventSource.onmessage = async (event) => {
  const data = JSON.parse(event.data)
  for (const item of data.parsed) {
    const priceName = PriceNames[item.id]
    const rawPrice = Number(item.price.price)
    const exponent = item.price.expo
    const priceParsed = rawPrice * Math.pow(10, exponent)
    console.log(`Price update for ${priceName}: ${priceParsed}`)
  }
}

eventSource.onerror = (error) => {
  console.error('Error receiving updates:', error)
  eventSource.close()
}

Then run the code using node index.js it will display something like that indefinitely:

Price update for BTC/USD: 96822.408314
Price update for ETH/USD: 3226.14867123
Price update for AVAX/USD: 36.50984607
Price update for SOL/USD: 187.21274583000002
Price update for BTC/USD: 96820.75503003
Price update for ETH/USD: 3226.10911453
Price update for AVAX/USD: 36.50984607
Price update for SOL/USD: 187.20670585
Price update for BTC/USD: 96822.45579042
Price update for ETH/USD: 3226.22825627
Price update for AVAX/USD: 36.51020737

That’s nice ! We now have a program that retrieve price data in real time, in the second part we will setup a simple alerting system.

Setup a alerting system

If a price is going up or going down we want to receive a notification to maybe re-invest or sell our position.

Create a array of alert object in your code and add the loop below:

import { HermesClient } from "@pythnetwork/hermes-client"

const connection = new HermesClient("https://hermes.pyth.network")

const PriceIds = {
  BTC_USD: "e62df6c8b4a85fe1a67db44dc12de5db330f7ac66b72dc658afedf0f4a415b43",
  ETH_USD: "ff61491a931112ddf1bd8147cd1b641375f79f5825126d665480874634fd0ace",
  AVAX_USD: "93da3352f9f1d105fdfe4971cfa80e9dd777bfc5d0f683ebb6e1294b92137bb7",
  SOL_USD: "ef0d8b6fda2ceba41da15d4095d1da392a0d2f8ed0c6c7bc0f4cfac8c280b56d",
}

const PriceNames = {
  [PriceIds.BTC_USD]: "BTC/USD",
  [PriceIds.ETH_USD]: "ETH/USD",
  [PriceIds.AVAX_USD]: "AVAX/USD",
  [PriceIds.SOL_USD]: "SOL/USD",
}

// Add a object for each alert you want to set
// you can adapt it to come from a configuration file or a database
const alerts = [
  { id: PriceIds.BTC_USD, direction: "up", targetPrice: 96000 },
  { id: PriceIds.ETH_USD, direction: "below", targetPrice: 3100 },
  { id: PriceIds.AVAX_USD, direction: "up", targetPrice: 35 },
  { id: PriceIds.SOL_USD, direction: "below", targetPrice: 200 },
]

const priceIds = Object.values(PriceIds);

const eventSource = await connection.getPriceUpdatesStream(priceIds, {
  parsed: true,
})

eventSource.onmessage = async (event) => {
  const data = JSON.parse(event.data)
  for (const item of data.parsed) {
    const priceName = PriceNames[item.id]
    const rawPrice = Number(item.price.price)
    const exponent = item.price.expo
    const priceParsed = rawPrice * Math.pow(10, exponent)
    console.log(`Price update for ${priceName}: ${priceParsed}`)
    // filter alerts for the current price
    const avalaibleAlerts = alerts.filter((alert) => alert.id === item.id)
    // check if the price triggers any alert
    for (const alert of avalaibleAlerts) {
      if (alert.direction === "up" && priceParsed >= alert.targetPrice) {
        console.log(`Alert for ${priceName} triggered: ${priceParsed} >= ${alert.targetPrice}`)
      }
      if (alert.direction === "below" && priceParsed <= alert.targetPrice) {
        console.log(`Alert for ${priceName} triggered: ${priceParsed} <= ${alert.targetPrice}`)
      }
    }
  }
}

eventSource.onerror = (error) => {
  console.error("Error receiving updates:", error)
  eventSource.close()
}

Now, re-run your script and maybe depending of the current market price you will see appeared alerts like that:

Alert for AVAX/USD triggered: 36.41689888 >= 35
Price update for SOL/USD: 186.55917611
Alert for SOL/USD triggered: 186.55917611 <= 200
Price update for BTC/USD: 96651
Alert for BTC/USD triggered: 96651 >= 96000
Price update for ETH/USD: 3215.6112736
Price update for AVAX/USD: 36.417797220000004
Alert for AVAX/USD triggered: 36.417797220000004 >= 35
Price update for SOL/USD: 186.56284226
Alert for SOL/USD triggered: 186.56284226 <= 200

Our alerts are simple console.log at the moment but you can easily setup alerts by sms, webhook, discord webhook and maybe emails.

Conclusion

You can now retrieve real time prices using the Pyth blockchain and setup alerts, don’t hesitate to learn more about Pyth Network and improve the tutorial code if you want to do something serious because I only scratch the surface. Maybe next step is to create a web app to allow users to set up their own alerts ?

Here you can find all the code used in this tutorial, don’t hesitate to give me a star on github !

Through this article I will show you how to implement a authentication system via jwt token in a Adonisjs v6 application. But before telling you how to do that I would like you to ask yourself the question if the jwt token is the best way in your case to authenticate a user of your application.

Differents kind of authentication

Before beginning I invite you to read this part of the documentation to know if it’s really the authentication type you want for your application.

If you prefer to use the authentication by OAT token you can read this article.

🇫🇷 Tu parles français ? voici le lien de l’article en français

Installation

Before beginning make you sure that you use node version 22 or at least node version 20.6 .

node -v
# v22.x.x

There is 3 differents official starter kit provided by the Adonisjs team to don’t start the code from scratch and save some times. For this application we will use the api starter kit with the authentication guard session because the jwt package that we will use is based in this guard.

npm init adonisjs@latest -- -K=api --auth-guard=session

Choose sqlite as the database for this tutorial it will really make the task easier.

After choose the differents options asked by the CLI and going to the new folder created we can go to the next step.

Then we can install the jwt package that will allow us to have a jwt guard for our application.

npm i @maximemrf/adonisjs-jwt

Don’t hesitate to give a ⭐️ to my jwt package, it really motivate me to contribute more to opensource and write some other articles ❤️

Configuration

To begin we will configurate our guard located at /config/auth.ts , replace the configuration by this one:

import { defineConfig } from '@adonisjs/auth'
import { sessionUserProvider } from '@adonisjs/auth/session'
import { jwtGuard } from '@maximemrf/adonisjs-jwt/jwt_config'
import type { InferAuthEvents, Authenticators } from '@adonisjs/auth/types'

const authConfig = defineConfig({
  default: 'jwt',
  guards: {
    jwt: jwtGuard({
      // tokenExpiresIn is the duration of the validity of the token, it's a optional value
      tokenExpiresIn: '1h',
      // if you want to use cookies for the authentication instead of the bearer token (optional)
      useCookies: true,
      provider: sessionUserProvider({
        model: () => import('#models/user'),
      }),
    }),
  },
})

If you would like to stock and send the jwt token via cookies let useCookies at true . Otherwise remove the option, the token will be passed by the authorization header (bearer). As well as for the expiration date of the tokens, it’s also a optional property.

Controller

Here is the login and register methods:

export default class AuthController {
  // login if we use authorization bearer
  async login({ request, response, auth }: HttpContext) {
    const { email, password } = await request.validateUsing(loginValidator)

    const user = await User.verifyCredentials(email, password)
    const token = await auth.use('jwt').generate(user)

    return response.ok({
      token: token,
      ...user.serialize(),
    })
  }
  // login if we use the cookies (useCookies: true)
  async login({ request, response, auth }: HttpContext) {
    const { email, password } = await request.validateUsing(loginValidator)

    const user = await User.verifyCredentials(email, password)
    await auth.use('jwt').generate(user)

    return response.ok({
      ...user.serialize(),
    })
  }
  async register({ request, response }: HttpContext) {
    const payload = await request.validateUsing(registerValidator)

    const user = await User.create(payload)

    return response.created(user)
  }
}

For more explanation about the validators and controllers used in this code I invite you to read this article.

Conclusion

Now you know how to create a JWT authentication system with Adonisjs. Feel free to read the official documentation to learn more.

Dans cet article je vais vous montrer comment implémenter un système d’authentification via jwt token dans une application Adonisjs v6. Mais avant de se lancer tête la première dans mon tutoriel je vous propose de vous poser la question si les jwt token sont vraiment la meilleur manière d’authentifier un utilisateur dans votre application.

Différent moyens de s’authentifier

Avant de commencer le tutoriel je vous invite à lire cette partie de la documentation pour savoir si c’est vraiment ce type d’authentification qui est le mieux pour votre application.

Pour comprendre les différentes façon d’authentifier un utilisateur, je vous invite à voir la rediffusion de ce live twitch où Romain Lanz, un core contributeur Adonisjs, qui vous explique les moyen de s’authentifier les plus commune avec leur avantages et leur inconvéniens.

Si vous voulez plutôt utiliser l’authentification par OAT token je vous invite à lire cet article.

Installation

Avant toute chose vérifiez que vous utilisez bien node v22 ou au moins node v20.6.

node -v
# v22.x.x

Il existe 3 starter kit officiel différent pour ne pas commencer à développer son application à partir de rien. Pour cette application on va utiliser le starter kit api avec le guard d’authentification session car le package jwt qu’on va utiliser se base sur ce guard.

npm init adonisjs@latest -- -K=api --auth-guard=session

Choisissez sqlite comme base de données ça va nous faciliter grandement la tâche pour ce tutoriel.

Après avoir rentré les différentes informations que vous demande le CLI et être aller dans le nouveau dossier créé nous pouvons passer à la suite.

Ensuite nous allons installer le package qui va nous permettre d’avoir un jwt guard pour notre api et utiliser les jwt token.

npm i @maximemrf/adonisjs-jwt

N’hésitez pas à mettre une ⭐️ à mon package jwt ça me motive vraiment à contribuer à l’opensource et d’écrire d’autre articles ❤️

Configuration

Dans un premier temps nous allons configurer notre guard situé à /config/auth.ts , remplacez la configuration par celle-ci:

import { defineConfig } from '@adonisjs/auth'
import { sessionUserProvider } from '@adonisjs/auth/session'
import { jwtGuard } from '@maximemrf/adonisjs-jwt/jwt_config'
import type { InferAuthEvents, Authenticators } from '@adonisjs/auth/types'

const authConfig = defineConfig({
  default: 'jwt',
  guards: {
    jwt: jwtGuard({
      // tokenExpiresIn est la durée de validité du token, c'est une valeur optionnelle
      tokenExpiresIn: '1h',
      // si vous souhaitez envoyer le token dans un cookie à la place du header authorization, vous pouvez définir useCookies à true
      useCookies: true,
      provider: sessionUserProvider({
        model: () => import('#models/user'),
      }),
    }),
  },
})

Si vous souhaitez stocker et envoyer le token jwt via les cookies laissez useCookies à true. Sinon enlevez complètement l’option, les token devront être passé via le header authorization (bearer). Pareil pour l’expiration des tokens c’est une propriété optionnelle.

Controller

Voici les méthodes de login et register:

export default class AuthController {
  // login si on utilise authorization bearer
  async login({ request, response, auth }: HttpContext) {
    const { email, password } = await request.validateUsing(loginValidator)

    const user = await User.verifyCredentials(email, password)
    const token = await auth.use('jwt').generate(user)

    return response.ok({
      token: token,
      ...user.serialize(),
    })
  }
  // login si on utilise les cookies (useCookies: true)
  async login({ request, response, auth }: HttpContext) {
    const { email, password } = await request.validateUsing(loginValidator)

    const user = await User.verifyCredentials(email, password)
    await auth.use('jwt').generate(user)

    return response.ok({
      ...user.serialize(),
    })
  }
  async register({ request, response }: HttpContext) {
    const payload = await request.validateUsing(registerValidator)

    const user = await User.create(payload)

    return response.created(user)
  }
}

Pour des explications plus précises sur le fonctionnement des Controllers/Validateurs utilisé dans ce code ce sont les même que ce tutoriel.

Conclusion

Vous savez maintenant comment créer un système d’authentification par JWT avec Adonisjs. N’hésitez pas à lire la documentation officielle pour en apprendre d’avantage.

In this article, I will show you how to set up an AdonisJS version 6 application and create an OAT (Opaque Access Token) authentication system.

Before starting the tutorial, I invite you to read this part of the documentation to find out if this type of authentication is really the best for your application.

Here is the github repository with the complete code.

Feel free to give a ⭐ to my repository, it really help me a lot !

🇫🇷 Tu parles français ? voici le lien de l’article en français

Installation

To begin, verify that you use node v22 or at least node v20.6

node -v # v22.x.x

There are three different official starter kits available so you don’t have to start developing your application from scratch.

There’s the “slim” kit, which contains just the core of the framework and the default file and folder structure of AdonisJS.

Next comes the “web” kit, which includes a variety of AdonisJS packages like Lucid, which is the framework’s ORM, and a template engine called Edge. The web kit is a good base for creating an application that renders views in HTML or Alpine.js, for example.

Finally, the third kit, and the one we will use, is the “API” kit. It allows you to easily create APIs that render JSON.

Below is the command to install it. Here, for the -K flag, we will choose “api” and we will specify that we want OAT tokens with this flag: --auth-guard=access_tokens

npm init adonisjs@latest -- -K=api --auth-guard=access_tokens

After entering the various details requested by the CLI and going into the newly created folder, we can proceed to the next step.

Migrations

During the installation of the kit, we could see that Lucid, the ORM of AdonisJS, was automatically configured to use SQLite as the database. For practical reasons, we will keep this database for the rest of the tutorial. The authentication package was also configured to use OATs, so there’s nothing more for us to do on that front.

If we run node ace migration:status, we can see that the kit has automatically created two migrations: one for the tokens and one for the user. All that's left is to migrate the database with the following command.

node ace migration:run

Controller

Once the tables have been migrated it is time to create our controller for authentication and give it the name “auth”.

node ace make:controller auth

The new controller is situated at app/controller

Creation of the register route

To begin we will create the register route which will allow us to register user to our application.

import type { HttpContext } from '@adonisjs/core/http'
import { registerValidator } from '#validators/auth'
import User from '#models/user'

export default class AuthController {
  async register({ request, response }: HttpContext) {
    const payload = await request.validateUsing(registerValidator)

    const user = await User.create(payload)

    return response.created(user)
  }
}

To validate data that will be transmitted to our backend we will create a validator.

node ace make:validator auth

Go to app/validator/auth.ts

We will define an object that will have 3 properties:

fullName, which must have a minimum length of 3 characters and a maximum of 64, and must be of type string.

Email, which must be of type string and be an email :) and, most importantly, must be unique in the database.

Password, of type string with a minimum length of 12 characters up to 512.

import vine from '@vinejs/vine'

export const registerValidator = vine.compile(
  vine.object({
    fullName: vine.string().minLength(3).maxLength(64),
    email: vine
      .string()
      .email()
      .unique(async (query, field) => {
        const user = await query.from('users').where('email', field).first()
        return !user
      }),
    password: vine.string().minLength(12).maxLength(512),
  })
)

We have one last step before testing our register route. Go to start/routes.ts to create the route using our controller.

We will replace the existing code by the code below:

import router from '@adonisjs/core/services/router'

const AuthController = () => import('#controllers/auth_controller')

router.group(() => {
  router.post('register', [AuthController, 'register'])
}).prefix('user')

We will import our controller and create a POST route linked to the register method of our controller.

You can see that I have created a router group with a user prefix. All the routes we put in this group will have the user prefix, which means our register route will not have the URL /register but /user/register.

Start the development server with this command: node ace serve and then test the route http://localhost:3333/user/register using Postman or another tool.

Example body:

{
    "fullName": "Maxime",
    "email": "max@ime.test",
    "name": "Maxime",
    "password": "12345678"
}

If everything goes well, the server should return a 201 Created response with the user's information.

To prevent the server from returning the hash of the password when you request the user, go to app/models/users.ts and add { serializeAs: null } in the argument of the @column decorator for the password.

@column({ serializeAs: null })
declare password: string

Now, when the server returns the user, it will replace the value of the password field with null. Feel free to visit here to learn more.

Creation of the login route

We will add a validator on the validator file.

import vine from '@vinejs/vine'

// new validator
export const loginValidator = vine.compile(
  vine.object({
    email: vine.string().email(),
    password: vine.string().minLength(8).maxLength(32),
  })
)

export const registerValidator = vine.compile(
  vine.object({
    fullName: vine.string().minLength(3).maxLength(64),
    email: vine
      .string()
      .email()
      .unique(async (db, value) => {
        const user = await db.from('users').where('email', value).first()
        return !user
      }),
    password: vine.string().minLength(12).maxLength(512),
  })
)

As well as a login method in our controller

import type { HttpContext } from '@adonisjs/core/http'
import User from '#models/user'
import { registerValidator, loginValidator } from '#validators/auth'

export default class AuthController {
  // new login method
  async login({ request, response }: HttpContext) {
    const { email, password } = await request.validateUsing(loginValidator)

    const user = await User.verifyCredentials(email, password)
    const token = await User.accessTokens.create(user)

    return response.ok({
      token: token,
      ...user.serialize(),
    })
  }
  async register({ request, response }: HttpContext) {
    const payload = await request.validateUsing(registerValidator)

    const user = await User.create(payload)

    return response.created(user)
  }
}

Then add the login route to the router

import router from '@adonisjs/core/services/router'

const AuthController = () => import('#controllers/auth_controller')

router.group(() => {
  router.post('register', [AuthController, 'register'])
  router.post('login', [AuthController, 'login'])
}).prefix('user')

When we test the route it will return the user as well as the token.

Creation of a token protected route

Now that we have our login and register routes, we will create a route that is accessible only by a registered user.

import router from '@adonisjs/core/services/router'
import { middleware } from './kernel.js'

const AuthController = () => import('#controllers/auth_controller')

router.group(() => {
  router.post('register', [AuthController, 'register'])
  router.post('login', [AuthController, 'login'])
}).prefix('user')

// add this route
router.get('me', async ({ auth, response }) => {
  try {
    const user = auth.getUserOrFail()
    return response.ok(user)
  } catch (error) {
    return response.unauthorized({ error: 'User not found' })
  }
})
.use(middleware.auth())

The route will return the user’s information. We can see that we defined a middleware with the api guard and used the auth object to access the user. To test the route, include the Authorization header with the value of the token (Bearer token).

Creation of the logout route

Now that we have our login and register routes, the only thing missing is the route for logging out. To log out a user, we simply need to delete their token from the database.

We will add a new logout method in our controller:

import type { HttpContext } from '@adonisjs/core/http'
import User from '#models/user'
import { registerValidator, loginValidator } from '#validators/auth'

export default class AuthController {
  async login({ request, response }: HttpContext) {
    const { email, password } = await request.validateUsing(loginValidator)
    const user = await User.verifyCredentials(email, password)
    const token = await User.accessTokens.create(user)
    return response.ok({
      token: token,
      ...user.serialize(),
    })
  }
  async register({ request, response }: HttpContext) {
    const payload = await request.validateUsing(registerValidator)
    const user = await User.create(payload)
    return response.created(user)
  }
  // Notre nouvelle route logout
  async logout({ auth, response }: HttpContext) {
    const user = auth.getUserOrFail()
    const token = auth.user?.currentAccessToken.identifier
    if (!token) {
      return response.badRequest({ message: 'Token not found' })
    }
    await User.accessTokens.delete(user, token)
    return response.ok({ message: 'Logged out' })
  }
}

The auth object allows us to retrieve the authenticated user and their token. We will then check that this token is not undefined (which shouldn't happen since the user is authenticated), delete the token, and inform the client that the operation was successful.

Then, add the new route:

router.group(() => {
  router.post('register', [AuthController, 'register'])
  router.post('login', [AuthController, 'login'])
  // the new logout route
  router.post('logout', [AuthController, 'logout']).use(middleware.auth())
}).prefix('user')

As with the me route, we will tell it to use the authentication middleware with use(middleware.auth()) to access the auth object in our controller method.

If you wanna see the complete code go to my github repository.

Conclusion

You now know how to create an OAT authentication system with AdonisJS. Feel free to read the official documentation to learn more.