Errors & Retrying

When an uncaught error is thrown inside your task, that task attempt will fail. You can configure retrying in two ways:

In your trigger.config file you can set the default retrying behavior for all tasks.
On each task you can set the retrying behavior.

Task-level retry settings override the defaults in your trigger.config file.

By default when you create your project using the CLI init command we disabled retrying in the DEV environment. You can enable it in your trigger.config file.

A simple example with OpenAI

This task will retry 10 times with exponential backoff.

openai.chat.completions.create() can throw an error.
The result can be empty and we want to try again. So we manually throw an error.

/trigger/openai.ts

import { task } from "@trigger.dev/sdk";
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

export const openaiTask = task({
  id: "openai-task",
  //specifying retry options overrides the defaults defined in your trigger.config file
  retry: {
    maxAttempts: 10,
    factor: 1.8,
    minTimeoutInMs: 500,
    maxTimeoutInMs: 30_000,
    randomize: false,
  },
  run: async (payload: { prompt: string }) => {
    //if this fails, it will throw an error and retry
    const chatCompletion = await openai.chat.completions.create({
      messages: [{ role: "user", content: payload.prompt }],
      model: "gpt-3.5-turbo",
    });

    if (chatCompletion.choices[0]?.message.content === undefined) {
      //sometimes OpenAI returns an empty response, let's retry by throwing an error
      throw new Error("OpenAI call failed");
    }

    return chatCompletion.choices[0].message.content;
  },
});

Combining tasks

One way to gain reliability is to break your work into smaller tasks and trigger them from each other. Each task can have its own retrying behavior:

/trigger/multiple-tasks.ts

import { task } from "@trigger.dev/sdk";

export const myTask = task({
  id: "my-task",
  retry: {
    maxAttempts: 10,
  },
  run: async (payload: string) => {
    const result = await otherTask.triggerAndWait("some data");
    //...do other stuff
  },
});

export const otherTask = task({
  id: "other-task",
  retry: {
    maxAttempts: 5,
  },
  run: async (payload: string) => {
    return {
      foo: "bar",
    };
  },
});

Another benefit of this approach is that you can view the logs and retry each task independently from the dashboard.

Retrying smaller parts of a task

Another complimentary strategy is to perform retrying inside of your task. We provide some useful functions that you can use to retry smaller parts of a task. Of course, you can also write your own logic or use other packages.

retry.onThrow()

You can retry a block of code that can throw an error, with the same retry settings as a task.

/trigger/retry-on-throw.ts

import { task, logger, retry } from "@trigger.dev/sdk";

export const retryOnThrow = task({
  id: "retry-on-throw",
  run: async (payload: any) => {
    //Will retry up to 3 times. If it fails 3 times it will throw.
    const result = await retry.onThrow(
      async ({ attempt }) => {
        //throw on purpose the first 2 times, obviously this is a contrived example
        if (attempt < 3) throw new Error("failed");
        //...
        return {
          foo: "bar",
        };
      },
      { maxAttempts: 3, randomize: false }
    );

    //this will log out after 3 attempts of retry.onThrow
    logger.info("Result", { result });
  },
});

If all of the attempts with retry.onThrow fail, an error will be thrown. You can catch this or let it cause a retry of the entire task.

retry.fetch()

You can use fetch, axios, or any other library in your code. But we do provide a convenient function to perform HTTP requests with conditional retrying based on the response:

/trigger/retry-fetch.ts

import { task, logger, retry } from "@trigger.dev/sdk";

export const taskWithFetchRetries = task({
  id: "task-with-fetch-retries",
  run: async ({ payload, ctx }) => {
    //if the Response is a 429 (too many requests), it will retry using the data from the response. A lot of good APIs send these headers.
    const headersResponse = await retry.fetch("http://my.host/test-headers", {
      retry: {
        byStatus: {
          "429": {
            strategy: "headers",
            limitHeader: "x-ratelimit-limit",
            remainingHeader: "x-ratelimit-remaining",
            resetHeader: "x-ratelimit-reset",
            resetFormat: "unix_timestamp_in_ms",
          },
        },
      },
    });
    const json = await headersResponse.json();
    logger.info("Fetched headers response", { json });

    //if the Response is a 500-599 (issue with the server you're calling), it will retry up to 10 times with exponential backoff
    const backoffResponse = await retry.fetch("http://my.host/test-backoff", {
      retry: {
        byStatus: {
          "500-599": {
            strategy: "backoff",
            maxAttempts: 10,
            factor: 2,
            minTimeoutInMs: 1_000,
            maxTimeoutInMs: 30_000,
            randomize: false,
          },
        },
      },
    });
    const json2 = await backoffResponse.json();
    logger.info("Fetched backoff response", { json2 });

    //You can additionally specify a timeout. In this case if the response takes longer than 1 second, it will retry up to 5 times with exponential backoff
    const timeoutResponse = await retry.fetch("https://httpbin.org/delay/2", {
      timeoutInMs: 1000,
      retry: {
        timeout: {
          maxAttempts: 5,
          factor: 1.8,
          minTimeoutInMs: 500,
          maxTimeoutInMs: 30_000,
          randomize: false,
        },
      },
    });
    const json3 = await timeoutResponse.json();
    logger.info("Fetched timeout response", { json3 });

    return {
      result: "success",
      payload,
      json,
      json2,
      json3,
    };
  },
});

If all of the attempts with retry.fetch fail, an error will be thrown. You can catch this or let it cause a retry of the entire task.

Advanced error handling and retrying

We provide a catchError callback on the task and in your trigger.config file. This gets called when an uncaught error is thrown in your task. You can

Inspect the error, log it, and return a different error if you’d like.
Modify the retrying behavior based on the error, payload, context, etc.

If you don’t return anything from the function it will use the settings on the task (or inherited from the config). So you only need to use this to override things.

OpenAI error handling example

OpenAI calls can fail for a lot of reasons and the ideal retry behavior is different for each. In this complicated example:

We skip retrying if there’s no Response status.
We skip retrying if you’ve run out of credits.
If there are no Response headers we let the normal retrying logic handle it (return undefined).
If we’ve run out of requests or tokens we retry at the time specified in the headers.

import { task } from "@trigger.dev/sdk";
import { calculateISO8601DurationOpenAIVariantResetAt, openai } from "./openai.js";

export const openaiTask = task({
  id: "openai-task",
  retry: {
    maxAttempts: 1,
  },
  run: async (payload: { prompt: string }) => {
    const chatCompletion = await openai.chat.completions.create({
      messages: [{ role: "user", content: payload.prompt }],
      model: "gpt-3.5-turbo",
    });

    return chatCompletion.choices[0].message.content;
  },
  catchError: async ({ payload, error, ctx, retryAt }) => {
    if (error instanceof OpenAI.APIError) {
      if (!error.status) {
        return {
          skipRetrying: true,
        };
      }

      if (error.status === 429 && error.type === "insufficient_quota") {
        return {
          skipRetrying: true,
        };
      }

      if (!error.headers) {
        //returning undefined means the normal retrying logic will be used
        return;
      }

      const remainingRequests = error.headers["x-ratelimit-remaining-requests"];
      const requestResets = error.headers["x-ratelimit-reset-requests"];

      if (typeof remainingRequests === "string" && Number(remainingRequests) === 0) {
        return {
          retryAt: calculateISO8601DurationOpenAIVariantResetAt(requestResets),
        };
      }

      const remainingTokens = error.headers["x-ratelimit-remaining-tokens"];
      const tokensResets = error.headers["x-ratelimit-reset-tokens"];

      if (typeof remainingTokens === "string" && Number(remainingTokens) === 0) {
        return {
          retryAt: calculateISO8601DurationOpenAIVariantResetAt(tokensResets),
        };
      }
    }
  },
});

import { OpenAI } from "openai";

export const openai = new OpenAI({ apiKey: env.OPENAI_API_KEY });

export function calculateISO8601DurationOpenAIVariantResetAt(
  resets: string,
  now: Date = new Date()
): Date | undefined {
  // Check if the input is null or undefined
  if (!resets) return undefined;

  // Regular expression to match the duration string pattern
  const pattern = /^(?:(\d+)d)?(?:(\d+)h)?(?:(\d+)m)?(?:(\d+(?:\.\d+)?)s)?(?:(\d+)ms)?$/;
  const match = resets.match(pattern);

  // If the string doesn't match the expected format, return undefined
  if (!match) return undefined;

  // Extract days, hours, minutes, seconds, and milliseconds from the string
  const days = parseInt(match[1] ?? "0", 10) || 0;
  const hours = parseInt(match[2] ?? "0", 10) || 0;
  const minutes = parseInt(match[3] ?? "0", 10) || 0;
  const seconds = parseFloat(match[4] ?? "0") || 0;
  const milliseconds = parseInt(match[5] ?? "0", 10) || 0;

  // Calculate the future date based on the current date plus the extracted time
  const resetAt = new Date(now);
  resetAt.setDate(resetAt.getDate() + days);
  resetAt.setHours(resetAt.getHours() + hours);
  resetAt.setMinutes(resetAt.getMinutes() + minutes);
  resetAt.setSeconds(resetAt.getSeconds() + Math.floor(seconds));
  resetAt.setMilliseconds(
    resetAt.getMilliseconds() + (seconds - Math.floor(seconds)) * 1000 + milliseconds
  );

  return resetAt;
}

Preventing retries

Using `AbortTaskRunError`

You can prevent retries by throwing an AbortTaskRunError. This will fail the task attempt and disable retrying.

/trigger/myTasks.ts

import { task, AbortTaskRunError } from "@trigger.dev/sdk";

export const openaiTask = task({
  id: "openai-task",
  run: async (payload: { prompt: string }) => {
    //if this fails, it will throw an error and stop retrying
    const chatCompletion = await openai.chat.completions.create({
      messages: [{ role: "user", content: payload.prompt }],
      model: "gpt-3.5-turbo",
    });

    if (chatCompletion.choices[0]?.message.content === undefined) {
      // If OpenAI returns an empty response, abort retrying
      throw new AbortTaskRunError("OpenAI call failed");
    }

    return chatCompletion.choices[0].message.content;
  },
});

Using try/catch

Sometimes you want to catch an error and don’t want to retry the task. You can use try/catch as you normally would. In this example we fallback to using Replicate if OpenAI fails.

/trigger/myTasks.ts

import { task } from "@trigger.dev/sdk";

export const openaiTask = task({
  id: "openai-task",
  run: async (payload: { prompt: string }) => {
    try {
      //if this fails, it will throw an error and retry
      const chatCompletion = await openai.chat.completions.create({
        messages: [{ role: "user", content: payload.prompt }],
        model: "gpt-3.5-turbo",
      });

      if (chatCompletion.choices[0]?.message.content === undefined) {
        //sometimes OpenAI returns an empty response, let's retry by throwing an error
        throw new Error("OpenAI call failed");
      }

      return chatCompletion.choices[0].message.content;
    } catch (error) {
      //use Replicate if OpenAI fails
      const prediction = await replicate.run(
        "meta/llama-2-70b-chat:02e509c789964a7ea8736978a43525956ef40397be9033abf9fd2badfe68c9e3",
        {
          input: {
            prompt: payload.prompt,
            max_new_tokens: 250,
          },
        }
      );

      if (prediction.output === undefined) {
        //retry if Replicate fails
        throw new Error("Replicate call failed");
      }

      return prediction.output;
    }
  },
});

Replay failed runs in bulk

After you deploy a fix, use bulk actions to replay multiple failed runs from the dashboard, or bulk actions with the SDK to replay them from backend code. Bulk replay creates an asynchronous action that targets selected run IDs or a runs.list() filter, so you can retry a known failure set without replaying each run individually.

Getting started

Fundamentals

Building with AI

Writing tasks

Agents

Configuration

Deployment

Private networking

Realtime

CLI

Observability

Using the Dashboard

Troubleshooting

Self-hosting

Open source

Help

A simple example with OpenAI

Combining tasks

Retrying smaller parts of a task

retry.onThrow()

retry.fetch()

Advanced error handling and retrying

OpenAI error handling example

Preventing retries

Using `AbortTaskRunError`

Using try/catch

Replay failed runs in bulk

​A simple example with OpenAI

​Combining tasks

​Retrying smaller parts of a task

​retry.onThrow()

​retry.fetch()

​Advanced error handling and retrying

​OpenAI error handling example

​Preventing retries

​Using AbortTaskRunError

​Using try/catch

​Replay failed runs in bulk

A simple example with OpenAI

Combining tasks

Retrying smaller parts of a task

retry.onThrow()

retry.fetch()

Advanced error handling and retrying

OpenAI error handling example

Preventing retries

Using `AbortTaskRunError`

Using try/catch

Replay failed runs in bulk