Documentation structure for LLMs (llms.txt)
CircleCI Logo

Debug jobs with SSH rerun

3 days ago
Cloud Server v4+

Troubleshoot problems in a job using the Rerun job with SSH option. Using this feature you can inspect log files, running processes, and directory paths.

Introduction

CircleCI gives you the option to access all jobs using SSH. Read our blog post on debugging CI/CD pipelines with SSH.

When you access a job with SSH, you are running an interactive login shell. You may be running the command on top of the directory where the command failed the first time, or from the directory one level up from where the command failed. Either way, you will not be initiating a clean run. You may wish to execute pwd or ls to ensure that you are in the correct directory.

Regular CircleCI pipelines execute each step in a non-interactive shell. Steps run using an interactive login shell, when using rerun with SSH, may succeed, but fail in non-interactive mode.

Steps to access a job with SSH

Using self-hosted runners? To use SSH reruns with CircleCI runners you will first need to enable the SSH rerun feature. Refer to the Machine Runner 3 or Container Runner pages for steps.

  1. If you have not already done so, add an SSH key to your User Settings. This key will be used to connect to container or virtual machine (VM) that is running your job.

    If you are using a GitHub or Bitbucket organization, you can add an SSH key to your VCS account (GitHub or Bitbucket) rather than in the CircleCI app under User Settings if you prefer. For more on organization types, see the Users, Organizations, and Integrations Guide.

  2. In the CircleCI web app, select your org from the org cards on your user homepage.

  3. Select Pipelines from the sidebar and locate your pipeline from the list. You can use the project, branch, date, and status search options to help.

  4. Select the job you want to access by selecting the job name.

  5. To start a job with SSH enabled, select the Rerun job with SSH option from the Rerun dropdown menu.

    The "Rerun job with SSH" feature is intended for debugging purposes. These jobs will be created inside of the same pipeline as the original job.

    Screenshot showing rerun job with SSH option
    Figure 1. Screenshot showing the "Rerun job with SSH" option in the "Rerun" dropdown menu

  6. To see the connection details, expand the Enable SSH section in the job output where you will see the SSH command needed to connect. The details are displayed again in the Wait for SSH section at the end of the job.

    After clicking Rerun Job With SSH, if your job does not populate the Enable SSH step, you have not added an SSH key to your User Settings.
    Screenshot showing SSH connection details in the Enable SSH section
    Figure 2. Screenshot showing SSH connection details in the Enable SSH section

  7. Run the connection command given in the job output and follow the instructions to SSH into the running job (using the same SSH key that you use for GitHub or Bitbucket) to perform whatever troubleshooting you need to.

    If you are using the Windows executor, you will need to pass in the shell you want to use when using SSH. For example, to run powershell in your build you would run: ssh -p <remote_ip> -- powershell.exe.

The job VM will remain available for an SSH connection for 10 minutes after the pipeline finishes running and then automatically shut down (or you can cancel it). After you SSH into the job, the connection will remain open for one hour for customers on the Free Plan, or two hours for all other customers.

If your job has parallel steps, CircleCI launches more than one VM to perform them. You will see more than one 'Enable SSH' and 'Wait for SSH' section in the job output.

If you rerun a workflow containing a job that was previously rerun with SSH, the new job run will have SSH enabled. The job will have SSH enabled even if the SSH capability is disabled at the project level.

Debugging: "Permission denied"

If you run into permission issues trying to SSH to your job, try the following in the sections below.

Verify authentication with GitHub/Bitbucket

  1. Test that your keys are set up as expected by running one of the following commands.

    For GitHub, run:

    $ ssh git@github.com

    Or, for Bitbucket, run:

    $ ssh -Tv git@bitbucket.org

  2. Check the output for the following messages:

    Hi :username! You've successfully authenticated...
    logged in as :username.

  3. If you do not see this output, try troubleshooting with the following resources:

Verify you are authenticating as the correct user

You must authenticate as the user who triggered the rerun and who has access to the project. Verify your username appears in the 'Enable SSH' step.

If you are authenticating as the wrong user, specify a different SSH key using ssh -i. See the next section for guidance on confirming which key is being offered.

Verify the correct key is offered to CircleCI

If you have verified that you can authenticate as the correct user, but you are still getting "Permission denied" from CircleCI, you may be offering the wrong credentials to CircleCI.

  1. Check which key is being offered that authenticates you by running one of the following commands:

    $ ssh -v git@github.com

    Or,

    $ ssh -v git@bitbucket.com

  2. In the output, look for a sequence like this:

    debug1: Offering public key: /Users/me/.ssh/id_ed25519_github
<...>
debug1: Authentication succeeded (publickey).

    This sequence indicates that the key /Users/me/.ssh/id_ed25519_github is the one which your VCS accepted.

  3. Run the SSH command for your CircleCI job, but add the -v flag. In the output, look for one or more lines like this:

    debug1: Offering public key: ...

  4. Make sure that the key which your VCS accepted (in our example, /Users/me/.ssh/id_ed25519_github) was also offered to CircleCI.

  5. If it was not offered, you can specify it using the -i command-line argument to SSH. For example:

    $ ssh -i /Users/me/.ssh/id_ed25519_github -p 64784 54.224.97.243

You can increase the verbosity level to get more detailed debugging information by using multiple -v flags. For example, use -vv for more details:

$ ssh -vv git@github.com

Or use -vvv for maximum verbosity:

$ ssh -vvv git@github.com