Test: GitHub cloud automations #340970

Refs: #339407, #339419, #340420, #340434, #340632, #340652

• anyOS Vritant Bhardwaj (@vritant24)
• anyOS Alex Ross (@alexr00)

Complexity: 5

Authors: Ben Villalobos (@benvillalobos)

Create Issue

Context

GitHub cloud automations are now available as an experimental feature in the Agents window. This test covers the basic scenarios: creating, editing, running, and deleting cloud automations from VS Code and checking that each change shows up on GitHub.

Expected limitations:

• You can't continue a cloud automation run from inside VS Code.
• You can't ask the AI to create a cloud automation for you.
• Cloud automations are only supported in private repositories for now.

Set up

• Turn on chat.automations.cloud.enabled.

Part 1: Create, edit, and delete cloud automations in VS Code

1. Create from a GitHub repo

• Go to the Automations view and click New Automation.
• In the workspace picker, click Work in GitHub and search for a repo.
• Pick a private repo that you don't have cloned locally.
  • Verify: the only session type is Cloud, and it's grayed out.
• Pick a model, click Configure Allowed Tools and disable any number of tools, then give it a schedule and create it.
• Go to https://github.com/<owner>/<repo>/agents/automations, using the private repo you picked.
  • Verify: the automation shows up there.
• In VS Code, open the automation's More menu and click Open on GitHub. On GitHub, click Edit.
  • Verify: everything you chose matches, including the prompt, model, and schedule.
  • Verify: the tools you disabled don't show up in the tools list.

2. Create from a local clone

• Click New Automation and pick a folder that's a clone of a private repo.
• Expand the session type picker.
  • Verify: all the options are there, including Cloud.
• Select Cloud, give it a schedule, and create it.
  • Verify: it shows up on the repo's automations page on GitHub.

3. Cloud only shows up for private repos

• Pick a clone of a public repo.
  • Verify: Cloud doesn't appear in the session type picker.
• Pick a folder that isn't a Git repo.
  • Verify: Cloud doesn't appear.
• Pick a clone of a private repo and select Cloud, then switch to any other folder.
  • Verify: Cloud is no longer selected and the session type goes back to Copilot.

4. Edit

• Edit an automation you created: change its title, prompt, model, and allowed tools, and save.
  • Verify: the changes show up on GitHub.
• Disable the automation from its card in the Automations view.
  • Verify: it shows as disabled on GitHub.
• Enable it again.
  • Verify: it shows as enabled on GitHub.

5. Delete

• Delete one of the automations from its card.
  • Verify: it's gone from the repo's automations page on GitHub.

6. Schedule types

• Edit the remaining automation and save it once with each schedule: Hourly, Manual, Daily, Weekdays, and Weekly.
  • Verify: after each save, GitHub shows the matching schedule.

7. Run and open on GitHub

• Run the automation from its card.
  • Verify: a run appears in its run history.
  • Verify: the run's status updates on its own when it finishes, without clicking Refresh.
• On the run history card, click Open on GitHub.
  • Verify: GitHub opens to that automation run.

8. Changes on GitHub show up in VS Code

• On GitHub, create an automation in a private repo that VS Code knows about (it shows up in the workspace picker).
• In the Automations view, click Refresh.
  • Verify: the new automation shows up.
• Edit that automation on GitHub and click Refresh.
  • Verify: the edit shows up in VS Code.
• Delete that automation on GitHub and click Refresh.
  • Verify: it's gone from VS Code.