Skip to content

Conversation

@rwoll
Copy link
Member

@rwoll rwoll commented Aug 18, 2025

When running in Scenario Automation (distinct from stest mode and set via an env var), as much production code as possible should be used so the real user experience is measured. However, certain codepaths — like telemetry — should be disabled regardless of the formal VSCode ExtensionMode, and certain codepaths (like full OAuth flow) must not be run.

When running local builds, extensionMode is ExtensionMode.Test while Marketplace downloaded builds are ExtensionMode.Production so relying solely on extensionMode is not possible.

An abandoned iteration of this PR attempted to colocate all this logic within one central location but I found that made the callsites that dependend on that logic particularly hard to follow, hence this PR proposes a less DRY, but more readable approach.

@rwoll rwoll marked this pull request as ready for review August 18, 2025 22:26
@rwoll rwoll requested a review from Copilot August 18, 2025 22:27
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces conditional logic to allow more production services to run when in Scenario Automation mode, which is a testing mode that aims to measure the real user experience while still disabling certain codepaths like telemetry and OAuth flows. The changes modify extension mode checks throughout the codebase to exclude scenario automation from test-mode restrictions.

  • Adds isScenarioAutomation checks to various extension mode conditions
  • Enables production services (URL opener, notifications) in scenario automation mode
  • Maintains telemetry and experimentation service restrictions in scenario automation
  • Adjusts timeout behavior for prompt elements to use production timeouts

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/extension/prompts/node/panel/symbolDefinitions.tsx Updates timeout logic to use production timeout when in scenario automation
src/extension/prompts/node/panel/safeElements.tsx Prevents throwing errors in scenario automation mode for better production simulation
src/extension/prompts/node/panel/referencesAtPosition.tsx Updates timeout logic to use production timeout when in scenario automation
src/extension/prompts/node/panel/definitionAtPosition.tsx Updates timeout logic to use production timeout when in scenario automation
src/extension/onboardDebug/vscode-node/copilotDebugCommandContribution.ts Adjusts version nonce generation to use production behavior in scenario automation
src/extension/log/vscode-node/loggingActions.ts Disables fetcher telemetry collection in scenario automation mode
src/extension/inlineChat/vscode-node/inlineChatCommands.ts Enables review service context value updates in scenario automation
src/extension/extension/vscode/services.ts Enables production URL opener and notification services in scenario automation
src/extension/extension/vscode/extension.ts Prevents early return from activation in scenario automation mode
src/extension/extension/vscode-node/services.ts Maintains telemetry and experimentation service restrictions in scenario automation

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

@rwoll rwoll requested a review from connor4312 August 18, 2025 22:32
@justschen justschen assigned rwoll and unassigned justschen Aug 18, 2025
connor4312
connor4312 previously approved these changes Aug 18, 2025
@vs-code-engineering vs-code-engineering bot added this to the August 2025 milestone Aug 18, 2025
@rwoll rwoll enabled auto-merge August 18, 2025 22:46
@rwoll rwoll disabled auto-merge August 18, 2025 22:46
@rwoll rwoll enabled auto-merge August 18, 2025 22:46
@rwoll rwoll added this pull request to the merge queue Aug 18, 2025
Merged via the queue into microsoft:main with commit 3cf3708 Aug 18, 2025
6 checks passed
@rwoll rwoll deleted the allow-more-services-in-scenario-automation-mode branch August 18, 2025 23:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants