NES: add applied-then-undone rejected-edit memory mode - #341436

HectorHZZZ · 2026-10-14T01:10:13Z

Summary

Adds a second rejected-edit memory mode, memory.rejectedEdits: "appliedThenUndone", next to the existing diffWithTags (#327367).

Instead of a <|rejected/|> tag plus an instruction, the rejected suggestion is written into edit_diff_history as ordinary entries, at the position where the rejection happened: the suggestion is applied and then undone by the developer. Later edits push the pair back like any other history entry.

There is no tag, no instruction, and no system prompt change. The prompt's postscript already says "Avoid undoing or reverting the developer's last change", and this mode relies on that existing behaviour.

--- src/dialog-form.tsx
+++ src/dialog-form.tsx
@@ -170,1 +170,1 @@
-    isRequired={false}
+    isRequired={true}        <- applied

--- src/dialog-form.tsx
+++ src/dialog-form.tsx
@@ -170,1 +170,1 @@
-    isRequired={true}
+    isRequired={false}       <- undone

Configuration

Everything is controlled through memory in the model configuration. With memory.rejectedEdits unset (the default), nothing changes: rejections are not recorded into the history, the prompt is unchanged, and a rejection marks the cache as before. Every other option applies only when rejectedEdits is set.

Option	Optional values	Default	Applies to	Meaning
`rejectedEdits`	`diffWithTags`, `appliedThenUndone`	unset (off)	-	Turns the memory on and selects how a rejection is written into `edit_diff_history`. `diffWithTags`: one entry with a `<\|rejected/\|>` tag plus an instruction (existing). `appliedThenUndone`: the suggestion applied, then undone, as ordinary entries with no tag and no instruction
`rejectedEditsMaxEditsAfter`	number	`4`	both modes	A rejection is used only while at most this many edit-history entries follow it, so older rejections are dropped to save prompt tokens. Consecutive edits to the same line count once
`rejectedEditsPlacement`	`inPlace`, `newest`	`inPlace`	both modes	`inPlace`: the rejection stays where it happened and later edits push it back like any other entry. `newest`: it is placed as the newest entries of the history on every request, while it is used
`rejectedEditsRepetitions`	number (at least 1)	`1`	`appliedThenUndone`	How many times the applied/undone pair is written
`rejectedEditsMaxCount`	number	all recorded rejections	both modes	Only this many of the most recent rejections are used
`rejectedEditsCacheScope`	`request`, `edit`, `none`	`request`	both modes	What a rejection marks as rejected in the next-edit cache. `request`: every cached suggestion of the same model request, including ones never shown (existing behaviour). `edit`: only the rejected suggestion, so the other suggestions of that response can still be shown. `none`: no cached suggestion is marked

appliedThenUndone additionally uses only rejections in the active document. Like every other history entry, a rejection keeps the line numbers it had when it happened.

Minimal configuration:

"memory": { "rejectedEdits": "appliedThenUndone" }

Running an experiment

Each experiment arm is one model configuration string, set through chat.advanced.inlineEdits.xtabProvider.modelConfigurationString. The arms differ only in memory; the rest of the configuration stays the same as the control. The memory options are read from the selected model's configuration, so the arm's string must use that model's name.

Arm	`memory`
Control	not set
Applied-then-undone, in place	`{ "rejectedEdits": "appliedThenUndone" }`
Applied-then-undone, newest	`{ "rejectedEdits": "appliedThenUndone", "rejectedEditsPlacement": "newest" }`
Applied-then-undone, in place, cache scope `edit`	`{ "rejectedEdits": "appliedThenUndone", "rejectedEditsCacheScope": "edit" }`
Tag (existing mode)	`{ "rejectedEdits": "diffWithTags" }`

For example, the second arm:

{
  "modelName": "<model name>",
  "promptingStrategy": "<strategy>",
  "includeTagsInCurrentFile": false,
  "memory": { "rejectedEdits": "appliedThenUndone" }
}

The value is this object serialized as a string. A version of the extension without this change does not know appliedThenUndone or the new options and ignores the whole configuration, so the experiment's version filter must include this change.

Offline evaluation

Offline evaluation details are in an internal issue: https://github.com/ai-platform-microsoft/aims-tscience-nl2code/issues/3348. In summary:

Applied-then-undone reduces exact repeats of a rejected suggestion much more than the tag does.
Suggestions unrelated to the rejection are affected less the further back the rejection is in the history.
Known risk: when the developer rejected one version of an edit on a line and wants a near-identical version, the model sometimes reproduces the rejected text. We plan to measure this during the flight.

Changes

xtabPromptOptions.ts:
- Add RejectedEditsMemoryMode.AppliedThenUndone and the options in the table above.
- isRejectedEditMemoryEnabled is now true for either mode.
diffHistoryForPrompt.ts:
- selectRejectedEditsForPrompt applies the conditions above.
- getEditDiffHistory takes the memory options and renders the applied/undone entries, at the rejection's position or as the newest entries. The pair counts as one entry towards nEntries. Over-budget rejections are skipped as before.
promptCrafting.ts / xtabProvider.ts:
- Use the selector in both the legacy and global-budget paths.
- The instruction is only added for diffWithTags.
nextEditProvider.ts / nextEditCache.ts: with rejectedEditsCacheScope: "edit", a rejection marks only the rejected cache entry.
Tests:
- Rendering, and each selection condition.
- The prompt is unchanged when there is no rejection.
- Prompt contents and config validation.

Validation

Validated with unit tests and in a local client build.

Copilot

🟡 Changes recommended

Rejection locations become stale after line-changing edits, and budget limits can emit incomplete synthetic histories.

2 open findings

Rebase rejection locations after document edits · New
Budget repeated rejection pairs atomically · New

What changed in this PR

Adds an opt-in applied-then-undone representation for rejected NES suggestions.

Changes:

Adds and validates the new memory mode.
Filters rejections by active document and cursor proximity.
Renders repeated apply/undo history entries with prompt tests.

File	Description
`xtabPromptOptions.spec.ts`	Tests configuration validation.
`xtabPromptOptions.ts`	Defines and enables the new mode.
`diffHistoryForPrompt.spec.ts`	Tests selection and rendering.
`promptCrafting.spec.ts`	Tests generated prompt content.
`xtabProvider.ts`	Applies rejection selection to global budgeting.
`promptCrafting.ts`	Integrates selection and mode-specific instructions.
`diffHistoryForPrompt.ts`	Implements proximity filtering and apply/undo rendering.

🧠 Review effort: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

HectorHZZZ · 2026-10-19T17:56:15Z

+	}
+	return rejectedEditHistory.filter(entry =>
+		entry.docId === activeDocId
+		&& entry.hunks.some(hunk => distanceToHunk(hunk, cursorLineOffset) <= APPLIED_THEN_UNDONE_MAX_CURSOR_DISTANCE)


The latest implementation no longer uses a cursor-proximity gate or checks whether the rejected lines are unchanged. It treats the rejection as ordinary historical edits, retaining its original line numbers and ordinal. Selection is limited to the active document and at most four later edit-history entries. The existing tests explicitly cover retaining a rejection after its lines are edited. Commit 62500dd also corrects the stale enum comment to match these rules. Could you confirm that this addresses the concern under the revised design?

HectorHZZZ · 2026-10-19T17:56:15Z

+		rejectedEditHistory.forEach((entry, i) => {
+			items.push({ ordinal: entry.ordinal, docId: entry.docId, isRejection: true, render: () => renderAppliedThenUndone(entry, workspacePath) });
+			items.push({ ordinal: newestOrdinal + 1 + i, docId: entry.docId, isRejection: true, render: () => renderDocDiff(entry.docId, entry.hunks.map(invertHunk), workspacePath, '') });


The latest implementation renders the applied/undone pair as a single history item and computes its token cost before inclusion, so an over-budget pair is skipped atomically. Commit 62500dd adds regression tests for a budget that fits only one diff, a budget one token below the complete pair, and an exact-fit budget with nEntries=0 and nEntries=1. They verify no partial pair is emitted, older normal history is preserved when a rejection is skipped, and a complete pair counts as one entry. All 126 tests across diffHistoryForPrompt.spec.ts, promptCrafting.spec.ts, and xtabPromptOptions.spec.ts passed locally. Leaving this thread open for reviewer confirmation.

Benjamin Steenhoek (bstee615)

Looks sensible overall, added some cleanup comments

Benjamin Steenhoek (bstee615) · 2026-10-19T19:55:42Z

+	if (mode !== RejectedEditsMemoryMode.AppliedThenUndone) {
+		return rejectedEditHistory;
+	}
+	return rejectedEditHistory.filter(entry =>


Should we make this filtering option applicable to any rejected edits mode? Should we make REJECTED_EDITS_MAX_EDITS_AFTER configurable so we can experiment on it without code changes

Done in 9018ee4. The limit is now a shared memory option, memory.rejectedEditsMaxEditsAfter (default 4), and applies to every rejectedEdits mode. It only takes effect when rejectedEdits is set, so traffic without the memory is unchanged. Only the active-document restriction stays specific to appliedThenUndone.

Benjamin Steenhoek (bstee615) · 2026-10-19T20:23:56Z

+	DiffWithTags = 'diffWithTags',
+	/**
+	 * Rejected hunks are rendered as ordinary edit history (applied, then undone by the developer),
+	 * with no tag and no instruction. Only rejections in the active document followed by at most four edit-history entries are used.


This comment might become outdated later

Done in 9018ee4: the number is removed from the comment. The default and its rationale now live on DEFAULT_REJECTED_EDITS_MAX_EDITS_AFTER.

Benjamin Steenhoek (bstee615) · 2026-10-19T20:26:00Z

+		}
+		items.push({ ordinal: entry.ordinal, docId: entry.docId, isRejection: false, render: () => generateDocDiff(entry, workspacePath) });
+	}
+	if (rejectedEditsMode === RejectedEditsMemoryMode.AppliedThenUndone) {


Same here, this could be simplified to set the render function based on the rejected edit mode.

Done in 9018ee4: the render function is chosen once from the mode.

Benjamin Steenhoek (bstee615) · 2026-10-19T20:28:46Z


+interface IDiffHistoryItem {
+	readonly ordinal: number;
+	readonly docId: DocumentId;


I'm not sure we need this extra bookkeeping for ordinal, docId, and isRejection since all those fields come from entry

Done in 9018ee4: the separate item type is gone. Edit and rejection entries are sorted together and rendered directly.

HectorHZZZ · 2026-10-19T21:54:52Z

Per the suggestion to include experiment options up front: added rejectedEditsPlacement (inPlace | newest), rejectedEditsRepetitions and rejectedEditsMaxCount next to rejectedEditsMaxEditsAfter, all under memory with defaults equal to the current behaviour. The PR description lists them.

HectorHZZZ · 2026-10-20T21:55:33Z

Added one more experiment option, rejectedEditsCacheScope (request | edit, default request). While testing locally I saw that rejecting one suggestion marks every cached suggestion of the same model request as rejected, including ones that were never shown, so later requests are answered from the cache without reaching the model. With edit, only the rejected entry is marked. It only takes effect when rejectedEdits is set, so traffic without the memory is unchanged.

vs-code-engineering · 2026-10-20T22:33:17Z

📬 CODENOTIFY

The following users are being notified based on files changed in this PR:

Ulugbek Abdullaev (@ulugbekna)

Matched files:

extensions/copilot/src/extension/inlineEdits/common/visibleLines.ts
extensions/copilot/src/extension/inlineEdits/node/nextEditCache.ts
extensions/copilot/src/extension/inlineEdits/node/nextEditProvider.ts
extensions/copilot/src/extension/inlineEdits/test/common/visibleLines.spec.ts
extensions/copilot/src/extension/inlineEdits/test/node/nextEditCacheCursorDistance.spec.ts
extensions/copilot/src/extension/inlineEdits/test/node/nextEditProviderCaching.spec.ts
extensions/copilot/src/extension/inlineEdits/vscode-node/inlineCompletionProvider.ts
extensions/copilot/src/extension/xtab/common/diffHistoryForPrompt.ts
extensions/copilot/src/extension/xtab/common/promptCrafting.ts
extensions/copilot/src/extension/xtab/node/xtabProvider.ts
extensions/copilot/src/extension/xtab/test/common/promptCrafting.spec.ts
extensions/copilot/src/extension/xtab/test/node/diffHistoryForPrompt.spec.ts
extensions/copilot/src/platform/inlineEdits/common/dataTypes/xtabPromptOptions.ts
extensions/copilot/src/platform/inlineEdits/test/common/xtabPromptOptions.spec.ts

HectorHZZZ · 2026-10-22T15:39:04Z

New option: rejectedEditsCacheScope: "none"

Today, when the user rejects a suggestion, every cached suggestion from the same model request is marked as rejected, including ones the user never saw. This option lets us test whether that over-blocks suggestions that would have been fine to show.

Value	What a rejection marks as rejected in the cache
`request` (default)	All cached suggestions of the same request, including unseen ones. This is the existing behavior.
`edit`	Only the suggestion that was rejected.
`none`	Nothing in the cache.

The default is unchanged, so nothing changes unless the option is set. It only takes effect together with memory.rejectedEdits; without it the whole request is always marked, as before.

none only turns off the cache marking. The separate check that stops the exact same edit from being shown again right after a rejection is not touched.

Also in this update: in appliedThenUndone mode, a rejected edit is only recorded in the edit history if all of it was inside the visible editor area when it was rejected. diffWithTags is unchanged.

Add a second `memory.rejectedEdits` mode next to `diffWithTags`. Instead of a `<|rejected/|>` tag plus an instruction, a rejected suggestion is written into `edit_diff_history` as ordinary entries at the position where the rejection happened: the suggestion applied, then undone by the developer. The prompt's postscript already says "Avoid undoing or reverting the developer's last change", so this mode adds no tag, no instruction and no system prompt change. - Rejections are recorded for either mode; `appliedThenUndone` uses only those in the active document. - The applied/undone pair is budgeted as a single history entry: it is included whole or skipped, and counts once towards `nEntries`. - The rejected-edit instruction is only added in `diffWithTags` mode. - With `memory.rejectedEdits` unset, the prompt is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…fter edits Offline, the effect of a rejection on repeated suggestions fades as later edits push it back, so a rejection is only included in the prompt while at most `memory.rejectedEditsMaxEditsAfter` edit-history entries follow it (default 4). This saves prompt tokens on rejections that no longer help. The limit applies to every `rejectedEdits` mode, so it also changes the existing `diffWithTags` mode: previously every recorded rejection (up to the tracker's limit of three) was eligible regardless of how many edits followed it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…it memory Add experiment options under `memory`, all defaulting to the current behavior: - `rejectedEditsPlacement` (`inPlace` | `newest`): keep a rejection where it happened, or place it as the newest history entries on every request while it is used. Applies to every mode. - `rejectedEditsRepetitions` (default 1): how many times the applied/undone pair is written in `appliedThenUndone` mode. The repeated pairs still count as one history entry. - `rejectedEditsMaxCount` (default: all recorded): use only this many of the most recent rejections. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…rejection marks Rejecting one suggestion marks every cached suggestion of the same model request as rejected, including ones that were never shown, so later requests are answered from the cache without reaching the model. Add `memory.rejectedEditsCacheScope` to experiment with this: - `request` (default): mark every cached suggestion of the request, as before. - `edit`: mark only the rejected cache entry, so the other suggestions of that response can still be shown. - `none`: mark nothing. The option only takes effect when `rejectedEdits` is set, so traffic without the memory is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…uld not fully see A suggestion that extends past the visible editor area may be rejected without the user having seen all of it, so recording it as a rejected edit can teach the model to avoid an edit the user never evaluated. Add `memory.rejectedEditsOnlyFullyVisible` (default false): when set, a rejection is recorded only if the whole edited line span, extended by the lines of the new text, lies within a single visible range. - The completion item records the edit in the document it applies to: for notebooks, the target cell document and a cell-relative range, not the notebook or the ghost-text display range. - Visibility is checked in the active editor when it shows that document, otherwise in any visible editor of it. - The option applies the same way to every `rejectedEdits` mode, so it can be flighted as its own arm. Co-authored-by: ulugbekna <ulugbekna@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI balanced review requested due to automatic review settings October 14, 2026 01:10

Copilot started reviewing on behalf of HectorHZZZ October 14, 2026 01:10

View session

Copilot AI reviewed Oct 14, 2026

View reviewed changes

vs-code-engineering Bot assigned Ulugbek Abdullaev (ulugbekna) Oct 14, 2026

Benjamin Steenhoek (bstee615) reviewed Oct 19, 2026

View reviewed changes

Joaquín Ruales (jruales) mentioned this pull request Oct 19, 2026

Connector authorization tests fail without default account service stub #342697

Closed

HectorHZZZ marked this pull request as ready for review October 20, 2026 22:32

hangzhao-microsoft and others added 5 commits October 22, 2026 23:05

Ulugbek Abdullaev (ulugbekna) force-pushed the hangzhao/nes-rejection-applied-then-undone branch from 0c58675 to a659184 Compare October 23, 2026 10:57

Ulugbek Abdullaev (ulugbekna) approved these changes Oct 23, 2026

View reviewed changes

Ulugbek Abdullaev (ulugbekna) enabled auto-merge October 23, 2026 11:07

Alex Ross (alexr00) approved these changes Oct 23, 2026

View reviewed changes

Alexandru Dima (alexdima) approved these changes Oct 23, 2026

View reviewed changes

Ulugbek Abdullaev (ulugbekna) merged commit 55e62e3 into microsoft:main Oct 23, 2026
59 checks passed

vs-code-engineering Bot added this to the 1.144.0 milestone Oct 23, 2026

Conversation

HectorHZZZ commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Configuration

Running an experiment

Offline evaluation

Changes

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Benjamin Steenhoek (bstee615) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HectorHZZZ commented Oct 19, 2026

Uh oh!

HectorHZZZ commented Oct 20, 2026

Uh oh!

vs-code-engineering Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📬 CODENOTIFY

Uh oh!

HectorHZZZ commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

HectorHZZZ commented Oct 14, 2026 •

edited

Loading

vs-code-engineering Bot commented Oct 20, 2026 •

edited

Loading