EPIC: Allow storage of draft PackageRevision resources when the mutation pipeline is failing

[liamfallon]

It should be possible to store PackageRevision and PackageRevisionResources resources as drafts even if the mutation pipeline is failing. A Propose or Approve should fail. The mutation pipeline status should reflect the mutation pipeline error.

See this discussion: Why is it not possible to create a draft PackageRevision with a failing pipeline? nephio-project/nephio#876
See this PR: Allow creating/updating draft package revisions with a failing pipeline #194

As part of this ,let's try to address how we should handle the render status in the Kptfile
Tried to find all the relevant issues/PRs:
Storage of Draft package revision resources when mutation pipelines fails #409

PIP: Async pipeline executions: #871

[aravindtga]

Feature Analysis

Architecture Analysis

Core Components

The mutation pipeline execution is handled by two primary functions:

1. DoPRResourceMutations (pkg/task/generictaskhandler.go)

   • Handles Draft → Draft transitions via PackageRevisionResources updates
   • Used when updating package contents directly

2. DoPRMutations (pkg/task/generictaskhandler.go)

   • Handles Draft → Proposed transitions via PackageRevision updates
   • Used when changing package lifecycle states

Execution Flow Analysis

Draft → Draft (Resource Updates)

porchctl rpkg push
↓
packagerevisionresources.go (Update function)
↓
engine.go (UpdatePackageResources function)
↓
generictaskhandler.go (DoPRResourceMutations function)
↓
pkg/task/render.go (apply function)

Draft → Proposed (Lifecycle Updates)

porchctl rpkg propose
↓
packagerevision.go (Update function)
↓
packagecommon.go (updatePackageRevision function)
↓
engine.go (UpdatePackageRevision function)
↓
generictaskhandler.go (DoPRMutations function)
↓
pkg/task/render.go (apply function)

Current Behavior Analysis

Failure Handling

Current State: Fail-fast approach

• When mutation pipeline fails, the entire operation terminates immediately
• No status updates are performed
• No changes are pushed to the git repository
• CLI displays error messages to the user

Pipeline Execution Behavior

1. Sequential Processing: Pipeline executes mutations sequentially
2. Immediate Termination: On first failure, execution stops completely
3. No Partial Updates: Remaining mutations are not executed
4. Resource Handling: render.go apply() returns empty PackageResources{} on failure

Low-Level Implementation Details

render.go(apply function) Behavior

• Success Case: Returns processed package resources
• Failure Case: Returns empty repository.PackageResources{} struct
• Error Propagation: Captures error in TaskResult.RenderStatus.Err

executor.go(runMutators) Behavior

• runMutators: Returns immediately on first mutator failure
• runValidators: Returns immediately on first validation failure
• No Continuation: Does not attempt to process remaining functions

Status Update Mechanisms

Available Status Fields

• PackageRevision.Status: Package-level status information
• PackageRevisionResources.Status: Resource-level status with render details

Open Questions & Decisions

1. Partial Pipeline Failure Handling

Decision: Persist what the user modified before the pipeline runs (successful pipeline changes don't need to be considered)

2. State Transition Scope

Decision: Only Draft → Draft transitions should have failure-tolerant behaviour. Draft → Proposed should maintain current fail-fast behaviour

3. Status Update Location

Decision: Mutation failure status should be recorded in PackageRevision.Status

4. Validator Pipeline Behaviour

Decision: Validators should maintain current fail-fast behaviour, no changes needed

[liamfallon]

Comments from SIG-Automation meeting on 2026-01-21:

• Remove RenderStatus from PackageRevisionRseources
• Check if Validation failures can be handled like mutation failures (Resources stored and status updated)
• Output in RenderStatus should be the same as “kpt fn render”
• Should the conditions and/or rendering status be stored in the kpt file? In the async solution, this is stored in the kpt file. As well as or instead of RenderStats on PR?
• We will follow up on a group chat on slack or at a future meeting.
• Also what is the overlap with Allow creating/updating draft package revisions with a failing pipeline #194
• We need to synchronize the order in which we drop functionality into kpt and porch

[aravindtga]

Feature comparison between #194 and #409

1. Architectural Patterns

PR-194: Kpt File based Architecture

┌─────────────────────────────────────────────────────────────┐
│                    User Updates Package                     │
└────────────────────────┬────────────────────────────────────┘
                         ↓
┌─────────────────────────────────────────────────────────────┐
│              Render Pipeline Executes                       │
│  • Mutators run                                             │
│  • Validators run                                           │
└────────────────────────┬────────────────────────────────────┘
                         ↓
                    [Success?]
                    /         \
                 Yes           No
                  ↓             ↓
    ┌─────────────────┐  ┌──────────────────────┐
    | Kptfile-Status  |  | Kptfile-Status       |
    │ Rendered=True   │  │ Rendered=False       │
    │ Reason=Rendered │  │ Reason=RenderFailed  │
    │ Message=Success │  │ Message=Error+JSON   │
    └────────┬────────┘  └──────────┬───────────┘
             │                       │
             └───────────┬───────────┘
                         ↓
         ┌───────────────────────────────┐
         │  Write Condition to Kptfile   │
         │  (Direct modification)        │
         └───────────────┬───────────────┘
                         ↓
         ┌───────────────────────────────┐
         │  Push Package to Git          │
         │  (Kptfile included)           │
         └───────────────┬───────────────┘
                         ↓
         ┌───────────────────────────────┐
         │  User Attempts Lifecycle      │
         │  Transition (Propose/Publish) │
         └───────────────┬───────────────┘
                         ↓
         ┌───────────────────────────────┐
         │  Engine Reads Kptfile         │
         │  Checks Rendered Condition    │
         └───────────────┬───────────────┘
                         ↓
                    [Rendered=True?]
                    /              \
                 Yes                No
                  ↓                  ↓
         ┌────────────────┐  ┌──────────────────┐
         │Allow Transition│  │ Reject Transition│
         └────────────────┘  └──────────────────┘

---

PR-409: CRD based Architecture

┌─────────────────────────────────────────────────────────────┐
│                    User Updates Package                     │
└────────────────────────┬────────────────────────────────────┘
                         ↓
┌─────────────────────────────────────────────────────────────┐
│              Render Pipeline Executes                       │
│  • Mutators run                                             │
│  • Validators run                                           │
└────────────────────────┬────────────────────────────────────┘
                         ↓
                    [Success?]
                    /         \
                 Yes           No
                  ↓             ↓
    ┌─────────────────┐  ┌──────────────────────┐
    │ Push Rendered   │  │ Push UNRENDERED      │
    │ Resources       │  │ Resources (original) │
    └────────┬────────┘  └──────────┬───────────┘
             │                       │
             └───────────┬───────────┘
                         ↓
         ┌───────────────────────────────────────┐
         │  Return RenderStatus to Engine        │
         │  (In-memory object)                   │
         └───────────────┬───────────────────────┘
                         ↓
         ┌───────────────────────────────────────┐
         │  Engine Returns to Registry Layer     │
         │  (renderStatus passed up)             │
         └───────────────┬───────────────────────┘
                         ↓
         ┌───────────────────────────────────────┐
         │  Registry Persists to CRD             │
         │  PackageRevision.status.renderStatus  │
         └───────────────┬───────────────────────┘
                         ↓
         ┌───────────────────────────────────────┐
         │  Persistence                          │
         │  • CR Cache                           │
         │  • DB Cache                           │
         └───────────────┬───────────────────────┘
                         ↓
         ┌───────────────────────────────────────┐
         │  User Attempts Lifecycle Transition   │
         │  (Propose/Publish)                    │
         └───────────────┬───────────────────────┘
                         ↓
         ┌───────────────────────────────────────┐
         │  Transition fails due to render       │
         │             failure                   │
         └───────────────────────────────────────┘

---

2. Potential Hybrid Approach

Core Principle

RenderStatus should be stored in the Kptfile as it is intrinsically tied to the package's rendering pipeline.

Key Design Decisions

1. Enhanced Per-Function Status Tracking

Current Limitation (PR-194):

• Only records overall pipeline success/failure in Kptfile.status.conditions
• When a function fails, no visibility into which functions succeeded/executed

Example Scenario:

pipeline:
  mutators:
    - image: set-namespace    # Function 1
    - image: set-labels       # Function 2 - FAILS HERE
    - image: apply-setters    # Function 3
    - image: set-annotations  # Function 4
  validators:
    - image: gatekeeper       # Function 5
    - image: kubeval          # Function 6

If Function 2 fails, current approach only shows:

• Overall: Rendered=False
• Function 1 status: Unknown
• Function 2 status: Failed
• Functions 3-6 status: Unknown (never executed)

Proposed Enhancement:

Option 1:

Record individual function results in Kptfile, with consolidated status in status.conditions.

Backward compatibility should be considered with this approach. If the spec changes are not suitable means, including the itemized results in the status conditions field can be considered.

pipeline:
  mutators:
    - image: ghcr.io/kptdev/krm-functions-catalog/set-namespace:v0.4.1
      configMap:
        namespace: demo-namespace
      exitCode: 0
      results:
        - message: all namespaces are already "demo-namespace". no value changed
          severity: info
    - image: ghcr.io/kptdev/krm-functions-catalog/set-labels:v0.2.0
      configMap:
        env: testing
        managed-by: porch
        team: platform
      exitCode: 0
      results:
        - message: set 18 labels in total
          severity: info
    - image: ghcr.io/kptdev/krm-functions-catalog/set-annotations1:v0.1.4
      configMap:
        app.kubernetes.io/managed-by: porch
        config.kubernetes.io/managed-by: kpt
      exitCode: 125
      results:
        - message: |-
            docker: Error response from daemon: failed to resolve reference "ghcr.
            io/kptdev/krm-functions-catalog/set-annotations1:v0.1.4": ghcr.
            io/kptdev/krm-functions-catalog/set-annotations1:v0.1.4: not found

            Run 'docker run --help' for more information
          severity: error
status:
  conditions:
  - type: Rendered
    status: "False"
    reason: MutationFailed
    message: "set-annotations1 field failed: validation error"

Option 2:
Follow the same pattern as nephio-project/nephio#194

status:
  conditions:
  - type: Rendered
    status: "False"
    message: |-
      fn.render: pkg /:
      	pkg.render:
      	pipeline.run: func eval "ghcr.io/kptdev/krm-functions-catalog/set-annotations123:v0.1.4"
        failed: rpc error: code = Unknown desc = unable to get the grpc client to the pod for ghcr.
        io/kptdev/krm-functions-catalog/set-annotations123:v0.1.4: unable to get the entrypoint for ghcr.
        io/kptdev/krm-functions-catalog/set-annotations123:v0.1.4: GET https://ghcr.io/token?
        scope=repository%3Akptdev%2Fkrm-functions-catalog%2Fset-annotations123%3Apull&service=ghcr.io: DENIED:
        requested access to the resource is denied{"result":{"kind":"FunctionResultList","apiVersion":"kpt.
        dev/v1","metadata":{"name":"fnresults","creationTimestamp":null},"exitCode":1,"items":[{"image":"ghcr.
        io/kptdev/krm-functions-catalog/set-namespace:v0.4.1","exitCode":0,"results":[{"message":"namespace
        \"demo-namespace\" updated to \"demo-namespace1\", 3 value(s) changed","severity":"info"}]},
        {"image":"ghcr.io/kptdev/krm-functions-catalog/set-labels:v0.2.0","exitCode":0,"results":[
        {"message":"set 18 labels in total","severity":"info"}]},{"image":"ghcr.
        io/kptdev/krm-functions-catalog/set-annotations123:v0.1.4","exitCode":1}]},"error":"fn.render: pkg
        /:\n\tpkg.render:\n\tpipeline.run: func eval \"ghcr.
        io/kptdev/krm-functions-catalog/set-annotations123:v0.1.4\" failed: rpc error: code = Unknown desc =
        unable to get the grpc client to the pod for ghcr.
        io/kptdev/krm-functions-catalog/set-annotations123:v0.1.4: unable to get the entrypoint for ghcr.
        io/kptdev/krm-functions-catalog/set-annotations123:v0.1.4: GET https://ghcr.io/token?
        scope=repository%3Akptdev%2Fkrm-functions-catalog%2Fset-annotations123%3Apull\u0026service=ghcr.io:
        DENIED: requested access to the resource is denied"}
    reason: RenderFailed

Option 3:
Follow the same pattern as nephio-project/nephio#194 and include the individual function results in status.conditions.

status:
   conditions:
    - type: Rendered
      status: "False"
      reason: MutationFailed
      message: "set-annotations1 field failed: validation error"
      functionResults:
        - name: set-namespace
          exitCode: 0
          results:
            - message: all namespaces are already "demo-namespace". no value changed
              severity: info
        - name: set-annotation
          exitCode: 1
          results:
            - message: |-
                docker: Error response from daemon: failed to resolve reference "ghcr.
                io/kptdev/krm-functions-catalog/set-annotations1:v0.1.4": ghcr.
                io/kptdev/krm-functions-catalog/set-annotations1:v0.1.4: not found

                Run 'docker run --help' for more information
             severity: error

---

2. Resource Push Strategy on Render Failure

Decision: Push draft packages to repository even when render fails

Current Status:

• DB-cache implementation: Complete (flag-based, PR available and not merged yet)
• CR-cache implementation: Planned (Default behaviour)

Option 1:

• Use the same flag/Introduce new flag to control the CR cache behaviour as same as the DB-Cache one.

Option 2:

• Remove the db-cache flag based draft storage approach to have storing the draft as default behaviour for both implementations.

---

3. RenderStatus Storage in Kpt Repository

Decision: Implement renderStatus storage in kpt repository, consumed by Porch

Architecture:

Kpt Repository (upstream)
    ↓
  Implements: Kptfile.status.renderStatus/conditions schema
    ↓
Porch (downstream)
    ↓
  Consumes: Reads renderStatus/conditions from Kptfile

• This also gives a clear error message when the pipeline fails as it is read directly from the Kptfile. Avoid the grpc based execution details on the render failures.

---

4. RenderStatus Storage in Porch APIs

Current State Analysis:

API Resource	Current RenderStatus Field	Populated?
PackageRevision	❌ None	N/A
PackageRevisionResources	✅ Has field	❌ No
PackageVariant	✅ Has field	✅ Yes

Proposed Solution:

Option 1: Add renderStatus to PackageRevision - (new status field) - This is the current structure in Packagevariant and packagerevisionresources

apiVersion: porch.kpt.dev/v1alpha1
kind: PackageRevision
status:
  renderStatus:              
    result:
      exitCode: 1
      items: [...]
    error: "render failed"

Option 2: Add status.Conditions to PackageRevision - As same as kptfile

apiVersion: porch.kpt.dev/v1alpha1  
kind: PackageRevision
status:
  conditions:              
    ...

Option 3: Update the PRR renderstatus alone and provide on-failure note to get PRR for more details about the failure.

apiVersion: porch.kpt.dev/v1alpha1
kind: PackageRevisionResources
status:
  renderStatus:              
    result:
      exitCode: 1
      items: [...]
    error: "render failed"

• Decision needs to be taken on which CR's are going to have the renderStatus/status.conditions field.

---

[aravindtga]

Here are the design decisions based on the discussion.

Please comment if there is anything needs to be changed/discussed. @liamfallon @efiacor @mozesl-nokia @rendre-greyling

1. Enhanced Per-Function Status Tracking

Decision: Implement detailed function-level status tracking in Kptfile

Structure:

status:
  conditions:
    - type: Rendered
      status: "False"
      reason: MutationFailed
      message: "set-annotations1 failed: validation error"
  functionResults:
    - name: set-namespace
      exitCode: 0
      results:
        - message: "namespace updated successfully"
          severity: info
    - name: set-annotations1
      exitCode: 1
      results:
        - message: "image not found"
          severity: error

Key Points:

• status.functionResults is a sibling to status.conditions (not nested within conditions)
• Provides visibility into which functions succeeded/failed during pipeline execution
• No spec changes - all modifications in status section only

---

2. Resource Push Strategy on Render Failure

Decision: Introduce new --push-on-failure flag with three options

Flag Options:

• None (default): Package is not pushed on render failure
• Unrendered: Push the initial/original package before rendering
• PartiallyRendered: Push the package at the state where rendering failed

Implementation Notes:

• This flag will be implemented in kptdev/kpt repository (may be with different flag name)
• Replaces the existing DB-cache flag
• Porch will consume this flag from kpt

---

3. RenderStatus Storage in Kpt Repository

Decision: Implement renderStatus storage in kpt repository, consumed by Porch

Architecture:

Kpt Repository (upstream)
  ↓ Implements
Kptfile with status.conditions + status.functionResults
  ↓ Consumed by
Porch (downstream)

---

4. RenderStatus Storage in Porch APIs

Decision: Add status.conditions and status.functionResults to PackageRevision

Per-Resource Behaviour:

Resource	Status Fields	Persistence
PackageRevision	status.conditions + status.functionResults	Persistent in CR
PackageRevisionResources	Existing status.renderStatus	Read from Kptfile (not persistent)
PackageVariant	Existing status.renderStatus	No changes

Structure for PackageRevision:

apiVersion: porch.kpt.dev/v1alpha1
kind: PackageRevision
status:
  conditions:
    - type: Rendered
      status: "False"
      reason: MutationFailed
      message: "..."
  functionResults:
    - name: ...
      exitCode: ...
      results: [...]

---