Feature installation order

[chrmarti]

As we are preparing to open up features to community contributions, we found that some features need to install after other features. This issue proposes to add support for soft-dependencies to arrive at a reproducible ordering following these ordering requirements:

A feature can declare a list of other features that should be installed first if they are configured to be installed. (The soft-dependency itself does not add another feature to the list of features to be installed.)

E.g., the NodeJS feature would list the Python feature because it has to install a system version of Python if no version of Python is already installed:

{
    "id": "node",
    "installAfter": [ "devcontainers/features/python" ],
    // ...
}

Features that an install in any order will be installed in a fixed order (e.g., ordered by id) to ensure the result is always the same.

The "features" map in the devcontainer.json will remain a map. See https://github.com/devcontainers/spec/pull/27/files#r869018039 for a discussion.

To allow for progress on user contributable features, we can use a stop-gap measure like an ordered list of features that depend on a specific installation order and either bake that into the CLI at build time or fetch it at runtime. This list will be temporary and documented as such.

/cc Chuck Lantz (@Chuxel) John Keech (@jkeech) Edmundo Gonzalez (@edgonmsft) joshspicer Brigit Murtaugh (@bamurtaugh) Alexandru Dima (@alexdima)

[Chuxel]

This list will be temporary and documented as such.

Christof Marti (@chrmarti) I think we'll still need some way to allow users to specify explicit ordering in the feature object in devcontainer.json. Not every feature will know about all other features even long term - and there's other reasons below. Right now this is impossible, which has been problematic for early adopters.

One way to do that could be to allow "installAfter" to also be explicitly mentioned by users in the object syntax in devcontainer.json as well. e.g.

"features": {
    "common": "latest",
    "docker-in-docker": {
        "installAfter": [ "common" ]
    }
}

This would enable similar ordering to an array with a slight variation on syntax for those that value predictability.

Otherwise, I think we'd likely need an array with a mode that allows you to opt into implicit vs explicit ordering, but I don't think we'd want the array syntax to be temporary if that's the direction we go.

To recap, explicit ordering allows end users to:

1. Have a well-known and predictable outcome as they add more features (given features need to adapt to the image contents). Reproducible image builds is a key best practice for creating container images. An unordered dependency graph is not transparent or predictable over time since execution order influences what each feature does. Adding a new feature to the end of the list will not affect anything above it, while a random order can result in more material changes.
   • Examples include the relationship between the Node and Python features, and the Git from Source feature and most other features. Order can result in different behaviors in installing OS dependencies.
   • Similar problems can occur between features authored by multiple users that have no knowledge of one another – and thus no possible dependency graph.
2. Have a way for a developer to easily add a local script that runs as a part of this sequence, without getting into dependency management or complex conditional logic that adapts to different ordering or image scenarios.
   • Existing examples include installing the preview dev CLI into the Azure CLI, installing more go tools (via go install), adding desktop customization for the desktop-lite script, etc.
   • A local feature with a single script can be dropped into the ordered execution for this to happen.
   • You could in concept have a local feature that tries to ensure ordering with an "installAfter", but what may actually want is "installBefore" in this case since it's repo specific... otherwise any additions could case the feature to happen at a different spot.
3. Simplify image build debugging (and fixing) if something goes wrong – whether related to their own features, 3rd party ones, their Dockerfile, or base image. Reordering can reduce duplicate installation and work around bugs, but devs can also add local features as a workaround to install something missing, edit folder structures / add symlinks, etc at the exact right spot in the execution flow.
   • This is partially important when using features from multiple authors.
   • Reordering would otherwise be impossible. You’d have to abandon the features.
   • Like (2), you could in concept have a local feature that tries to fix it with an "installAfter", but what you'd more likely want is "installBefore"... otherwise any additions could case the fix feature to happen at a different spot.

[joshspicer]

I believe we should provide some supported (permanent) mechanism for allowing an end-user to control feature installation. I think it is preferable that a user does not need to concern themselves with install order, but as a "break glass" solution it should be intuitive how to order when the need arises.

Our "first party" features are written in such a way that most installation orders will work (no errors), although in some situations there is a preferred order. Most of the time the preferred order will eliminate the need for a "soft dependency" for a given feature to be installed twice. We try to optimize our scripts to re-use existing tools if present already in the container, but that isn't going to always be trivial to account for. Today we force our preferred order, regardless of any order specified from the user. We can do this since we have full control over the ecosystem at the moment (aside from a few adventurous preview users). That ordering is declared from this file.

When we factor in community ("third party") feature contributions, we now have significantly less control over how the features are implemented. Additionally, we currently don't have a proposal for a centralized "repository" of features, instead opting for a more "Self-hosted" model. We don't plan to strictly moderate the implementation of features, and thus authors are allowed flexibility in authorship. We are pushing a lot of added responsibility onto community authors to "get it right". Many times features will install fine without a specified order, but in the instances where there is an obvious ordering problem that our tooling did not properly infer, I believe the most useful solution we can provide to users for that problem is allowing a specified order.

Not every feature will know about all other features - and there's other reasons below. Right now this is impossible, which has been problematic for early adopters.

I agree. One solution to this is the array notation (proposed here). I believe our tooling can (and should) implement some "ordering logic" that helps prevent a user from hitting obvious/identifiable error, but as a final solution a user should be given the tools to solve a problem themselves that could result from ordering. I especially like Chuck Lantz (@Chuxel) 's point (2), where users may want to prototype a feature locally or perform a simple extension of already installed features.

The implementation of container features today is inherently ordered due to the programatic injection of Dockerfile RUN lines for each feature. Preventing a user from altering (or at the very least overriding), will create situations where a user may need to fallback to manually editing their Dockerfile. In my view, the fundamental goal of features is to abstract away (for an end-user) the need to have any dockerfile at all in their project, i n lieu of a single .devcontainer.json.

[kieferrm]

The argumentation seems to have two sides

• features are purposefully loosely specified and therefore messy situations will arise
• by default, users should not have to worry about in what sequence features need to be listed

If this observation is correct, it seems to me that the syntax should explicitly express both of those aspects. I.e. the syntax should not suggest to me that by default I do have to worry about the sequence and when I look at a devcontainer file it should immediately be clear to me that it defines order to clean up a messy situation.

Thus, it would make sense to me that

1. features define their dependencies
2. the features of a dev container are defined as a map and are being installed based on 1.
3. the dev container file may contain a property overrideFeatureInstallOrder (or a different name along those lines) that allows a user to specify the installation order, and the value of this property takes precedence over 1.

[chrmarti]

A separate overrideFeatureInstallOrder may only work well if we make sure it is only rarely needed and we consider it a bug on the feature that requires the override (or a feature request on us if the feature cannot address it).

On reproducible image builds: We will compute a fixed order based on the soft-dependencies (using a stable sorting where soft-dependencies do not require an order). We will also need a lockfile to make the build reproducible over time as feature scripts update.

On local feature scripts: We can install features that do not participate in the soft-dependencies graph last. They will not participate either on purpose or by accident, running them last should work in both cases. A local feature can then still choose to participate in the soft-dependencies.

[chrmarti]

My main worry with overrideFeatureInstallOrder is that it would break the feedback loop we would otherwise get from users and feature authors that would allow us to improve the implementation and API. We could show a warning on overrideFeatureInstallOrder in the editors to counter that.

[Chuxel]

The argumentation seems to have two sides

• features are purposefully loosely specified and therefore messy situations will arise
• by default, users should not have to worry about in what sequence features need to be listed

If this observation is correct, it seems to me that the syntax should explicitly express both of those aspects. I.e. the syntax should not suggest to me that by default I do have to worry about the sequence and when I look at a devcontainer file it should immediately be clear to me that it defines order to clean up a messy situation.

Thus, it would make sense to me that

1. features define their dependencies
2. the features of a dev container are defined as a map and are being installed based on 1.
3. the dev container file may contain a property overrideFeatureInstallOrder (or a different name along those lines) that allows a user to specify the installation order, and the value of this property takes precedence over 1.

Kai Maetzel (@kieferrm) 100%. I was thinking of a property inline in the object syntax - but something like an override property would work as well. Strict ordering should not be the default, but possible to do. I'd talked with Craig Peters (@craiglpeters) yesterday on the Codespaces team and we both were leaning towards the object semantic as the default with an override form for the reasons you mention.

My main worry with overrideFeatureInstallOrder is that it would break the feedback loop we would otherwise get from users and feature authors that would allow us to improve the implementation and API. We could show a warning on overrideFeatureInstallOrder in the editors to counter that.

Christof Marti (@chrmarti) I'm pretty confident that if people find out that they need to use this property frequently that we'll get feedback on something being wrong. The idea really is that you shouldn't have to do this.

The core challenge is that, unlike a classic package or module system, we only have complete control of the "packaging" aspect of the overall process. How the installation itself affects other things is dictated by things like apt, npm, pip, nix, etc. So that inherently puts us in a spot where predictability is difficult.

However, I strongly believe we should look for ways to avoid these problems as well, so agree with that point. I also agree with your thought that we should probably consider a way to make user creation something that can be done independent of features since its placement has a dramatic affect on ordering right now.

On reproducible image builds: We will compute a fixed order based on the soft-dependencies (using a stable sorting where soft-dependencies do not require an order). We will also need a lockfile to make the build reproducible over time as feature scripts update.

Yeah the specific issue is when you add one more thing, that can end up in a (to the user) random spot in the ordering and affect things in ways we cannot predict. In primary cases, that won't be a problem, but we'll need a way to handle exceptions. We've seen enough already at this early stage that having some mechanism will help a lot.

On local feature scripts: We can install features that do not participate in the soft-dependencies graph last. They will not participate either on purpose or by accident, running them last should work in both cases. A local feature can then still choose to participate in the soft-dependencies.

This is a good idea - I like it! The one thing it does not solve is where you need that to happen between two features as a fix, but with an override, this should reduce the number of incidents significantly. Love that.

[joshspicer]

The core challenge is that, unlike a classic package or module system, we only have complete control of the "packaging" aspect of the overall process.

I want to emphasize this point made by Chuck. We currently are NOT building a package manager (we have no centralized database of features, there is no lock file, we have no proposal for a strict versioning schema other than "what makes sense" to an author). We have also explored building on top of an existing package manager, but Edmundo Gonzalez (@edgonmsft) 's investigations concluded it did not make sense given the specific case studies we investigated.

We indeed have control over packaging via our provided GitHub action (https://github.com/microsoft/publish-dev-container-features-action, soon to be migrated over to the devcontainers org following a revised packaging schema). But at packaging time, we again only have a view into the immediate set of features - there is no central "features" database we can query to add metadata).

[joshspicer]

The argumentation seems to have two sides

1. features are purposefully loosely specified and therefore messy situations will arise
2. by default, users should not have to worry about in what sequence features need to be listed

I just want to voice that I don't think (2) is necessarily a hard requirement, although that is certainly the way the discussion is heading. I think that our tooling should provide some sortof mechanism for detecting invalid orderings, but installing tools into a Docker image in inherently ordered and thus by us taking the ability to define an order away from a user, we're necessitating moving that complexity onto someone else (feature authors and ourselves as developers of the ordering algorithm).

On local feature scripts: We can install features that do not participate in the soft-dependencies graph last. They will not participate either on purpose or by accident, running them last should work in both cases. A local feature can then still choose to participate in the soft-dependencies.

Agreed, this makes the most sense in a completely opaque ordering model!

Although - if a user wanted to then utilize several "local" features, we run into the exact same problem as if a user wanted to utilize several "remote" features.

Perhaps the core take-away is that we don't have enough user feedback yet on how one wants to utilize and stretch the power we provide with features.

[Chuxel]

I just want to voice that I don't think (2) is necessarily a hard requirement, although that is certainly the way the discussion is heading.

joshspicer I think this is representative of our aspiration. We have a checkbox UX right now with this in mind... pick your base, then add a few optional features and you are done. Practically speaking we've found it is more complicated than just that, but this is also not an absolute either. Anything that's a straight CLI you curl and drops in /usr/local/bin works pretty well this way. We may want to favor that acquisition path - particularly when signing is available to verify contents like there is for the Go install - since it will be less succeptible to the problem.

The core issue that tends to cause things like Go to need ordering is that it needs to set privs to enable a non-root user to access or install contents in folders it creates. Given how common that is, I could totally see us going down the path where remoteUser supports creating said user rather than just referencing like it is now. That would then happen first, and at that point the remaining challenges would be more exception based as Christof Marti (@chrmarti) has pointed out in the past.

In fact, we could pass that user name into the features, which would simplify logic we have now to figure out what that user is. (e.g. it's there as a DEV_CONTAINERS_REMOTE_USER env var). This is not really practical with the script form where many of these scripts started, but with dedicated features, it would work really well and also simplify authoring.

[chrmarti]

Although - if a user wanted to then utilize several "local" features, we run into the exact same problem as if a user wanted to utilize several "remote" features.

I was thinking that a local feature could also use the "installAfter" property to get in control of the order. We just order features without any dependencies or dependents last to increase the chances of success for features that are unaware of their dependencies (and need to be fixed).

[joshspicer]

I want to publicly share an example proposal of Chuck Lantz (@Chuxel) 's shared in a different medium.

There is then a “featureOrder” (or similar) property in devcontainer.json that dictates ordering behaviors. This has three possible values “auto” (the default), “strict”, and “manual”. This is used in conjunction with the array syntax.

In “auto” mode, the dev container CLI will tweak the order in the array based on what is in “runsAfter” for a given feature. Specifically:

For each feature in the array, see if any features listed in “runsAfter” are also in devcontainer.json.

If they are, change the actual ordering to be sure that the current feature runs after each item in this list.

In the event ordering is changed, output a warning that the order was changed in the logs with a note on how to stop this from happening. (e.g., “Change featureOrder to strict to disable this behavior”)

In “manual” mode, the execution order is forced to be exactly what is in the array in devcontainer.json – regardless of whether or not this is the recommended order. If something is out of order based on its “runsAfter” values, a warning is placed in the logs to aid debugging should the build fail unexpectedly due to a feature issue.

In “strict” mode, the execution order is also always based on what is in the array in devcontainer.json. However, if something is out of order based on its “runsAfter” values, the build fails with an explicit error saying what is out of order.

This idea of "modes" auto (default), strict, and manual seems to provide the most power and flexibility to the end-user. Users that want the "magic" experience don't tinker with the featureOrder, and those that identify unsolvable problems (or would prefer to iterate on a feature without worrying about the plethora of dependency models we have discussed), can then change their installation mode.

This seems like a nice compromise. The "magic" dependency detection exploration can continue, while users today are not blocked from using features in a way that works for them.

I expect that users in "auto" mode will have their devcontainers break as the team continues to iterate on automatic feature re-ordering. These "modes" will provide a way for users to unblock themselves and continue to stay in our ecosystem. The alternative would be to leave features altogether in favor of manually writing their Dockerfile.

[chrmarti]

I have put together a PR demonstrating an implementation of the proposal: devcontainers/cli#56

[joshspicer]

Although - if a user wanted to then utilize several "local" features, we run into the exact same problem as if a user wanted to utilize several "remote" features.

I was thinking that a local feature could also use the "installAfter" property to get in control of the order. We just order features without any dependencies or dependents last to increase the chances of success for features that are unaware of their dependencies (and need to be fixed).

Makes sense! In isolation all the local features could then be completely controlled.