Should the project's package file support direct dependencies only?

[agross]

Currently there's the idea to use a project-specific packages file to define the subset of package IDs that paket needs to reference in the respective project. This is to have any number of dependencies in the packages.fsx but only reference the test-specific subset among these for test projects.

Definition direct dependency: A dependency that is declared in packages.fsx
Definition indirect dependency: A dependency of a direct dependency. Indirect dependencies might change with future releases of direct dependencies (adding or removing indirect dependencies).

Both direct and indirect dependencies and their calculated versions will to be downloaded to <packages.fsx location>/packages/ and saved to packages.lock.

My question is whether it's a good idea to restrict the dependencies that are allowed in the project-specific packages file to direct dependencies.

Now on to the discussion, assuming there is no packages.lock (see below why this is a good idea):

Restricting packages to direct dependencies

Pros

• The user must be very explicit what packages all of the projects need
• Future package updates of direct dependencies removing indirect dependencies won't cause any harm. Paket will still download them according to packages.fsx, thus making them available.

Cons

• If the user requires an indirect dependency that is downloaded through virtue of the fact that a direct dependency requires it, the indirect dependency would have to become a direct dependency for it to be allowed in the project-specific packages file. I.e. both the direct and the indirect dependencies need to be put in packages.fsx.

Allowing direct and indirect dependencies

Pros

• The packages.fsx might become a bit shorter, listing only high-level direct dependencies.

Cons

• Future updates of direct dependency removing indirect dependencies will cause paket to not download it, thereby break the build. The user would then have to update packages.fsx to make the previously-indirect dependency a direct dependency.

Why not having a lockfile sometimes is a good idea

In Ruby (with bundler) there is the notion that library developers should not check in their Gemfile.lock files (= packages.lock). The reason is that reproducible builds are generally not that much important for libraries and that you rather want to know whether the library still works with the latest dependencies according to the version constrains in the Gemfile (= packages.fsx).

This makes it very easy to just rerun bundle install and your test suite (even in CI) from time to time without the need to change any files.

I think for .NET libraries the same pattern may emerge, with services like https://gemnasium.com/ that check your libraries' dependencies weekly and notify you about (semver) breaking changes in your libraries' dependency set.

For applications, on the other hand, the situation is a bit different. For them you want reproducible builds, regardless of dependency updates (according to `packages.fsx´). This is where the lockfile comes into play. It records the fact that you proved your app works against a specific set of dependencies in specific versions. Using the lockfile enables you to recreate the same runtime environment over and over again.

You can read more about the way Ruby handles this here:
http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/
http://bundler.io/v1.3/rationale.html#checking-your-code-into-version-control

I hope I could get my point across. Looking forward to your feedback!

[agross]

BTW, bundler allows referencing indirect dependencies. I'd say with great power comes great responsibility.

[forki]

I think we make it optional to add indirect dependencies to the packages file.

Will implement this. Would also be good to put this question into the docs

[ilkerde]

I opt for allowing direct and indirect dependencies.

I think it is fairly ok to keep direct deps in packages.fsx and have the graph frozen in lock file. I would love to see that Paket is smart enough to provide me a check whether a project starts to reference a yet indirect reference and possibly even lets me elevate it to a direct one.

In the end, Paket would have two sets at hand: with packages.fsx a great summary of what really is demanded (logically associated), plus with all packages.xmls a list of what is needed to run it.

[forki]

@ilkerde

First of all we changed our naming a bit. We now have package.list files. They replace the old packages.config files which are next to csproj files. See https://github.com/fsprojects/Paket/blob/master/src/Paket/packages.list for sample.

At the moment we support both, direct and indirect package names in this file. If you only specify the direct ones, then Paket will figure it out for you.

[ilkerde]

@forki I'm obviously not up to the pace you are on. I don't have an idea on how packages.fsx relates to packages.list. If I assume that packages.list is a substitute of packages.config, I would prefer to list all packages (both direct and indirect ones) in packages.list, without giving the "user" the option to omit them. I have no good feeling at making Paket part of the overall dependencies needed to get an application running. But this might equally well be my tendency on conservative dependency paranoia.

[forki]

Paket is not needed to get the app running. Only to update dependency definitions.
Yes packages.list is a replacement for packages.config - but for compilation only the csproj files are needed.
Paket install can modify your csproj files according to packages.lock and packages.list.

Will update docs tomorrow

[ilkerde]

@forki Thanks for pointing that out. I am well aware that the references are in *proj files. The case here is that if you only refer to direct dependencies in packages.list. Hence the clarity and consistency is gone (from my pov), since it diverges from *proj content. In case the packages are not in place, I would need Paket to restore everything for me. Just the very same as Nuget.

My point is, that John Average (that's me) may wonder why there are so many (missing) references in project, will have a look at the packages.list file and even wonder more why there are so few packages listed.

[forki]

/cc @agross - I'm not really sure what we should do here
On Sep 4, 2014 11:06 PM, "ilkerde" notifications@github.com wrote:

@forki https://github.com/forki Thanks for pointing that out. I am well
aware that the references are in *proj files. The case here is that if you
only refer to direct dependencies in packages.list. Hence the clarity and
consistency is gone (from my pov), since it diverges from *proj content. In
case the packages are not in place, I would need Paket to restore
everything for me. Just the very same as Nuget.

My point is, that John Average (that's me) may wonder why there are so
many (missing) references in project, will have a look at the
packages.list file and even wonder more why there are so few packages
listed.

—
Reply to this email directly or view it on GitHub
#38 (comment).

[agross]

Well, we've talked about it. There are two options, being explicit about all project dependencies in packages.list or let the tool do the heavy lifting and compute indirect dependencies for the direct ones we specify in packages.list.

Since we don't tell paket about indirect dependencies in packages.fsx, why should we be (IMHO) overly explicit in packages.list? My two cents.

If @ilkerde is favoring to be explicit in packages.list, paket won't stop him from doing so. I think this model serves us both very well.

[ilkerde]

Fair enough. Does it mean then that I can configure Paket to be "strict" or "loose"?

[ilkerde]

@agross @forki The longer I think about it, the more doubtful I get about omitting indirect dependencies in References.list. As far as I understood, we aim to replace packages.config with References.list (in the long run). IMHO, this alone would satisfy to strictly go for all dependencies instead of only having the direct ones listed.

The main reason is simply sanity. A (quasi) mirror of dependencies in References.list would enable us to perform a few nice sanity checks on .proj files. If a reference is deleted from .proj, we beg the user to decide whether he really wants to ditch it (thereby removing from References.list as well) or restore his faux-pas. In a transition from Nuget scenario (might be interesting), we can check for integrity of packages.config <-> References.list <-> .proj. Speaking of inter-proj-level checks, we can leverage the full Referenes.list for a few consistency checks offline without calling out to package sources to resolve indirect dependencies.

@agross I know you feel fine with the laissez-faire model to just let the user decide what's in References.list. However, I want you to please rethink and judge again. I think it is a quite important decision.

[forki]

@agross came up with a setting in the Paket. dependencies file:

mode "strict"

Or implicit:

mode "loose"

I assume we would also have to transport it to the lockfile

[agross]

Another option: references "strict". Makes it clear(er) what the setting is about.