Add support for backports repos

[geofft]

If -backports=true is provided to any of the publish subcommands, the
published repo will have the lines

NotAutomatic: yes
ButAutomaticUpgrades: yes

in its Release file, which is necessary when creating or mirroring a
backports repo -- the repo should be available, but apt should not
choose packages in that repo by default. This flag can be later unset
with -backports=false.

This isn't a complete fix for #184 since we don't capture all the
fields, but it solves the most common use case.

[geofft]

The test failures appear to be because squeeze and squeeze-lts don't exist any more as of a few days ago....

Also I guess I should add test coverage and documentation, but Go is new to me and I don't quite follow the test framework.

[geofft]

Hey @smira, is this a reasonable approach to solving this? Would you like me to write docs and try to figure out how to write tests?

[smira]

@geofft docs is something which is missing.

I should have fixed all the squeeze failures in system tests.

In short, you can launch whole test suite by running:

 make system-test

This would do one-time pull for fixtures and start test suite. It might take a long time to run.

So once you've pulled fixtures, you can re-build aptly with:

 gom build -o $GOPATH/bin/aptly main.go

And then you can do:

 system/run.py *Publish*

To run all the publish-related tests.

Also you can ask test runner to re-capture expected output on failures with:

system/run.py --capture *Publish*

[smira]

I can take a look at system tests this week, I'm still trying to keep up with my pending queue :(

[sandrotosi]

I think this is a suboptimal solution. What aptly should do is to stop stripping NotAutomatic/ButAutomaticUpgrades fields from the remote repo Release file, and thus preserve them throughout the mirror/snapshot/publish chain.

with the approach proposed in this PR we need to somehow identify if we are dealing with a backport repo (grep the mirror url? code that in the mirror name? what about if a third-party repo is configured as a backport but doesnt explicitly call the repo '.backport.'?) and pass the backport=true option, which is difficult/cumbersome to script/automate and would result in manual intervention.

the even better approach for aptly would be to preserve all the extra fields from the remote Release file, except those that needs to change of course (like date, hashes, etc)

[geofft]

Yeah, that was the approach I initially wanted to do. The trouble is that a publish doesn't necessarily map to a single mirror. You can imagine doing something like

$ aptly mirror create main http://ftp.debian.org/debian/ jessie main
$ aptly mirror create bpo http://ftp.debian.org/debian/ jessie-backports main
$ aptly mirror update main; aptly mirror update bpo
$ aptly snapshot create main-20160405 from mirror main
$ aptly snapshot create bpo-20160405 from mirror bpo
$ aptly publish snapshot -component=main,bpo main-20160405 bpo-20160405 jessie

which converts two upstream Release files (/debian/dists/jessie/Release and /debian/dists/jessie-backports/Release) into one aptly-generated Release file (/dists/jessie/Release). What should it do in that case? And this isn't actually that weird; if you create the bpo mirror with -filter for the packages you want out of backports, it's entirely reasonable to combine them into one repo that does not have NotAutomatic: yes.

If there's a good way to handle this case, I'm happy to implement it. Maybe the best option is to give an error message if the Release files don't match in their extra configuration, and allow you to specify something on the command line (e.g. aptly publish snapshot -follow-release-file-from=main -component=main,bpo ....) I'm open to suggestions.

What other options are there in Release files besides NotAutomatic and ButAutomaticUpgrades? It'd be good to know for testing.

[sandrotosi]

On Tue, Apr 5, 2016 at 3:54 PM, Geoffrey Thomas notifications@github.com
wrote:

Yeah, that was the approach I initially wanted to do. The trouble is that
a publish doesn't necessarily map to a single mirror. You can imagine doing
something like

$ aptly mirror create main http://ftp.debian.org/debian/ jessie main
$ aptly mirror create bpo http://ftp.debian.org/debian/ jessie-backports main
$ aptly mirror update main; aptly mirror update bpo
$ aptly snapshot create main-20160405 from mirror main
$ aptly snapshot create bpo-20160405 from mirror bpo
$ aptly publish snapshot -component=main,bpo main-20160405 bpo-20160405 jessie

which converts two upstream Release files (/debian/dists/jessie/Release
and /debian/dists/jessie-backports/Release) into one aptly-generated
Release file (/dists/jessie/Release). What should it do in that case? And
this isn't actually that weird; if you create the bpo mirror with -filter
for the packages you want out of backports, it's entirely reasonable to
combine them into one repo that does not have NotAutomatic: yes.

If there's a good way to handle this case, I'm happy to implement it.
Maybe the best option is to give an error message if the Release files
don't match in their extra configuration, and allow you to specify
something on the command line (e.g. aptly publish snapshot
-follow-release-file-from=main -component=main,bpo ....) I'm open to
suggestions.

you could

• merge the options (in case of conflicting options, you might want to
  error & stop)
• take the intersection (so only the options present in all the release
  files)
• throw an error and halt the operation because maybe you are doing
  something you really shouldnt

also this is a rather corner case, and while we have at hand a real users
problem (where many more have complained about it) i doubt this will be a
widespread issue, which could be resolved with a local repo for example.

What other options are there in Release files besides NotAutomatic and
ButAutomaticUpgrades? It'd be good to know for testing.

you can look it up by yourself i guess

[geofft]

I agree that it's a corner case, but I need to implement something for this case. (And meanwhile my code works, and nobody else has proposed another PR for solving this.) I think either merging or taking the intersection are both wrong if done silently.

What options are you interested in? My use case, and as far as I can tell, the use case of everyone else in #184, is just backports. From https://wiki.debian.org/RepositoryFormat , the other options include

• No-Support-for-Architecture-all, "An optional field with a temporary and very specific usecase", and specific to how the Contents files are being generated; since aptly regenerates them, we shouldn't copy them
• Valid-Until: relevant to signing, probably should be managed by aptly instead of copied from the mirror
• NotAutomatic and ButAutomaticUpgrades
• Acquire-By-Hash: definitely should be managed by aptly instead of copied from the mirror

so I think it actually does not make sense to copy unknown fields from the Release file, because all of them other than NotAutomatic / ButAutomaticUpgrades have meanings that don't make sense for aptly mirrors. I'll try to implement something to pick up these two flags, though.

[sandrotosi]

What options are you interested in? My use case, and as far as I can tell,
the use case of everyone else in #184
https://github.com/smira/aptly/issues/184, is just backports.

the only option we are really interested in is to preserve NotAutomatic /
ButAutomaticUpgrades if the remote repository contains them. all the other
options could remain unimplemented/undefined and addresses as someone comes
up with the requirement for it (as done in this case) and do not need to be
addressed now

[smira]

What we could do is:

• allow to specify any key/value pairs which would become part of Release file after publishing
• automatically inherit those key/value pairs if publish is happening on a snapshot tree which has only one mirror as root source (inheriting them from mirror's Release file).

Thoughts?

[sandrotosi]

On Thu, Apr 7, 2016 at 10:52 AM, Andrey Smirnov notifications@github.com
wrote:

What we could do is:

• allow to specify any key/value pairs which would become part of
  Release file after publishing
• automatically inherit those key/value pairs if publish is happening
  on a snapshot tree which has only one mirror as root source (inheriting
  them from mirror's Release file).

1. would resolve the immediate issue with backports and 1. would allow for
   more complex scenarios - like it

[jlu5]

Hi everyone,
It's been nearly a year since the last comment. Is there any more progress on this?

[linuxluser]

Currently, I work around this by using the -skip-signing flag when publishing my jessie-backports mirror, modify the Release file to include the special fields, and then use gpg --sign ... manually on the Release file. It works but is not ideal.

It is true that there needs to be a better way, overall, to have better control over any fields, however this is a pretty important case that I think deserves a flag.

Better yet, make the new -backports flag default to 'true' if the upstream Release file contains both NotAutomatic: yes and ButAutomaticUpgrades: yes, if and only if it is not manually added and set the 'false' by the user. This way, the expected thing is the default thing.

[geofft]

This has been superseded by #577 - thanks, @sliverc!