generate checksums for all Components files [$15]

[btkostner]

It would be amazing if aptly would generate AppStream data when it publishes a repository. This would allow packages in the published repo to be seen by new-age app centers, including gnome software.

Here is a generator written in D for reference.

[ximion]

As long as Aptly reads the dep11/ directory and puts the correct checksums for Components-* into the InRelease file, asgen and Aptly should work well together.
I would advise against writing an own metadata generator, since that is a huge pain, and keeping it up-to-date is an issue. But making asgen easier to use from other tools would be possible (at time, it is mainly built for large distro repositories and to be a standalone tool).

[btkostner]

I'm changing this issue's scope to copy files and generate checksums for all Components files. This will be the easiest solution to this issues, and some other issues related to metadata generation.

Currently:

• asgen will output a directory similar to data/xenial/main/Components-amd64.yml.gz

Expected behavior:

• On publish aptly copies folder so we have a directory similar to public/stable/dists/xenial/main/Components-amd64.yml.gz
• Aptly includes checksums for all Components files in /aptly/public/stable/dists/xenial/main

[btkostner]

I also put a $15 bounty on this for anyone who want's to add it:
https://www.bountysource.com/issues/33086807-generate-checksums-for-all-components-files

[ximion]

Just something to add before it gets forgotten: The same thing needs to be done with the icons-* tarballs in the same dep11 subdirectory.
Also, the checksums for the uncompressed files need to be in InRelease, but the actual uncompressed files need not (and probably should not) be present.

[danirabbit]

Bumped the bounty to a total of $50 :)

[hsitter]

I have been looking into this, alas, it is a bit tricky. I'd absolutely love some input on this @smira

We need a bunch of information which could be different per source/publishedrepo/usecase.
Namely the following appstream-generator attributes:

• UseIconTheme (overrides icon to use)
• MediaBaseURL (fully qualified URI to a media directory)
• HTMLBaseURL (URL to a html directory, if I am not mistaken this can actually be relative)
• DataPriority (allows overriding of other appstream data)
  The other attributes we can pretty much fill with information we already have in aptly anyway.

With that in mind I am currently thinking about the following changes to make this happen:

api/publish.go for both initial publishing and updating gets a new attribute AppStream: {} with the aforementioned attributes for appstream-generator.
Final payload should be something like

{
  "Prefix": "floof",
  "SourceKind": "local",
  "Sources": [{"Name": "local-repo"}],
  "AppStream": {
    "MediaBaseURL": "http://floof.com/media",
    "HTMLBaseURL": "http://floof.com/html",
    "IconTheme": "breeze",
    "DataPriority": 1
  }
}

We probably ought to mangle this information internally a bit more. MediaBaseURL for example should probably be expanded to be "prefix-unique" by appending the actual publishing prefix, so what we give to asgen would be $MediaBaseURL/$Prefix -> http://floof.com/media/floof.

deb/index_files.go gets a new Dep11Asset type, this type is used for all files we want to dump in dist/$component/dep11/ (namely icons-* and Components-*). Ideally this should be only an addition. Which is a bit meh because index_files does assume the bufio.Writer gets decompressed data, while appstream gives us compressed data.

To that end utils/compress.go should probably get a new helper to decompress a file and load it into memory. This should work for starters but will ramp up memory consumption with appstream's icons.tar, so long-term refactoring of index_files to not assume uncompressed data would probably be good.

deb/publish.go probably gets a new argument for the AppStream options we got via the API, it gets a new loopedy loop which does:

• construct a Config for asgen using available information
• adds options obtained via API
• writes that into a path under $AptlyRoot/asgen/$Prefix
• runs asgen
• foreach export/data/$Dist/$Component/*
  • decompresses file into memory
  • constructs Dep11Asset
  • writes decompressed data into bufio of the asset
  • (the assets get finalized alongside the rest of the index_files stuff) at which point they get compressed again

This should give us a largely working lineup for local repositories. How to deal with remote publishing endpoints I am not really sure to be honest.

All of this also assumes that some other piece of software grabs $AptlyRoot/asgen/$Prefix/export/media (i.e. the screenshots) to publish them under the URL we were provided.

We will probably also need some global .aptly.conf option to configure an appstream-generator run string so that one can "intercept" and redirect the appstream-generator run (e.g. to run in a container, or on a different host). Running appstream-generator on the same system as aptly is a bit meh from a security POV.