Preparing module META6.json information for the future

[lizmat]

A long time ago Synopsis 22 (S22) served as the basis of the implementation of the current Raku ecosystem.

Most of its elements were implemented on zef and several authoring tools, such as App::Mi6, and others were not.

The coming years reproducibility and accountability will become much more important, especially in production environments: e.g. dependencies will need to be known at all stages of building and packaging processes. And all of these types of information will need to be available in a common format such as SBOMs, especially in Europe with the Cyber Resilience Act.

So I think it is time to revisit S22 and come to a new accepted standard for Raku modules that are made available for general consumption.

The result of this issue should be a Pull Request defining this new standard.

[lizmat]

Things I think will need to be addressed:

• all references to modules must have at least an :auth<> specification, so that there can be no confusion about which module is going to be required (and thus allowing a rogue upload of a module with a higher version number to usurp the intended functionality).
• only allowable "auth" types to be "cpan:" and "zef:": any other auth types (such as "github:") should not be allowed, as there is no guarantee on the validity of the source, or dependability on what constitutes a "version".
• the support for build-depends and `test-depends" should be deprecated and signalled as such by authoring tools.
• the extended depends section should also allow for a "author" stage dependency specification (apart from the "build", "test" and "runtime" stages).
• the "support" section should check for validity of all the data given as much as possible (e.g. is the email address valid).
• some sort of security contact needs to be specified, possibly in the "support" section, but perhaps in entirely different section of its own.
• some way to express what kind of support can be expected from the maintainers of the module.
• some way to express the intended purpose of the module.
• a way to express where external SBOMs for the distribution can be obtained.
• a way to express what SBOMs are provided by the distribution itself.

[lizmat]

Related: with the lack of having anything similar to CPAN Testers in Raku, the next best thing are blin output runs. The results of a blin run of a specific module should also be referable by a deducible URL, and possibly mentioned in the "support" section, or maybe in another section. Or is should live in the SBOM generation logic, of which a first stab has been done in SBOM::Raku.

[sjn]

• only allowable "auth" types to be "cpan:" and "zef:": any other auth types (such as "github:") should not be allowed, as there is no guarantee on the validity of the source, or dependability on what constitutes a "version".

At some point, we may expect that the provenance and pedigree of an artifact may be verifiable, at which point authorities like github: should be safe to use.

While this isn't the case today, I can imagine a time when the Raku tooling supports in-toto, sigstore, etc.. I think then, it would be nice to allow these new (safe, verifiable) sources 🙂

[sjn]

some way to express what kind of support can be expected from the maintainers of the module.

If someone wants to join the effort in creating a SUPPORT.json specification (in addition to a CONTRIBUTE.json), then please reach out to me! There's an ongoing effort on this topic in OWASP CycloneDX OSS Sustainability WG and eventually ECMA TC54 (TBA ultimo August 2025) where I'm involved.

[sjn]

I've found that some of the terms used in the field of package and supply-chain metadata can be somewhat confusing and inconsistent (take a look at the CPANSec Glossary for a taste of this).

Would it make sense to clean up and/or disambiguate the Raku vocabulary too?

e.g.

• Dependency - A component (module, executable, or service) that is used for the software to function during runtime. Has a single specific artifact, distribution, authority and version associated with it. Some dependencies are only used during certain phases in the component lifecycle. (See also Dependency)
  • author
  • build
  • test
  • runtime
• Requirement - a component that needs to be provided (through dependency resolution, downloading, building and/or installing, etc.) before the software can work as expected. Has a specific structure in order to conform well with expectations from build, test and deployment/packaging tooling.
• Distribution - the archive file that contains the modules, code, tests, metadata and other information required to build, test and install the provided modules.
• Provide - the Modules that a Distribution makes available after successfully building and installing it.
• Component - a single software entity that may be used or depended on by another component or application.
• Artifact - A zip file, tarball or other single file used to share a distribution.
• Pedigree
• Provenance
• …

(lots more to discuss and add here! 😅)

[finanalyst]

The contents of the META6 file needs some TLC.

bin

zef expects these to be Raku executables. However, some people put Bash files there, which fail if the file is installed with zef but work if the distribution is cloned. This is probably a documentation issue: mention the restriction and how to overcome it.

doc / docs

The documentation states the section should be doc but it has been pointed out that docs is better.

installation

While doc is specified, either zef does not appear to install (in the sense of the contents being programmatically reachable) anything except the contents of bin resources and lib.

So, adding new sections to META6 should be associated with ways that the contents can be accessed.

[hythm7]

only allowable "auth" types to be "cpan:" and "zef:": any other auth types (such as "github:") should not be allowed, as there is no guarantee on the validity of the source, or dependability on what constitutes a "version".

Does this apply to community distributions only?

If not, what will happen if an organization or a company hosts their own distributions and want the auth to be :auth<some-org:>?

[lizmat]

If not, what will happen if an organization or a company hosts their own distributions and want the auth to be :authsome-org:?

Good question. Such an organization wouldn't want to just allow any auth string: so I guess we'll need to introduce some way of introducing a valid auth. Note to self: this will also affect PURL.

[ugexe]

Good question. Such an organization wouldn't want to just allow any auth string: so I guess we'll need to introduce some way of introducing a valid auth. Note to self: this will also affect PURL.

I'm really not following why we would do any validation or enforcement of specific auths at a language level. If an organization doesn't want to use a given auth string in actual code then they would enforce that during code review. If they want to enforce preventing such a dependency from being installed on a system then they would handle that at a tooling level (such as lock files which include hashes and urls from where things came from), artifact management such as artifactory, etc. What is being suggested here seems like it would disallow loading arbitrary auths, but I don't really feel like that is a problem on production systems based on my experience.

[librasteve]

i am broadly supportive of this initiative

here are the latest stats frpm https://raku.land

Ecosystem	#	%
CPAN	553	23.2%
p6c	369	15.5%
zef	1,466	61.4%

I expect that the CPAN 23.2% is weighted heavily to older modules

I propose that we use this major change to also deprecate CPAN

Why?

Sadly CPAN has become a junkyard of rotting code, time to cut loose from the perl legacy

[finanalyst]

If not, what will happen if an organization or a company hosts their own distributions and want the auth to be :auth<some-org:>?

The intent of the change in META6 is to enable Raku modules to be verifiable so that there is a SBOM for any module released for public use. This will enable verifiable Raku modules to EASILY meet the new EU Cyber resiliance legislation.

When a company releases software that is used in the EU, it is by law responsible for its software and will be liable for losses due to changes in the software.

I think: The SBOM requirements are the workaround the EU has devised to allow for FOSS modules to be used as dependencies in software that is publicly released. If software with FOSS dependencies is released, the responsibility is on the owner/releaser of the software to ensure that it is tested with a fixed version of each dependency. If a dependent FOSS module is changed by a developer, it can only be released by incrementing the version. If the new version causes an error, then it will only matter if the owner of the final software has not verified usage of the new version, so liability is on the owner, not on an upstream developer.

Modules with auths having zef: or cpan: are required to have a version, in that once uploaded the code in a version is 'set in stone'. An auth type without a static version cannot be guaranteed to have the same code. Github repos may have versions, but the code can be changed.

What about github releases, where the version is static?

Consequently, if a private company wishes to have its own code, then it will need to create an SBOM of its own, relying on the SBOM of the FOSS developers.

This means, IMHO, that if the META6 of a distribution has auth types which are not known to be version static, then the SBOM tool will not produce a result.

However, the SBOM tool will itself be FOSS, and so it can be designed to have other auth types. So a company with a private repository can run the SBOM tool with its own type. The company is in any case liable for any loss due to changes in its software.

[lizmat]

Raku/RakuDoc-v2#67 (comment)

Adding a "contributors" or "credit" field.

[finanalyst]

On one hand, we could specify that any field not specified can be added. This offers flexibility.
But IMHO I think that for fields to be useful, they need to be extractable by an external tool.

However, I would like to avoid the huge number of fields that the epub specification has. They are endless and its difficult for someone who wants to create an epub to know which are needed.