Community
Participate
Working Groups
I recently had a customer which wanted to use pure SWT and JFace without RCP. We is using Maven and wanted to define a dependency to SWT. It turned out that SWT is outdated at Maven central. I suggest to publish our artifacts also to Maven, not sure what the process for this is, so I suggest to have this bug here as "master" bug for this work.
We've started to experiment this with some project. The first one is Californium and they've written a wiki page to outline the process (https://wiki.eclipse.org/Californium/Signing_Process). We've implemented for a few other projects now (e.g. Eclipse Paho now publishes its build to maven central) and we are confident it is the way to go. I still need to write some documentation about the process and announce it on cross-projec. If you want to start right away, I suggest you open a bug against Community/Hudson asking for a OSSRH project account and a project GPG keypair.
JDT has frequent requests, mentioning that our artifacts appear on Maven central irregularly, and when they do they not always contain correct released versions. This is bad for our users and bad for the reputation of Eclipse. For a recent example see (mentioned in bug 496062) http://search.maven.org/#artifactdetails%7Corg.eclipse.jdt.core.compiler%7Cecj%7C4.5.1%7Cjar which identifies like this: <scm> <url>http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.jdt.core/</url> <connection>:pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse</connection> </scm> <developers> <!-- I didn't develop ECJ, but the Central Sync Requirements mandate a developer in the POM: https://docs.sonatype.org/display/Repository/Central+Sync+Requirements --> <developer> <name>Ralph Schaer</name> <email>ralphschaer@gmail.com</email> </developer> </developers> (NB the CVS url!!) So my questions: (A) Is the approach in this bug mature to be centrally applied for artifacts of the Eclipse project on a regular basis? (B) Can releng please help on bug 476650 for publishing the non-osgi, pom-less artifact ecj? (Also some signing issue might be involved). (C) To resolve bug 496062 I suggest to publish 4.5.2 to Maven central before another volunteer from the community does it with incomplete understanding of the Eclipse release process. (D) If doing this centrally for all SDK artifacts is asking too much from releng(?), should we use repo.eclipse.org for a hand-over: sub-projects publish to repo.eclipse.org from where releng forwards release versions to Maven central?
Ed W. is requesting the same at even larger scale, see bug 497982.
*** Bug 394799 has been marked as a duplicate of this bug. ***
*** Bug 434164 has been marked as a duplicate of this bug. ***
*** Bug 477402 has been marked as a duplicate of this bug. ***
I believe, when publishing to Maven central we should ensure that as much as possible these artifacts are consumable in plain maven (non-tycho) builds. This requires to declare "real" dependencies in all poms. I have positive experience with letting b3 extract these dependencies from MANIFEST.MF, and could help on setting up this part for the Eclipse project. Some things to figure out: (1) by default b3 assigns all artifacts to groupId p2.osgi.bundle, this has to be avoided (I don't think b3 actually reads any existing poms, does it?). (2) generated poms should not blindly replace existing poms, some existing information should probably be retained, like license, scm url etc. (3) check if explicit pom dependencies cause a problem to tycho builds consuming these artifacts. For (1) we can define suitable <mavenMappings> elements in the b3aggr. (2) sounds like a task for a simple xslt. Did any of the projects already pushing to Maven central solve these problems in a way that we can learn from?
(In reply to Stephan Herrmann from comment #7) > I have positive experience with letting b3 extract these dependencies from > MANIFEST.MF, and could help on setting up this part for the Eclipse project. Off topic: "We" call that the "CBI aggregator" now :) (Since it "moved"). Latest version built on "Mars.2" can be obtained from http://download.eclipse.org/cbi/updates/aggregator/ide/4.5/ or http://download.eclipse.org/cbi/updates/aggregator/headless/4.5/ But the reason I am writing is that if you have some "working example" of using it to produce maven artifacts, it would be very helpful for you to try using the new version to make sure it produces the same output as the "4.4 version" did. If so, good. If not, a bug in https://bugs.eclipse.org/bugs/enter_bug.cgi?product=CBI&component=CBI%20p2%20Repository%20Aggregator would be appropriate. Or, if "public" you could just describe/provide the use case via bug 506245. But none of the above is urgent, so np if you do not have the time or motivation. (It is actually helpful just to know that anyone has been using it for that :) Thanks,
Created attachment 265571 [details] aggregation model I started to play with mavenizing all SDK artifacts using the CBI aggregator. PFA my current take at an aggregation model. When running outside Eclipse.org infra simply swap enablement of the two repositories (to use http: rather than direct file access). The result from aggregating Neon.2 RC2 can be found at http://build.eclipse.org/tools/objectteams/sdk_aggr/result/final/ The main goal here is to create maven dependencies using coordinates that make sense in the maven universe. For org.eclipse.* artifacts I use this mapping: ^(org\.eclipse\.[^.]*)(.*)$ => $1:$1$2 translating e.g.: org.eclipse.jdt.core => org.eclipse.jdt:org.eclipse.jdt.core For other artifacts different rules apply. I started to create the corresponding maven mappings but had considerably hard time actually finding all these artifacts in maven-land. Here's how the non-eclipse poms are currently rendered: a/jre/javase/a.jre.javase/1.6.0/a.jre.javase-1.6.0.pom com/jcraft/jsch/0.1.53.v201508180515/jsch-0.1.53.v201508180515.pom com/ibm/icu/com.ibm.icu/56.1.0.v201601250100/com.ibm.icu-56.1.0.v201601250100.pom javax/annotation/javax.annotation-api/1.2.0.v201602091430/javax.annotation-api-1.2.0.v201602091430.pom javax/xml/javax.xml/1.3.4.v201005080400/javax.xml-1.3.4.v201005080400.pom javax/servlet/jsp/javax.servlet.jsp/2.2.0.v201112011158/javax.servlet.jsp-2.2.0.v201112011158.pom javax/servlet/javax.servlet-api/3.1.0.v201410161800/javax.servlet-api-3.1.0.v201410161800.pom javax/inject/javax.inject/1.0.0.v20091030/javax.inject-1.0.0.v20091030.pom javax/el/javax.el/2.2.0.v201303151357/javax.el-2.2.0.v201303151357.pom org/apache/httpcomponents/httpcore/4.3.3.v201411290715/httpcore-4.3.3.v201411290715.pom org/apache/httpcomponents/httpclient/4.3.6.v201511171540/httpclient-4.3.6.v201511171540.pom org/apache/commons/org.apache.commons.logging/1.1.1.v201101211721/org.apache.commons.logging-1.1.1.v201101211721.pom org/apache/commons/org.apache.commons.jxpath/1.3.0.v200911051830/org.apache.commons.jxpath-1.3.0.v200911051830.pom org/apache/commons/org.apache.commons.codec/1.6.0.v201305230611/org.apache.commons.codec-1.6.0.v201305230611.pom org/apache/jasper/org.apache.jasper.glassfish/2.2.2.v201501141630/org.apache.jasper.glassfish-2.2.2.v201501141630.pom org/apache/batik/org.apache.batik.css/1.7.0.v201011041433/org.apache.batik.css-1.7.0.v201011041433.pom org/apache/batik/org.apache.batik.util/1.7.0.v201011041433/org.apache.batik.util-1.7.0.v201011041433.pom org/apache/batik/org.apache.batik.util.gui/1.7.0.v200903091627/org.apache.batik.util.gui-1.7.0.v200903091627.pom org/apache/lucene/lucene-analyzers/3.5.0.v20120725-1805/lucene-analyzers-3.5.0.v20120725-1805.pom org/apache/lucene/lucene-core/3.5.0.v20120725-1805/lucene-core-3.5.0.v20120725-1805.pom org/apache/felix/org.apache.felix.gogo.command/0.10.0.v201209301215/org.apache.felix.gogo.command-0.10.0.v201209301215.pom org/apache/felix/org.apache.felix.gogo.runtime/0.10.0.v201209301036/org.apache.felix.gogo.runtime-0.10.0.v201209301036.pom org/apache/felix/org.apache.felix.gogo.shell/0.10.0.v201212101605/org.apache.felix.gogo.shell-0.10.0.v201212101605.pom org/apache/ant/org.apache.ant/1.9.6.v201510161327/org.apache.ant-1.9.6.v201510161327.pom org/tukaani/xz/org.tukaani.xz/1.3.0.v201308270617/org.tukaani.xz-1.3.0.v201308270617.pom org/eclipse/jetty/orbit/com.sun.el/2.2.0.v201303151357/com.sun.el-2.2.0.v201303151357.pom org/w3c/css/org.w3c.css.sac/1.3.1.v200903091627/org.w3c.css.sac-1.3.1.v200903091627.pom org/w3c/dom/org.w3c.dom.smil/1.0.1.v200903091627/org.w3c.dom.smil-1.0.1.v200903091627.pom org/w3c/dom/org.w3c.dom.svg/1.1.0.v201011041433/org.w3c.dom.svg-1.1.0.v201011041433.pom org/w3c/dom/org.w3c.dom.events/3.0.0.draft20060413_v201105210656/org.w3c.dom.events-3.0.0.draft20060413_v201105210656.pom org/hamcrest/core/org.hamcrest.core/1.3.0.v201303031735/org.hamcrest.core-1.3.0.v201303031735.pom org/sat4j/pb/org.sat4j.pb/2.3.5.v201404071733/org.sat4j.pb-2.3.5.v201404071733.pom org/sat4j/core/org.sat4j.core/2.3.5.v201308161310/org.sat4j.core-2.3.5.v201308161310.pom org/ow2/asm/asm-tree/5.0.1.v201404251740/asm-tree-5.0.1.v201404251740.pom org/ow2/asm/asm/5.0.1.v201404251740/asm-5.0.1.v201404251740.pom org/junit/org.junit/4.12.0.v201504281640/org.junit-4.12.0.v201504281640.pom Of course, these will not be pushed to central by us, but the above listing indicates how those artifacts are being referenced in dependencies of our artifacts. I could use a helping hand to find and validate those coordinates. My current thinking is, that we'll just prune the <dependencies> sections from these poms, and insert them into our "real" poms before pushing to central. OTOH, I see that Xtext, e.g., has pushed artifacts with similarly terse poms, as those generated by the aggregator, e.g.: http://repo1.maven.org/maven2/org/eclipse/xtext/org.eclipse.xtext/2.9.2/org.eclipse.xtext-2.9.2.pom What do people think about this?
(In reply to Stephan Herrmann from comment #9) > Created attachment 265571 [details] > aggregation model > > I started to play with mavenizing all SDK artifacts using the CBI aggregator. > > PFA my current take at an aggregation model. When running outside > Eclipse.org infra simply swap enablement of the two repositories (to use > http: rather than direct file access). I uploaded another version. To try it outside Eclipse.org you'll have to replace /home/data/httpd/download.eclipse.org with http://download.eclipse.org
@Mikaël, any news on the account/signing topic?
(In reply to Stephan Herrmann from comment #11) > @Mikaël, any news on the account/signing topic? Should this be setup for https://hudson.eclipse.org/platform/ or https://hudson.eclipse.org/releng/ ? (ie which HIPP will eventually do the publication?). Adding Frederic as he is in charge of creating the OSSRH accounts and setup the GPG signing for HIPPs.
What do you expect to get published? Releases only?
(In reply to Dani Megert from comment #13) > What do you expect to get published? Releases only? Yes, definitely releases only. In particular we should ensure that per artifact for each major.minor.micro there is only one artifact in Maven central (i.e., no qualifier-only updates). We still have the issue, that Maven will see our artifacts as snapshot builds (due to presence of a qualifier). It takes some effort to convince Maven users, that our artifacts *are* releases. If we would publish both kinds of artifacts with no visible distinction (like 1.2.3 vs. 1.2.3-SNAPSHOT) then all kinds of confusion could result.
(In reply to Stephan Herrmann from comment #10) > I uploaded another version. To try it outside Eclipse.org you'll have to > replace /home/data/httpd/download.eclipse.org with > http://download.eclipse.org Just FYI, I have opened bug 508209 to address this issue.
(In reply to Mikaël Barbero from comment #12) > (In reply to Stephan Herrmann from comment #11) > > @Mikaël, any news on the account/signing topic? > > Should this be setup for https://hudson.eclipse.org/platform/ or > https://hudson.eclipse.org/releng/ ? (ie which HIPP will eventually do the > publication?). > > Adding Frederic as he is in charge of creating the OSSRH accounts and setup > the GPG signing for HIPPs. I don't know how exactly responsibilities are divided between these two HIPPs, but from my guess it should be the releng one. Sravan, can you recommend? It will be a once-per-release activity: several post-build steps and then the upload to OSSRH. Is selecting the HIPP important for choosing the group of committers that can configure and trigger the corresponding jobs?
(In reply to Stephan Herrmann from comment #16) > (In reply to Mikaël Barbero from comment #12) > > (In reply to Stephan Herrmann from comment #11) > > > @Mikaël, any news on the account/signing topic? > > > > Should this be setup for https://hudson.eclipse.org/platform/ or > > https://hudson.eclipse.org/releng/ ? (ie which HIPP will eventually do the > > publication?). > > > > Adding Frederic as he is in charge of creating the OSSRH accounts and setup > > the GPG signing for HIPPs. > > I don't know how exactly responsibilities are divided between these two > HIPPs, but from my guess it should be the releng one. > > Sravan, can you recommend? It will be a once-per-release activity: several > post-build steps and then the upload to OSSRH. > > Is selecting the HIPP important for choosing the group of committers that > can configure and trigger the corresponding jobs? I recommend releng hipp. I would like to keep platform hipp to gerrit and testing only. We can add a group to the hudson job. It won't be a problem.
(In reply to Sravan Kumar Lakkimsetti from comment #17) > I recommend releng hipp. +1
I cleaned up my list of 3rd party dependencies (and ran the aggr against Neon2.RC3) (original file at http://build.eclipse.org/tools/objectteams/sdk_aggr/result/final/poms.txt ) FOUND: -------- com/ibm/icu/icu4j/56.1.0.v201601250100 com/jcraft/jsch/0.1.53.v201508180515 javax/annotation/javax.annotation-api/1.2.0.v201602091430 javax/el/el-api/2.2.0.v201303151357 javax/servlet/javax.servlet-api/3.1.0.v201410161800 javax/servlet/jsp/javax.servlet.jsp-api/2.2.0.v201112011158 javax/inject/javax.inject/1.0.0.v20091030 junit/junit/4.12.0.v201504281640 org/apache/ant/ant/1.9.6.v201510161327 org/apache/httpcomponents/httpclient/4.3.6.v201511171540 org/apache/httpcomponents/httpcore/4.3.3.v201411290715 org/apache/lucene/lucene-analyzers/3.5.0.v20120725-1805 org/apache/lucene/lucene-core/3.5.0.v20120725-1805 org/apache/felix/org.apache.felix.gogo.command/0.10.0.v201209301215 org/apache/felix/org.apache.felix.gogo.runtime/0.10.0.v201209301036 org/apache/felix/org.apache.felix.gogo.shell/0.10.0.v201212101605 org/eclipse/jetty/orbit/com.sun.el/2.2.0.v201303151357 org/hamcrest/hamcrest-core/1.3.0.v201303031735 org/ow2/asm/asm/5.0.1.v201404251740 org/ow2/asm/asm-tree/5.0.1.v201404251740 org/tukaani/xz/1.3.0.v201308270617 NOT FOUND: ---------- javax/xml/javax.xml/1.3.4.v201005080400 org/apache/commons/org.apache.commons.logging/1.1.1.v201101211721 org/apache/commons/org.apache.commons.jxpath/1.3.0.v200911051830 org/apache/commons/org.apache.commons.codec/1.6.0.v201305230611 org/apache/jasper/org.apache.jasper.glassfish/2.2.2.v201501141630 org/apache/batik/org.apache.batik.css/1.7.0.v201011041433 org/apache/batik/org.apache.batik.util/1.7.0.v201011041433 org/apache/batik/org.apache.batik.util.gui/1.7.0.v200903091627 org/w3c/dom/org.w3c.dom.smil/1.0.1.v200903091627 org/w3c/dom/org.w3c.dom.svg/1.1.0.v201011041433 org/w3c/dom/org.w3c.dom.events/3.0.0.draft20060413_v201105210656 VERSION PROBLEMS: ----------------- org/w3c/css/org.w3c.css.sac/1.3.1.v200903091627 -> only 1.3 available as org.w3c.css:sac org/sat4j/pb/org.sat4j.pb/2.3.5.v201404071733 org/sat4j/core/org.sat4j.core/2.3.5.v201308161310 -> only 2.3.1 available as org.sat4j:org.sat4j.core / org.sat4j:org.sat4j.pb Of the not-found artifacts, some are actually available with some org.eclipse.* groupId, not sure if those could possibly be seen as the canonical artifacts of their kind? Who pulls these dependencies into the SDK? Would they know the most suitable maven coordinates of these artifacts? Maybe for a first version we can live with just any artifact available on central, but in the end, using sub-optimal coordinates will cause duplicates in consuming projects. Aside: I'm always flabbergasted by the creativity in naming things as group:artifact :)
Comparing poms ... original: http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/tree/org.eclipse.jdt.core/pom.xml?h=R4_6_maintenance vs. generated by aggregator: http://build.eclipse.org/tools/objectteams/sdk_aggr/result/final/org/eclipse/jdt/org.eclipse.jdt.core/3.12.2.v20161117-1814/org.eclipse.jdt.core-3.12.2.v20161117-1814.pom ... I don't see a lot of information in the original that would be relevant for consumers via Maven Central. Would it suffice to just insert the copyright notice (making Igor Fedorenko the most famous person in the repo :) ? I believe we also need to insert scm-coordinates, authors etc. How do other projects publishing the Maven Central handle this?
(In reply to Stephan Herrmann from comment #20) > I believe we also need to insert scm-coordinates, authors etc. How do other > projects publishing the Maven Central handle this? OK, scm-coordinates *are* in the parent pom, but only as a property, e.g: http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/tree/pom.xml?h=R4_6_maintenance <properties> <tycho.scmUrl>scm:git:git://git.eclipse.org/gitroot/jdt/eclipse.jdt.core.git</tycho.scmUrl> </properties>
I was just reminded of bug 314992: should we publish org.eclipse.jdt.core-3.12.2.v20161117-1814 or org.eclipse.jdt.core-3.12.2-v20161117-1814 (dot vs. dash)? The former is more natural from osgi p.o.v. but is not a parsable maven version: <major>.<minor>.<revision>([ -<qualififer> ] | [ -<build> ]) As a result maven would fall back to plain string comparison on versions which can cause all kinds of havoc. Given that the aggregator is able to produce the maven compatible versions, I'd vote for using those for anything we publish to Maven Central. Result can be seen at http://build.eclipse.org/tools/objectteams/sdk_aggr/result-strict-maven/final/ Perhaps we need to fix bug 508412 first? I.e.: Is maven-metadata.xml required for upload or will it be created on OSSRH?
(In reply to Stephan Herrmann from comment #19) > NOT FOUND: > ---------- A tip: Maven Central allows you to search for particular classnames [1]; this helps in cases where the GAV is hard to guess. For legacy reasons, some Apache Commons libraries are available under groupId/artifactId commons-$PROJECT. > org/apache/commons/org.apache.commons.logging/1.1.1.v201101211721 <http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22commons-logging%22%20AND%20a%3A%22commons-logging%22> > org/apache/commons/org.apache.commons.jxpath/1.3.0.v200911051830 <http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22commons-jxpath%22%20AND%20a%3A%22commons-jxpath%22> Hope that helps. [1] <http://search.maven.org/#advancedsearch%7Cgav>
There are some problems with the Batik artifacts. I've mentioned them in this old blog post: https://blog.pdark.de/2012/02/29/how-i-came-to-hate-orbit/ My final solution was to move everything from orbit into the namespace org.eclipse.orbit:orbit.*, i.e. batik became org.eclipse.orbit:orbit.org.apache.batik Plus I've written a tool which cleans junk out of Orbit artifacts. Orbit's Batik, for example, contains code from commons-io. That has to go or people will get strange errors like the compiler complaining that certain methods don't exist. Unlike OSGi, Maven projects have only a single classpath, so you can't have two IOUtils with different methods.
(In reply to Mikaël Barbero from comment #12) > Adding Frederic as he is in charge of creating the OSSRH accounts and setup > the GPG signing for HIPPs. OSSRH requires a group ID during the account creation. What would be the right one to use here? In general the group ID references the project, so when the HawkBit project wants to deploy to Maven Central, it has the group ID "org.eclipse.hawkbit". According to that rule the OSSRH account for the releng project would use a group ID like "org.eclipse.releng" or "org.eclipse.platform.releng". I guess that does not really fit in with artifacts being deployed that have another group ID, e.g. org.eclipse.swt, org.eclipse.equinox. To accommodate other projects, the group ID has to be broader, as in "org.eclipse". Not sure if that can cause other problems. Alternatively accounts for every platform project (e.g. SWT, Equinox, JFace) with separate group IDs would need to be created, which would certainly add some complexity to the deployment process. WDYT?
(In reply to Frederic Gurr from comment #25) > (In reply to Mikaël Barbero from comment #12) > > Adding Frederic as he is in charge of creating the OSSRH accounts and setup > > the GPG signing for HIPPs. > > OSSRH requires a group ID during the account creation. What would be the > right one to use here? That's a tough one ... > In general the group ID references the project, so when the HawkBit project > wants to deploy to Maven Central, it has the group ID "org.eclipse.hawkbit". That also corresponds to what I see in our poms, e.g., JDT artifacts specify groupId org.eclipse.jdt (although "project" is an ambiguous term in the context of the Eclipse Project). > According to that rule the OSSRH account for the releng project would use a > group ID like "org.eclipse.releng" or "org.eclipse.platform.releng". > I guess that does not really fit in with artifacts being deployed that have > another group ID, e.g. org.eclipse.swt, org.eclipse.equinox. To accommodate > other projects, the group ID has to be broader, as in "org.eclipse". Not > sure if that can cause other problems. I have some sympathy to using groupId "org.eclipse" for the Eclipse Project, but - I don't know if we (who?) can overrule the existing decision as reflected in poms - users may already use dependencies in accordance with our poms, which would break if we change this now. - I only *assume* without evidence that using "org.eclipse" would *not* conflict with other projects publishing directly in 3-part groups "org.eclipse.xyz". IOW: do groupIds have hierarchical semantics or is "org.eclipse" indenpendent of "org.eclipse.xyz"? Technically, we could easily remap groupIds during aggregation. > Alternatively accounts for every platform project (e.g. SWT, Equinox, JFace) > with separate group IDs would need to be created, which would certainly add > some complexity to the deployment process. > > WDYT? To be specific, at level 3 I see 26 groups: ant debug equinox jface pde search ui compare e4 help jsch platform swt update core ecf jdt ltk rcp team cvs emf jetty osgi sdk text I think, if we want to go to Maven Central, we have to bite this bullet. We certainly still want to give a single group of committers (releng (plus a few)?) the permissions to publish artifacts from all of these groups. Certainly, this is a decision we should make only once - no experimenting now, and revising later. @Dani, can you speak for the PMC, or does this need more research/discussion?
(In reply to Frederic Gurr from comment #25) > In general the group ID references the project, so when the HawkBit project > wants to deploy to Maven Central, it has the group ID "org.eclipse.hawkbit". > According to that rule the OSSRH account for the releng project would use a > group ID like "org.eclipse.releng" or "org.eclipse.platform.releng". > I guess that does not really fit in with artifacts being deployed that have > another group ID, e.g. org.eclipse.swt, org.eclipse.equinox. To accommodate > other projects, the group ID has to be broader, as in "org.eclipse". Why? What's wrong with a groupId:artifact of org.eclipse.platform:org.eclipse.swt or org.eclipse.platform:org.eclipse.equinox? The groupId doesn't have to be a prefix of the artifactId. (In fact, in the non-OSGi world, often only the groupId follows the reverse-domain-name structure.)
I didn't say it's wrong. I just want it to be discussed and decided. :)
Given that we already have groupId, artifactId in our poms, changing the policy would need stronger reasons than convenience on our side, IMHO. Group org.eclipse.platform would only fit for parts of the Eclipse project, so an appropriate proposal could be: - org.eclipse.platform - org.eclipse.jdt - org.eclipse.pde For JDT & PDE this wouldn't change anything wrt current poms. Would it improve anything for Platform and its sub projects? Would this outweigh the problems of breaking existing clients? Or, is breaking existing builds actually "good", meaning: migrating from previous groupId to a new groupId could be seen as the intentional act of moving from previous "inofficially published" Eclipse artifacts to the official artifacts to be published via this bug? If the answer here is "yes", than this could speak *for* groupId "org.eclipse", pending analysis, where and how that groupId may already be used elsewhere (see http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.eclipse%22 ).
(In reply to Aaron Digulla from comment #24) > There are some problems with the Batik artifacts. I've mentioned them in > this old blog post: > https://blog.pdark.de/2012/02/29/how-i-came-to-hate-orbit/ Thanks, this post reminds my of a third options for versioning: org.eclipse.jdt.core-3.12.2.v20161117-1814 org.eclipse.jdt.core-3.12.2-v20161117-1814 org.eclipse.jdt.core-3.12.2 For maven users the last option would be most natural, clearly marking the artifact as a release. I'll see if I can tweak the aggregator to use this version format (=> bug 508549).
(In reply to Stephan Herrmann from comment #30) > Thanks, this post reminds my of a third options for versioning: > > org.eclipse.jdt.core-3.12.2.v20161117-1814 > org.eclipse.jdt.core-3.12.2-v20161117-1814 > org.eclipse.jdt.core-3.12.2 > > For maven users the last option would be most natural, clearly marking the > artifact as a release. I'll see if I can tweak the aggregator to use this > version format (=> bug 508549). Please! That would also solve all problems with projects who make changes and forget to update the micro version since Maven Central will reject those submissions making the error obvious.
(In reply to Andreas Sewe from comment #27) > What's wrong with a groupId:artifact of > org.eclipse.platform:org.eclipse.swt or > org.eclipse.platform:org.eclipse.equinox? The groupId doesn't have to be a > prefix of the artifactId. (In fact, in the non-OSGi world, often only the > groupId follows the reverse-domain-name structure.) The problems start when you don't have a groupId, for example, when you have to deploy the JARs in a web container without OSGi. The usual case is that you end up with five "util-X.Y.Z.jar" in WEB-INF/lib/. If you're lucky, the versions are different. In Eclipse this is rarely an issue because most bundles end up in a single folder somewhere and name collisions there would break the IDE. So most bundles names can be mapped directly to artifactId and the groupId can use the Eclipse project name. The big exception is Orbit, as I explained above. Orbit bundles should not be deployed to Maven central or, if you must, you have to add a prefix to the artifactId.
Publishing Eclipse artifacts to Maven Central would help with the formatter-maven-plugin development. https://github.com/revelc/formatter-maven-plugin/issues/149
*** Bug 199302 has been marked as a duplicate of this bug. ***
FYI - A Github repo with latest SWT artifacts in a Maven repo can be found here: https://github.com/maven-eclipse/maven-eclipse.github.io
(In reply to Lars Vogel from comment #35) > FYI - A Github repo with latest SWT artifacts in a Maven repo can be found > here: https://github.com/maven-eclipse/maven-eclipse.github.io Yea, many people create mavenized versions of our stuff. That's part of the problem, not of the solution.
(In reply to Stephan Herrmann from comment #36) > That's part of the > problem, not of the solution. Completely agree. I think it demonstrates how important your work is.
I *think* this is a minor issue but thought I would point it out now, for those interested in "mavenizing" Eclipse bundles. There are several hundred "warnings" in our large debug output (~300 M) that say things like [DEBUG] Dependency from /opt/public/eclipse/builds/4M/gitCache/eclipse.platform.releng.aggregator/eclipse.platform.releng.tychoeclipsebuilder/sdk to nested classpath entry /opt/public/eclipse/builds/4M/gitCache/eclipse.platform.releng.aggregator/eclipse.jdt.debug/org.eclipse.jdt.debug/jdi.jar can not be represented in Maven model and will not be visible to non-OSGi aware Maven plugins I *think* this mostly involves features and test bundles. And, even though there are hundreds of messages I think the unique bundles involved are only 10 or so. All the platform's debug build output is in a file named mb060_run-maven-build_output.txt In the "release engineering logs" section. http://download.eclipse.org/eclipse/downloads/drops4/R-4.6.1-201609071200/buildlogs.php Or, for full link for "4.6.1" release: *[You typically want to download these logs, and then look at them with an editor that can handle large files -- the browser doesn't do so well with 300M page :) ]* http://download.eclipse.org/eclipse/downloads/drops4/R-4.6.1-201609071200/buildlogs/mb060_run-maven-build_output.txt
After fixing bug 508549 I created a new repo at http://build.eclipse.org/tools/objectteams/sdk_aggr/result-maven-release/final/ New: - consistently use 3-part versions to mark everything as release - mapped commons-* to their original coordinates
Some more analysis regarding the remaining unresolved dependencies, with more requests for comments: javax/xml/javax.xml/1.3.4.v201005080400 => If I understand correctly, which should just delete this dependency and assume its classes are available from the JRE. org/apache/jasper/org.apache.jasper.glassfish/2.2.2.v201501141630 => ??? Is this candidate really the only one available: http://search.maven.org/#artifactdetails%7Corg.eclipse.jetty.orbit%7Corg.apache.jasper.glassfish%7C2.2.2.v201112011158%7Corbit org/w3c/dom/org.w3c.dom.smil/1.0.1.v200903091627 org/w3c/dom/org.w3c.dom.svg/1.1.0.v201011041433 org/w3c/dom/org.w3c.dom.events/3.0.0.draft20060413_v201105210656 => I could find only dependencies from *features*. Since features are irrelevant to maven, these dependencies can probably be ignored. org/w3c/css/org.w3c.css.sac/1.3.1.v200903091627 => Used by several plugins from Platform/UI. I only found this candidate with matching version: http://search.maven.org/#artifactdetails%7Corg.eclipse.birt.runtime%7Corg.w3c.css.sac%7C1.3.1.v200903091627%7Cjar Does anybody know a better candidate?? org/apache/batik/org.apache.batik.css/1.7.0.v201011041433 org/apache/batik/org.apache.batik.util/1.7.0.v201011041433 org/apache/batik/org.apache.batik.util.gui/1.7.0.v200903091627 => used only by org.eclipse.e4.ui.css.core (plus some features, which are irrelevant). Could someone from Platform/UI please confirm that this is a good candidate: http://search.maven.org/#artifactdetails%7Corg.apache.xmlgraphics%7Cbatik-css%7C1.7%7Cjar It seems, the util and util.gui artifacts are only transitive dependencies. org/sat4j/pb/org.sat4j.pb/2.3.5.v201404071733 org/sat4j/core/org.sat4j.core/2.3.5.v201308161310 => used by org.eclipse.equinox.p2.director Identified as http://search.maven.org/#artifactdetails%7Corg.ow2.sat4j%7Corg.ow2.sat4j.core%7C2.3.5%7Cjar http://search.maven.org/#artifactdetails%7Corg.ow2.sat4j%7Corg.ow2.sat4j.pb%7C2.3.5%7Cjar
I run https://github.com/maven-eclipse/maven-eclipse.github.io , would are there any plans to put historical dependencies in versions older than 4.6.1 in this repository? It would eliminate any need for old 3rd party repositories
Found this one: (In reply to Stephan Herrmann from comment #40) > org/apache/jasper/org.apache.jasper.glassfish/2.2.2.v201501141630 > > => ??? > Is this candidate really the only one available: > http://search.maven.org/#artifactdetails%7Corg.eclipse.jetty.orbit%7Corg.apache.jasper.glassfish%7C2.2.2.v201112011158%7Corbit This orbit artifact actually corresponds to: - http://search.maven.org/#artifactdetails%7Corg.glassfish.web%7Cjavax.servlet.jsp%7C2.2.2%7Cjar - plus ecj classes (?!?!) This cannot be handled by maven mappings alone. Users would have to manually provide ecj somehow. OTOH, the only consumer of this dependency is org.eclipse.help.feature.group, which I will exclude from the bunch anyway (features are of no interest for maven).
(In reply to Leon Blakey from comment #41) > I run https://github.com/maven-eclipse/maven-eclipse.github.io , would are > there any plans to put historical dependencies in versions older than 4.6.1 > in this repository? It would eliminate any need for old 3rd party > repositories I'm still working on the mechanics, and then I'm not to decide which versions will be published via this new channel. I will soon write a request for approval to the Eclipse PMC, in which I should probably propose also a strategy as you are asking about. My current feeling is to propose Neon.2 and moving forward, *not* going backwards in history. Let's see.
Since some problematic dependencies are *only* pulled in via features, I pondered the options: tell the aggregator to mirror only bundles, but then I'd have to list (more or less) all bundles. That's when I decided to use a minimalistic "garbage collector" in the shape of a set of shell scripts: - remove all directories ending in "feature.group" - find all poms other than in org/eclipse - for each pom search for references in any other pom - if none found remove the bundle - iterate until a fixpoint if reached - remove empty directories This allowed me to get rid of various 3rd party source bundles plus these: commons-codec/commons-codec commons-logging/commons-logging javax/annotation/javax.annotation-api javax/xml/javax.xml javax/servlet/jsp/javax.servlet.jsp-api javax/servlet/javax.servlet-api javax/el/el-api org/apache/httpcomponents/httpcore org/apache/httpcomponents/httpclient org/apache/jasper/org.apache.jasper.glassfish org/apache/felix/org.apache.felix.gogo.command org/w3c/dom/org.w3c.dom.svg org/w3c/dom/org.w3c.dom.events org/w3c/dom/org.w3c.dom.smil The resulting repo can be seen at http://build.eclipse.org/tools/objectteams/sdk_aggr/result-maven-release-featureless/final These are the remaining non-eclipse artifacts with current maven mapping: com/jcraft/jsch/0.1.53 com/ibm/icu/icu4j/56.1.0 commons-jxpath/commons-jxpath/1.3.0 javax/inject/javax.inject/1.0.0 junit/junit/4.12.0 org/apache/batik/org.apache.batik.css/1.7.0 org/apache/batik/org.apache.batik.util/1.7.0 org/apache/batik/org.apache.batik.util.gui/1.7.0 org/apache/lucene/lucene-analyzers/3.5.0 org/apache/lucene/lucene-core/3.5.0 org/apache/felix/org.apache.felix.gogo.runtime/0.10.0 org/apache/felix/org.apache.felix.gogo.shell/0.10.0 org/apache/ant/ant/1.9.6 org/tukaani/xz/1.3.0 org/w3c/css/org.w3c.css.sac/1.3.1 org/hamcrest/hamcrest-core/1.3.0 org/ow2/sat4j/org.ow2.sat4j.core/2.3.5 org/ow2/sat4j/org.ow2.sat4j.pb/2.3.5 org/ow2/asm/asm-tree/5.0.1 org/ow2/asm/asm/5.0.1 ----------------------------------------------------------------- Of these the following still remain problematic: ----------------------------------------------------------------- org/apache/batik/org.apache.batik.css/1.7.0 org/apache/batik/org.apache.batik.util/1.7.0 org/apache/batik/org.apache.batik.util.gui/1.7.0 org/w3c/css/org.w3c.css.sac/1.3.1 ----------------------------------------------------------------- => Platform/UI : anyone listening? Please see comment 40. -----------------------------------------------------------------
Created attachment 265765 [details] aggregation model with complete maven mappings This aggregation model should cover remapping of all 3rd party dependencies to something validly available from Maven Central.
Created attachment 265767 [details] "garbage collector" shell script
=== Step completed: === Aggregate a Maven repository with these properties: - include only bundles, no features - all artifacts have 3-part (release) version numbers - all 3rd party artifacts are mapped to GAV that exist on Maven Central - only those 3rd party artifacts included that are *directly* referenced from org.eclipse artifacts Result visible at http://build.eclipse.org/tools/objectteams/sdk_aggr/result-conditioned/final/ Full list of 3rd party artifacts (to be referenced, *not* published by us): .../final> find * -name eclipse -prune -o -name "[0-9]*" -print -prune com/jcraft/jsch/0.1.53 com/ibm/icu/icu4j/56.1.0 commons-jxpath/commons-jxpath/1.3.0 javax/inject/javax.inject/1.0.0 junit/junit/4.12.0 org/apache/xmlgraphics/batik-css/1.7.0 org/apache/lucene/lucene-analyzers/3.5.0 org/apache/lucene/lucene-core/3.5.0 org/apache/felix/org.apache.felix.gogo.runtime/0.10.0 org/apache/felix/org.apache.felix.gogo.shell/0.10.0 org/apache/ant/ant/1.9.6 org/tukaani/xz/1.3.0 org/ow2/sat4j/org.ow2.sat4j.core/2.3.5 org/ow2/sat4j/org.ow2.sat4j.pb/2.3.5 org/ow2/asm/asm-tree/5.0.1 org/ow2/asm/asm/5.0.1 === Mechanics: === == aggregate == Aggregation script is attachment 265765 [details], feed this into latest CBI aggregator. Result will appear in <something>/final == remove features == .../final> find * -name \*feature.group -exec /bin/rm -rf {} \; -prune == remove garbage == .../final> gc.sh > gc1.log .../final> for d in `cat toremove.txt` ; do /bin/rm -r $d; done .../final> mv toremove.txt toremove-1.txt .../final> gc.sh > gc2.log .../final> for d in `cat toremove.txt` ; do /bin/rm -r $d; done .../final> mv toremove.txt toremove-2.txt Here, gc.sh is attachment 265767 [details]. At 2 iterations we have a fixpoint. gc2.log also tells us for each 3rd party bundle, by which eclipse bundle it is used (only listing the first found). Also: .../final> /bin/rm -r p2.packed *.jar p2.index a
=== POM content? === Next on my list: what exact poms should we use for publication? - The existing poms in git don't seem to have much information that's valuable for Maven-based consumers. - The pom produced by the aggregator has all the dependency information and looks like a good starting point. Things to add(?): - copyright - packaging? (say "eclipse-plugin" or just pretend "jar" for consumption w/o tycho?) - <scm> element - information is available inside MANIFEST.MF/Eclipse-SourceReferences: - <issueManagement> element - point to bugs.eclipse.org for all artifacts Is any additional information required by OSSRH? For those projects that have already gone this route, what did they put into their poms? Maybe one of them already has a nice script for merging poms from different inputs?
(In reply to Stephan Herrmann from comment #48) > Is any additional information required by OSSRH? AFAIK, Maven Central requires *-sources.jar artifacts. Cursory inspection of the Maven repository at [1] suggests that *.source bundles became artifacts in their own right [2]. It would IMHO be highly useful with those sources could be matched to their respective binary artifacts [3] so that tools like m2e can automatically download the associated sources. [1] <http://build.eclipse.org/tools/objectteams/sdk_aggr/result-conditioned/final/> [2] <http://build.eclipse.org/tools/objectteams/sdk_aggr/result-conditioned/final/org/eclipse/emf/org.eclipse.emf.ecore.source/2.12.0/> [3] <http://build.eclipse.org/tools/objectteams/sdk_aggr/result-conditioned/final/org/eclipse/emf/org.eclipse.emf.ecore.source/2.12.0/>
(In reply to Andreas Sewe from comment #49) > (In reply to Stephan Herrmann from comment #48) > > Is any additional information required by OSSRH? > > AFAIK, Maven Central requires *-sources.jar artifacts. Cursory inspection of > the Maven repository at [1] suggests that *.source bundles became artifacts > in their own right [2]. It would IMHO be highly useful with those sources > could be matched to their respective binary artifacts [3] so that tools like > m2e can automatically download the associated sources. > > [1] > <http://build.eclipse.org/tools/objectteams/sdk_aggr/result-conditioned/final/> > > [2] > <http://build.eclipse.org/tools/objectteams/sdk_aggr/result-conditioned/final/org/eclipse/emf/org.eclipse.emf.ecore.source/2.12.0/> > > [3] > <http://build.eclipse.org/tools/objectteams/sdk_aggr/result-conditioned/final/org/eclipse/emf/org.eclipse.emf.ecore.source/2.12.0/> > Thanks, I filed bug 508910
For those not following the eclipse-pmc list, FYI: https://dev.eclipse.org/mhonarc/lists/eclipse-pmc/msg02829.html
Note to self: remember to build the repo including all platforms
From https://dev.eclipse.org/mhonarc/lists/eclipse-pmc/msg02832.html : "We use the following 3-part group ids 'org.eclipse.platform', 'org.eclipse.jdt' and 'org.eclipse.pde'." @Fred, this should unblock the creation of (three) OSSRH accounts, right? I will shortly create my repo-du-jours to reflect this new policy. Hopefully also leveraging the change from bug 508910. I'd like to target the first upload to approximately coincide with Neon.2, if possible. Does this sound feasible from infra p.o.v.? My last pending task is to merge necessary content into generated poms. Let me ask once more: do any other projects have some automation for this already? And: what elements exactly are required by OSSRH?
It seems to me that eclipse.org is big enough that it shouldn't be trying to do the OSSRH path with Sonatype, but should have a separate arrangement with Sonatype, like the Apache Software Foundation does. ASF owns `org.apache` and has its own Nexus staging repository, which feeds into Sonatype's Maven Central repository. Eclipse could do something similar with control over `org.eclipse`. Then maybe Eclipse won't have to follow all the normal OSSRH rules, and could have its own Maven deployment rules (sources, javadocs, pom fields, etc.) for all eclipse.org projects. This would make it easier for other Eclipse projects (http://projects.eclipse.org/), so they don't have to do their own thing for deploying artifacts. Overall, it'd be a great idea if Eclipse.org intends to be a community for more projects than just the Eclipse Platform.
Thanks, Christopher for you recommendations. I'm not sure about the timing, though. Do you have the contacts to negotiate such an arrangement with Sonatype? OTOH, I'm working on fulfilling the OSSRH rules as we speak. As of now I have the strong feeling that going the discussed route via OSSRH will be much faster than waiting for a "better" strategy to materialize. Regarding other projects: On the long run the idea is not only to publish the Eclipse Platform, but this will also set another example where other Eclipse projects can copy from when they want to publish to Maven Central. Mikaël can probably say more regarding other Eclipse projects.
Under http://build.eclipse.org/tools/objectteams/sdk_aggr/repo-201612131600/ I've created a new version of the repo. ======== Updates: ======== - source bundles correctly named and co-located (see bug 508910) - groupIds as decided by the PMC and mentioned in comment 53 - poms enriched with info from MANIFEST.MF + some static stuff Example: http://build.eclipse.org/tools/objectteams/sdk_aggr/repo-201612131600/org/eclipse/jdt/org.eclipse.jdt.core/3.12.2/org.eclipse.jdt.core-3.12.2.pom The repo contains all transitive dependencies for completeness (so people can verify that non-eclipse artifacts are indeed available from Maven Central), but subject for publishing are only the mentioned three groups - http://build.eclipse.org/tools/objectteams/sdk_aggr/repo-201612131600/org/eclipse/jdt/ - http://build.eclipse.org/tools/objectteams/sdk_aggr/repo-201612131600/org/eclipse/pde/ - http://build.eclipse.org/tools/objectteams/sdk_aggr/repo-201612131600/org/eclipse/platform/ (Indeed even the one artifact in group org/eclipse/birt/runtime is already avaible from Maven Central). ======= Issues: ======= Some manifests don't seem to contain any scm info (Eclipse-SourceReference): org.eclipse.jetty.continuation org.eclipse.jetty.io org.eclipse.jetty.util org.eclipse.jetty.security org.eclipse.jetty.server org.eclipse.jetty.servlet org.eclipse.ecf org.eclipse.ecf.filetransfer org.eclipse.ecf.identity org.eclipse.ecf.provider.filetransfer org.eclipse.ecf.provider.filetransfer.ssl org.eclipse.ecf.provider.filetransfer.httpclient4 org.eclipse.ecf.provider.filetransfer.httpclient4.ssl org.eclipse.ecf.ssl org.eclipse.jetty.http org.eclipse.emf.common org.eclipse.emf.ecore org.eclipse.emf.ecore.change org.eclipse.emf.ecore.xmi ==>> Where do we get this info from? (I thought this entry is asserted by our repo reports??) Some minor issue regarding the little tool I use to update the poms: - I coulnd't translate the names of these fragments: %win32FragmentName for org.eclipse.core.resources.win32.x86 %win32FragmentName for org.eclipse.core.resources.win32.x86_64 This is due to the fact that my tool cannot find plugin.properties in the host bundle. - I also need to re-create .sha1 and .md5 of modified poms. (do we need to provide these checksums? will they be created by nexus?) - We eventually need a home for my scripts and this little pom enricher (the latter written in Java) When done I will attach everything to this bug, to be picked up by releng. Our javadoc is packaged differently than expected, but according to OSSRH creating a jar containing only a README would be OK. In our case the readme would simple mention the doc plugin containing the corresponding javadoc: - org.eclipse.jdt.doc.isv - org.eclipse.pde.doc.user - org.eclipse.platform.doc.isv I hope we are nearly done here :)
(In reply to Stephan Herrmann from comment #55) > Thanks, Christopher for you recommendations. I'm not sure about the timing, > though. I have a number of contacts at Sonatype so if there is anything specific that we need to ask, I'll be happy to reach out.
(In reply to Stephan Herrmann from comment #55) > Mikaël can probably say more regarding other Eclipse projects. We can't go the way Christopher is recommending as, while it's attractive, Sonatype would ask the Eclipse Foundation to install the Pro version of Nexus, which is not full OSS. The bylaws of the Foundation forbids us to use proprietary software. So it's a showstopper. See https://dev.eclipse.org/mhonarc/lists/dash-dev/msg01348.html for Jason's statement about the usage of Nexus Pro.
(In reply to Stephan Herrmann from comment #56) > ======= > Issues: > ======= > Some manifests don't seem to contain any scm info (Eclipse-SourceReference): Follow-up in - bug 509285 (EMF) - bug 408760 comment 7 (ECF) - https://github.com/eclipse/jetty.project/issues/1177 (jetty)
(In reply to Stephan Herrmann from comment #59) > - https://github.com/eclipse/jetty.project/issues/1177 (jetty) Resolved: jetty is already on Maven Central (in groupId org.eclipse.jetty), so I just need to update the maven mappings, and we will *not* publish jetty via this bug.
New repo created, direct links are: * Platform: http://build.eclipse.org/tools/objectteams/sdk_aggr/repo-201612151340/org/eclipse/platform/ * JDT: http://build.eclipse.org/tools/objectteams/sdk_aggr/repo-201612151340/org/eclipse/jdt/ * PDE: http://build.eclipse.org/tools/objectteams/sdk_aggr/repo-201612151340/org/eclipse/pde/ All scm issues fixed: - jetty: no longer included - ecf: per information from bug 408760 - emf: in analogy to ecf => "Manual" fix for now. ECF should be good for next release. EMF?? "Manually" patched non-translatable %win32FragmentName I changed my aggregation strategy: - Previously I aggregated everything below org.eclipse.sdk.feature.group. + Now I start from "everything" from the release repo and then "manually" remove stuff that should be irrelevant in maven: - *feature.group - *feature.jar - tooling* - org/eclipse/*/*test* - everything that has only a pom but no jar - *-sources.jar* (needed to compensate for quirk in the fix for bug 508910) - anything not directly referenced from pom in org/eclipse/{platform,jdt,pde} As a result we may have a few more artifacts, not sure.
Stephan, maybe this work can be done in the Eclipse releng project and build machine? If you agree, we could suggest to nominate you as releng committer.
(In reply to Lars Vogel from comment #62) > Stephan, maybe this work can be done in the Eclipse releng project and build > machine? If you agree, we could suggest to nominate you as releng committer. Just a minute ago I directly contacted Mikaël, Fred & Sravan to discuss the next steps. At the end of the day, being a releng committer could help some, but if I can contribute via gerrit or such that's fine by me.
Why is "objectteams" part of this work/picture? This sounds wrong to me.
(In reply to Dani Megert from comment #64) > Why is "objectteams" part of this work/picture? This sounds wrong to me. :) That's just the workspace where I'm performing all the experiments on build.eclipse.org. If you see a problem with this, then, yes, I'll have to ask for a releng login ...
(In reply to Stephan Herrmann from comment #59) > (In reply to Stephan Herrmann from comment #56) > > ======= > > Issues: > > ======= > > Some manifests don't seem to contain any scm info (Eclipse-SourceReference): > > Follow-up in > - bug 509285 (EMF) Martin Taal is publishing latest EMF as we speak, see the thread started at https://dev.eclipse.org/mhonarc/lists/emf-dev/msg01817.html > - bug 408760 comment 7 (ECF) ECF is both a prereq and a consumer of Platform. Details are now in bug 509381. ECF (Scott) confirmed interest in participating in this effort.
I've given my sources for all repository manipulations a *temporary* git home (in the objectteams git - just so I can continue prototyping without needing a releng committer for every move). For the curious: all can be seen here: http://git.eclipse.org/c/objectteams/org.eclipse.objectteams.git/tree/releng/EnrichPoms Additionally, I have configured two hudson jobs on releng hipp to reproducibly perform all the steps mentioned before. The results can now be seen via job https://hudson.eclipse.org/releng/view/Publish%20to%20Maven%20Central/job/CBIaggregator/ The resulting repo will be stored in a Job-specific sub dir, currently: https://hudson.eclipse.org/releng/view/Publish%20to%20Maven%20Central/job/CBIaggregator/ws/repo-10/ All this is now fairly stable, waiting for the OSSRH accounts, to start experimenting with uploading and using their validation on staging. @Fred? Are you awfully busy right now?
(In reply to Stephan Herrmann from comment #67) > All this is now fairly stable, waiting for the OSSRH accounts, to start > experimenting with uploading and using their validation on staging. > > @Fred? Are you awfully busy right now? Sorry, for the delay. OSSRH accounts have been created. I'm setting up the GPG signing right now.
(In reply to Frederic Gurr from comment #68) > (In reply to Stephan Herrmann from comment #67) > > All this is now fairly stable, waiting for the OSSRH accounts, to start > > experimenting with uploading and using their validation on staging. > > > > @Fred? Are you awfully busy right now? > > Sorry, for the delay. OSSRH accounts have been created. I'm setting up the > GPG signing right now. Thanks a lot, Fred. In case I get the upload to work soon: does anyone listening here plan to review the content to be published, or is it OK for me to click "release" as soon as I get all requirements worked out / pass OSSRH validations?
OSSRH Tickets https://issues.sonatype.org/browse/OSSRH-27089 (Platform), https://issues.sonatype.org/browse/OSSRH-27090 (JDT) and https://issues.sonatype.org/browse/OSSRH-27091 (PDE) are now resolved. I've also created and configured the signing keys on the Releng HIPP. The public keys are https://pgp.mit.edu/pks/lookup?op=vindex&search=0xB6D3AB9BCC641282 (Platform) https://pgp.mit.edu/pks/lookup?op=vindex&search=0x7AA1F762B414F87E (JDT) https://pgp.mit.edu/pks/lookup?op=vindex&search=0x3C44082386085154 (PDE). They are signed by the webmaster's key. As a committer, you should sign them too. To select the different keys during a Maven build two things need to be set. * You have to use dedicated Maven settings which are named 'Deploy to OSSRH - Platform', 'Deploy to OSSRH - JDT', 'Deploy to OSSRH - PDE' respectively. These settings have to be selected in the Maven build step in your jobs (under Advanced). * The default key must be specified in the maven-gpg-plugin: <configuration> <gpgArguments> <arg>--default-key</arg> <arg><GPG KEY ID></arg> </gpgArguments> </configuration> GPG KEY IDs: -Platform: CC641282 -JDT: B414F87E -PDE: 86085154 The settings contain a server definition named 'ossrh' to let you upload things to Sonatype's server. This server id should be used in a distributionManagement repository somewhere specifying the URL. See http://central.sonatype.org/pages/ossrh-guide.html#releasing-to-central and http://central.sonatype.org/pages/ossrh-guide.html#ossrh-usage-notes for details. The GPG passphrase is also configured (encrypted) in the settings (as described at https://maven.apache.org/plugins/maven-gpg-plugin/usage.html#Configure_passphrase_in_settings.xml). You can now use the maven-gpg-plugin, it should work as is without further configuration if you properly selected the 'Deploy to OSSRH - *' setting. Let me know when you promoted your first release so I can comment on the OSSRH tickets. Or you can do this yourself.
Results from first validations run on OSSRH (using equinox & swt as my guinea pig): (1) we do need <developers> elements in poms (2) we do need javadoc artifacts (3) some artifacts are missing a source artifact Ad (1): I prototyped a lookup using the first segments of the git repo and map this to the project leads listed for that project: https://git.eclipse.org/c/equinox - Ian Bull - Pascal Rapicault - Thomas Watson https://git.eclipse.org/c/platform - Dani Megert If this granularity is sufficient, the mapping can be created manually. Currently, in addition to the developer's name I only list his/her role: "Project Lead". Ad (3): known culprits at this point: - org.eclipse.swt.tools - org.eclipse.osgi.compatibility.plugins Anyone have an idea why those don't have sources? Should I expect more exceptions?
The developers section probably doesn't even need specific people. You could probably just list "Eclipse Contributors" or "Eclipse Team". Surprised this is required, as it's a pain to maintain that section, and the Maven POM isn't the best place to record that info. I believe this is one of the fields not required for ASF projects, but is required by OSSRH.
(In reply to Christopher Tubbs from comment #72) > You could > probably just list "Eclipse Contributors" or "Eclipse Team". +1, I agree that we should avoid listing individuals.
(In reply to Lars Vogel from comment #73) > (In reply to Christopher Tubbs from comment #72) > > You could > > probably just list "Eclipse Contributors" or "Eclipse Team". > > +1, I agree that we should avoid listing individuals. I don't want to give incorrect information, and I think only real people can be developers. For the need to specify a <developers> element see http://central.sonatype.org/pages/requirements.html#developer-information and for its content see https://maven.apache.org/pom.html#Developers The following satisfies this rule as per validation on OSSRH: <developers> <developer> <name>Dani Megert</name> <roles> <role>Project Lead</role> </roles> </developer> </developers> Listing only project leads avoids the issue of tedious maintenance of data.
Here's another tiny anomaly: per the bundle symbolic name the following artifacts are classified as belonging to platform (groupId): - org.eclipse.ui.views.log - org.eclipse.ui.trace - org.eclipse.ltk.ui.refactoring - org.eclipse.ltk.core.refactoring (determined by not matching to JDT nor PDE). The source code of these artifacts is, however, kept in PDE (former 2) and JDT (latter 2) repositories. Unless s.o. sees a problem with this, I will continue going by the qualified name, ignoring the git-repo membership, i.e., the above will indeed by assigned to groupId org.eclipse.platform.
(In reply to Stephan Herrmann from comment #71) > (3) some artifacts are missing a source artifact > > > Ad (3): known culprits at this point: > - org.eclipse.swt.tools > - org.eclipse.osgi.compatibility.plugins > Anyone have an idea why those don't have sources? Should I expect more > exceptions? After a first full upload for org.eclipse.platform (took 1h) the only remaining validation errors complain about missing -sources artifacts. OSSRH complains about 8 such artifacts, but the full list should read: org.eclipse.ant.optional.junit-3.3.200-sources.jar org.eclipse.core.filesystem.aix.ppc-1.1.0-sources.jar org.eclipse.core.filesystem.aix.ppc64-1.1.0-sources.jar org.eclipse.core.filesystem.hpux.ia64-1.1.0-sources.jar org.eclipse.core.filesystem.linux.ppc-1.0.200-sources.jar org.eclipse.core.filesystem.linux.ppc64-1.4.0-sources.jar org.eclipse.core.filesystem.linux.ppc64le-1.4.0-sources.jar org.eclipse.core.filesystem.linux.x86-1.4.200-sources.jar org.eclipse.core.filesystem.linux.x86_64-1.2.200-sources.jar org.eclipse.core.filesystem.macosx-1.3.0-sources.jar org.eclipse.core.filesystem.win32.x86-1.4.0-sources.jar org.eclipse.core.filesystem.win32.x86_64-1.4.0-sources.jar org.eclipse.core.net.linux.x86-1.2.0-sources.jar org.eclipse.core.net.linux.x86_64-1.2.0-sources.jar org.eclipse.core.net.win32.x86-1.1.0-sources.jar org.eclipse.core.net.win32.x86_64-1.1.0-sources.jar org.eclipse.core.resources.win32.x86-3.5.100-sources.jar org.eclipse.core.resources.win32.x86_64-3.5.0-sources.jar org.eclipse.cvs-1.4.301-sources.jar org.eclipse.equinox.console.jaas.fragment-1.0.0-sources.jar org.eclipse.equinox.launcher.cocoa.macosx.x86_64-1.1.401-sources.jar org.eclipse.equinox.launcher.gtk.aix.ppc-1.1.401-sources.jar org.eclipse.equinox.launcher.gtk.aix.ppc64-1.1.401-sources.jar org.eclipse.equinox.launcher.gtk.hpux.ia64-1.1.401-sources.jar org.eclipse.equinox.launcher.gtk.linux.ppc-1.1.401-sources.jar org.eclipse.equinox.launcher.gtk.linux.ppc64-1.1.401-sources.jar org.eclipse.equinox.launcher.gtk.linux.ppc64le-1.1.401-sources.jar org.eclipse.equinox.launcher.gtk.linux.s390-1.1.401-sources.jar org.eclipse.equinox.launcher.gtk.linux.s390x-1.1.401-sources.jar org.eclipse.equinox.launcher.gtk.linux.x86-1.1.401-sources.jar org.eclipse.equinox.launcher.gtk.linux.x86_64-1.1.401-sources.jar org.eclipse.equinox.launcher.gtk.solaris.sparcv9-1.1.401-sources.jar org.eclipse.equinox.launcher.gtk.solaris.x86_64-1.1.401-sources.jar org.eclipse.equinox.launcher.win32.win32.x86-1.1.401-sources.jar org.eclipse.equinox.launcher.win32.win32.x86_64-1.1.401-sources.jar org.eclipse.osgi.compatibility.plugins-1.0.0-sources.jar org.eclipse.platform.doc.isv-4.6.2-sources.jar org.eclipse.platform.doc.user-4.6.1-sources.jar org.eclipse.rcp-4.6.2-sources.jar org.eclipse.releng.tools-3.9.0-sources.jar org.eclipse.sdk-4.6.2-sources.jar org.eclipse.swt-3.105.2-sources.jar org.eclipse.swt.tools-3.105.2-sources.jar org.eclipse.update.core-3.2.800-sources.jar ------------------------------------------------------------- Please: can anyone tell why these source bundles are missing? -------------------------------------------------------------
(In reply to Stephan Herrmann from comment #76) > (In reply to Stephan Herrmann from comment #71) > > (3) some artifacts are missing a source artifact > > > > > > Ad (3): known culprits at this point: > > - org.eclipse.swt.tools > > - org.eclipse.osgi.compatibility.plugins > > Anyone have an idea why those don't have sources? Should I expect more > > exceptions? > > After a first full upload for org.eclipse.platform (took 1h) the only > remaining validation errors complain about missing -sources artifacts. OSSRH > complains about 8 such artifacts, but the full list should read: > [...] Maybe OSSRH detects that the others don't contain .class files. At this point I decided to build sources jars on the fly for these artifacts: org.eclipse.core.net.win32.x86_64 org.eclipse.core.net.win32.x86 org.eclipse.core.net.linux.x86_64 org.eclipse.core.net.linux.x86 org.eclipse.swt.tools org.eclipse.releng.tools org.eclipse.update.core org.eclipse.osgi.compatibility.plugins While this involved more manual tweaks than is desirable on the long run, it served its purpose for now and I got a thumbs up from OSSRH validation (platform & jdt for now, pde to follow soon).
After consulting with Dani I changed the <developers> section to <developers> <developer> <url>https://projects.eclipse.org/projects/eclipse.jdt.core/who</url> </developer> </developers> etc. (to my own surprise OSSRH validation accepts this without a <name> element). Documentation of the full process can now be found in https://wiki.eclipse.org/Platform-releng/Publish_To_Maven_Central Last blocking issue: the upload for PDE fails with permission problems: Failed to execute goal org.apache.maven.plugins:maven-gpg-plugin:1.6:sign-and-deploy-file (default-cli) on project org.eclipse.pde.publish2maven: Failed to deploy artifacts: Could not transfer artifact org.eclipse.pde:org.eclipse.pde:jar:3.12.2 from/to ossrh (https://oss.sonatype.org/service/local/staging/deploy/maven2/): Failed to transfer file: https://oss.sonatype.org/service/local/staging/deploy/maven2/org/eclipse/pde/org.eclipse.pde/3.12.2/org.eclipse.pde-3.12.2.jar. Return code is: 401, ReasonPhrase: Unauthorized. -> [Help 1]
(In reply to Stephan Herrmann from comment #78) > After consulting with Dani I changed the <developers> section to > > <developers> > <developer> > <url>https://projects.eclipse.org/projects/eclipse.jdt.core/who</url> > </developer> > </developers> +1
(In reply to Stephan Herrmann from comment #78) > Last blocking issue: the upload for PDE fails with permission problems: > > Failed to execute goal > org.apache.maven.plugins:maven-gpg-plugin:1.6:sign-and-deploy-file > (default-cli) on project org.eclipse.pde.publish2maven: Failed to deploy > artifacts: Could not transfer artifact > org.eclipse.pde:org.eclipse.pde:jar:3.12.2 from/to ossrh > (https://oss.sonatype.org/service/local/staging/deploy/maven2/): Failed to > transfer file: > https://oss.sonatype.org/service/local/staging/deploy/maven2/org/eclipse/pde/ > org.eclipse.pde/3.12.2/org.eclipse.pde-3.12.2.jar. Return code is: 401, > ReasonPhrase: Unauthorized. -> [Help 1] The provided credentials work as expected. I can log into the OSSRH Nexus and also upload SNAPSHOT artifacts with curl, e.g.: curl -u pde-dev https://oss.sonatype.org/content/repositories/snapshots/org/eclipse/pde/pde-test/org.eclipse.pde.core-3.11.1.pom --request PUT --data org.eclipse.pde.core-3.11.1.pom I haven't figured out yet why release artifacts can't be deployed.
(In reply to Frederic Gurr from comment #80) > I haven't figured out yet why release artifacts can't be deployed. Are you saying you can reproduce the problem regarding release artifacts? FWIW, the full log of my latest attempt can be found in https://hudson.eclipse.org/releng/view/Publish%20to%20Maven%20Central/job/PublishPDEToMaven.Neon/ws/.log/artifact-upload.txt The full command was (e.g.): /shared/common/apache-maven-latest/bin/mvn -f pde-pom.xml -s /opt/public/hipp/homes/genie.releng/.m2/settings-deploy-ossrh-pde.xml gpg:sign-and-deploy-file -Durl=https://oss.sonatype.org/service/local/staging/deploy/maven2/ -DrepositoryId=ossrh -Dfile=org/eclipse/pde/org.eclipse.pde/3.12.2/org.eclipse.pde-3.12.2.jar -DpomFile=org/eclipse/pde/org.eclipse.pde/3.12.2/org.eclipse.pde-3.12.2.pom where pde-pom.xml is https://hudson.eclipse.org/releng/view/Publish%20to%20Maven%20Central/job/PublishPDEToMaven.Neon/ws/pde-pom.xml
Yes, I can reproduce your problem. From what I gathered in forums and stackoverflow, the 401 can be a quite generic error for multiple problems.
(In reply to Frederic Gurr from comment #82) > Yes, I can reproduce your problem. Thanks for confirming. > From what I gathered in forums and stackoverflow, the 401 can be a quite > generic error for multiple problems. Let me know if you have an idea of anything I can do / test. I just triple checked that the jobs for jdt and pde do exactly the same, just using different settings and working on different artifacts (which all result from the same previous job). Should we ask in their Jira what's different between these two accounts? If no resolution is in (near) sight, I might as well release what we already have: platform and jdt, WDYT?
(In reply to Stephan Herrmann from comment #83) > Should we ask in their Jira what's different between these two accounts? We can certainly ask to rule out problems on the OSSRH side. > If no resolution is in (near) sight, I might as well release what we already > have: platform and jdt, WDYT? Is it time-critical to release platform and jdt? If yes, then I don't object. I will do some more testing in the meantime.
(In reply to Frederic Gurr from comment #84) > (In reply to Stephan Herrmann from comment #83) > > Should we ask in their Jira what's different between these two accounts? > > We can certainly ask to rule out problems on the OSSRH side. > > > If no resolution is in (near) sight, I might as well release what we already > > have: platform and jdt, WDYT? > > Is it time-critical to release platform and jdt? If yes, then I don't object. Not critical by the minute, but I really want to avoid that some external Maven enthusiast beats us to it and starts publishing Neon.2 artifacts on his own. > I will do some more testing in the meantime. thanks
I managed to upload PDE artifacts (org.eclipse.pde) manually through the OSSRH Nexus web UI without any problems. So there is no general problem with PDE artifacts. I can't spot any difference between JDT and PDE that explains why PDE uploads return 401 errors and JDT doesn't, so I am at a loss here. @Mikael: do you have any idea?
From a quick look I don't see what to do. Maybe you should try to run maven with verbose output (-V -X -e).
(In reply to Mikaël Barbero from comment #87) > From a quick look I don't see what to do. Maybe you should try to run maven > with verbose output (-V -X -e). voila: https://hudson.eclipse.org/releng/view/Publish%20to%20Maven%20Central/job/PublishPDEToMaven.Neon/ws/.log/artifact-upload.txt I couldn't draw any new insights from this, though. (In reply to Frederic Gurr from comment #86) > I managed to upload PDE artifacts (org.eclipse.pde) manually through the > OSSRH Nexus web UI without any problems. So there is no general problem with > PDE artifacts. Alright, so for now I could try to just zip everything and manually upload it as a bundle.
Confusing version of org.eclipse.swt artifacts: from the output of the platform publish task it looks as if SWT artifacts will be published with a version of 3.105.2 which will probably be confusing to users that look up https://search.maven.org/#search%7Cga%7C1%7Corg.eclipse.swt and will there find artifacts with version 4.3. Is it planned to change this or will this stay as is? Best would be if Maven Central could clean up unofficial org.eclipse.swt artifacts - not sure if that is possible. Filtering them from search results would also be a great help. And also it's confusing that the version does not match what is called 4.6.2 on the download page: http://download.eclipse.org/eclipse/downloads/drops4/R-4.6.2-201611241400/#SWT Huge thanks for driving the Maven effort! Br, Christoph
(In reply to Christoph Obexer from comment #89) I'm not touching any version numbers :) (except for removing the 4th (qualifier) segment). Technically, Eclipse 4.6.2 contains SWT 3.105.2 (there never was a major incompatible change since 3.0.0). This implies, that any SWT uploads with versions higher than 3.105.2 are indeed incorrect, using the Eclipse version to mark an SWT artifact (and our download page supports this mistake indeed!). I believe it's a golden rule of Maven Central to never delete any artifact (s.o. might rely on it). So, I see no way we can really resolve this confusion. But by publishing into the groupId "org.eclipse.platform" the new "official" artifacts can at least be distinguished from previous uploads. It's probably a good idea to widely publish the new groupId and what the new poms look like, so that consumers can easily recognize the official ones (after we've pushed the button, that is).
I worked around the upload problem by manually uploading a jar containing all PDE artifacts, which after a few attempts validated OK on OSSRH. At that point I hit the buttons for all three projects and also requested enablement of rsync via the OSSRH tickets. Just for PDE I stumbled on the very last yard: when nexus processed my artifact bundle, it messed up the directory structure, see https://issues.sonatype.org/browse/OSSRH-27091?focusedCommentId=388865&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-388865 I hope this can be resolved on their machine (and via https://issues.sonatype.org/browse/OSSRH-27091). On the eclipse.org side I consider this task resolved. Follow-up work will happen in bug 510072 For the curious, the released stuff can be found here: https://oss.sonatype.org/content/groups/public/org/eclipse/platform/ https://oss.sonatype.org/content/groups/public/org/eclipse/jdt/ https://oss.sonatype.org/content/groups/public/org/eclipse/pde/org/eclipse/pde/api/tools/ee/osgiminimum12/1/0/300/org/eclipse/pde/ (the last link being the goofed-up structure <sigh>, but content OK) I'll blog about all this as soon as all has well arrived at Maven Central.
*** Bug 489900 has been marked as a duplicate of this bug. ***
*** Bug 502304 has been marked as a duplicate of this bug. ***
Some problems I found adding SWT. Not sure if I should add here or make a new ticket org.eclipse.platform:org.eclipse.swt:3.105.2 depends on "org.eclipse.swt.gtk.linux.aarch64" which doesn't exist. It already depends on linux.x86 and linux.x86_64 so I'm not sure what it means. org.eclipse.swt.gtk.linux.arm is also missing Was it intended for the platform packages to depend on org.eclipse.platform:org.eclipse.swt? Because by default that means all platforms are on the classpath causing loading errors. To do it properly with profiles I'd have to exclude org.eclipse.platform:org.eclipse.swt in each profile dependency.
(In reply to Leon Blakey from comment #94) > Some problems I found adding SWT. Not sure if I should add here or make a > new ticket > > org.eclipse.platform:org.eclipse.swt:3.105.2 depends on > "org.eclipse.swt.gtk.linux.aarch64" which doesn't exist. It already depends > on linux.x86 and linux.x86_64 so I'm not sure what it means. > org.eclipse.swt.gtk.linux.arm is also missing > > Was it intended for the platform packages to depend on > org.eclipse.platform:org.eclipse.swt? Because by default that means all > platforms are on the classpath causing loading errors. To do it properly > with profiles I'd have to exclude org.eclipse.platform:org.eclipse.swt in > each profile dependency. Related to bug 510072 comment 1 f. Let's address it via that bug, to make sure Neon.3 will be published in a better structure. Thanks.
(In reply to Leon Blakey from comment #94) > org.eclipse.platform:org.eclipse.swt:3.105.2 depends on > org.eclipse.swt.gtk.linux.arm is also missing I think *org.eclipse.swt.gtk.linux.arm* is bogus, as the last segment should represent the correct architecture identifier. I believe this was introduced in relation to bug 406749.
The little hick-up regarding the directory structure of PDE artifacts has been kindly resolved on OSSRH. A public announcement of the results can be found in https://objectteams.wordpress.com/2017/01/09/eclipse-neon-2-is-on-maven-central/ This is it. ("It" for Neon.2. I know we can still do better, but that will be for Neon.3, no longer under the heading "Start publishing ..."). Thanks to everybody who helped realize this.
Even though I double-checked it, there was a problem with the encrypted OSSRH password for PDE, that has been resolved. After fixing the password, I shortly questioned my sanity, because I still saw errors in the log files. Apparently the output of the mvn calls in publishPDE.sh get appended to the logs and the logs don't get cleared between runs. m( I've fixed this by enabling the "Clean workspace before build" option for now.
(In reply to Frederic Gurr from comment #98) > Even though I double-checked it, there was a problem with the encrypted > OSSRH password for PDE, that has been resolved. Thanks, that's great news. > After fixing the password, I shortly questioned my sanity, because I still > saw errors in the log files. Apparently the output of the mvn calls in > publishPDE.sh get appended to the logs and the logs don't get cleared > between runs. m( > I've fixed this by enabling the "Clean workspace before build" option for > now. Sorry, "clean workspace" was my intention, too (for the Publish* group of jobs), but must have forgotten for PDE. Also the logging is quite convoluted due to running maven extra verbosely, plus a few tolerated errors (some -sources.jar are sought but not present, because nothing there).
Most recent SWT on maven central is 3.105.3 from Neon.3 https://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.eclipse.platform%22%20AND%20a%3A%22org.eclipse.swt.win32.win32.x86_64%22 When will Oxygen make it to maven central?
(In reply to Ned Twigg from comment #100) > Most recent SWT on maven central is 3.105.3 from Neon.3 > > https://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.eclipse. > platform%22%20AND%20a%3A%22org.eclipse.swt.win32.win32.x86_64%22 > > When will Oxygen make it to maven central? Future uploads to maven central are handled via bug 510072
Hi, I noticed that jdt-core is now in Maven Central, which is great. Thanks for that. However, I was wondering if jsdt-core could also be pushed there sometime soon? For neon. It would save me a lot of trouble. Is there already an existing issue for this? If not, I can create one. I just wanted to check here first, since this is where most of the discussion seemed to be happening about this.
(In reply to Christopher Tubbs from comment #102) > Hi, I noticed that jdt-core is now in Maven Central, which is great. Thanks > for that. However, I was wondering if jsdt-core could also be pushed there > sometime soon? For neon. It would save me a lot of trouble. > > Is there already an existing issue for this? If not, I can create one. I > just wanted to check here first, since this is where most of the discussion > seemed to be happening about this. The Platform doesn't ship/own JSDT you have to report it against JSDT in bugzilla.
*** Bug 496062 has been marked as a duplicate of this bug. ***
Closing.
