1586748 - No GV Nightly since Oct 04: beetmover jobs aren't part of the graph anymore

Status: VERIFIED FIXED

(Firefox Build System :: Task Configuration, defect, P1)

Description

[Johan Lorenzo [:jlorenzo]]

The latest ship_geckoview jobs[1] don't have any beetmover jobs anymore, causing maven.mozilla.org to not serve the latest geckoview nightlies.

This is caused by lines like [3] (which were landed as part of bug 1580356): the beetmover jobs depend on some signed bits, otherwise they get filtered out.

[1] For instance: https://tools.taskcluster.net/groups/MzzN5YhUSlaP9lYvgeaGcQ
[2] https://maven.mozilla.org/?prefix=maven2/org/mozilla/geckoview/geckoview-nightly-x86_64/
[3] https://hg.mozilla.org/mozilla-central/rev/1ffe2e3d6e19a351575ac986c43155ed8e8d90bc#l3416.57

Comment 1

[Nick Alexander :nalexander [he/him]]

jlorenzo: so I agree this is "caused" by the lines that you link, but what is to be done? The BM jobs very carefully arrange to depend on both build and build-signing jobs, and at this time there's no sensible notion of "signing" mobile/android jobs. (We'd be signing an AAR. It would make more sense to sign the old target.maven.zip, but that's gone. Is there any advantage to signing an AAR? Is the signature even part of the BM job?). It seems to me that BM should move the results of the Ngv job rather than muck around with quasi-signed artifacts, but I don't have strong opinions here. Thoughts?

Comment 2

[Johan Lorenzo [:jlorenzo]]

I'm sorry for the lack of context of my first comment. Starting bug 1548545, we've been gpg-signing maven artifacts, including AARs[1]. See this beetmover task definition[2], as an example:

    {
      "locale": "en-US",
      "paths": {
        "public/build/maven/org/mozilla/geckoview/geckoview-nightly-armeabi-v7a/71.0.20191002033852/geckoview-nightly-armeabi-v7a-71.0.20191002033852-javadoc.jar.asc": {
          "checksums_path": "geckoview-nightly-armeabi-v7a-71.0.20191002033852-javadoc.jar.asc",
          "destinations": [
            "maven2/org/mozilla/geckoview/geckoview-nightly-armeabi-v7a/71.0.20191002033852/geckoview-nightly-armeabi-v7a-71.0.20191002033852-javadoc.jar.asc"
          ]
        },
        "public/build/maven/org/mozilla/geckoview/geckoview-nightly-armeabi-v7a/71.0.20191002033852/geckoview-nightly-armeabi-v7a-71.0.20191002033852-sources.jar.asc": {
          "checksums_path": "geckoview-nightly-armeabi-v7a-71.0.20191002033852-sources.jar.asc",
          "destinations": [
            "maven2/org/mozilla/geckoview/geckoview-nightly-armeabi-v7a/71.0.20191002033852/geckoview-nightly-armeabi-v7a-71.0.20191002033852-sources.jar.asc"
          ]
        },
        "public/build/maven/org/mozilla/geckoview/geckoview-nightly-armeabi-v7a/71.0.20191002033852/geckoview-nightly-armeabi-v7a-71.0.20191002033852.aar.asc": {
          "checksums_path": "geckoview-nightly-armeabi-v7a-71.0.20191002033852.aar.asc",
          "destinations": [
            "maven2/org/mozilla/geckoview/geckoview-nightly-armeabi-v7a/71.0.20191002033852/geckoview-nightly-armeabi-v7a-71.0.20191002033852.aar.asc"
          ]
        },
        "public/build/maven/org/mozilla/geckoview/geckoview-nightly-armeabi-v7a/71.0.20191002033852/geckoview-nightly-armeabi-v7a-71.0.20191002033852.pom.asc": {
          "checksums_path": "geckoview-nightly-armeabi-v7a-71.0.20191002033852.pom.asc",
          "destinations": [
            "maven2/org/mozilla/geckoview/geckoview-nightly-armeabi-v7a/71.0.20191002033852/geckoview-nightly-armeabi-v7a-71.0.20191002033852.pom.asc"
          ]
        }
      },
      "taskId": "XzYkpxoSQP-8vu1xwBm-Jw"
    }

This is the reason why the beetmover jobs depend on build-signing. To me, the fix is to remove the disable-build-signing: true lines from:

• android-api-16-nightly/opt.
• android-aarch64-nightly/opt
• android-x86_64-nightly/opt
• android-x86-nightly/opt

Please let me know if you need more information 🙂

[1] For instance: https://maven.mozilla.org/?prefix=maven2/org/mozilla/geckoview/geckoview-nightly-x86_64/71.0.20191004094656/, see the .asc files.
[2] https://tools.taskcluster.net/groups/Yi9DecphRRWvoJY15d_Pkg/tasks/drbDj483Q5W07Y2J894q6A/details

Comment 3

[Nick Alexander :nalexander [he/him]]

Try build with the simple backout is percolating at https://treeherder.mozilla.org/#/jobs?repo=try&revision=2c734fe28520b7b366f9fde446831b0515b8e774. My guess is that we want to remove the beetmover kind entirely, unless it has application across trees. There are no Fennec beets remaining!

Comment 4

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1586748 - Re-enable build-signing for N and Ngv tasks. r?jlorenzo

Bug 1580356, which removed Fennec from mozilla-central, was too
aggressive. That ticket reasoned that since there's no longer an APK
to sign, that build-signing should be disabled. However,
build-signing is more general: since Bug 1548545, GeckoView artifacts
have been GPG signed. The beetmover tasks expect those signed
artifacts, and without them, nothing gets published to
maven.mozilla.org.

This simply reverts the parts of Bug 1580356 that removed the
build-signing tasks from the taskgraph.

Comment 5

[Nick Alexander :nalexander [he/him]]

jlorenzo: the patch you pushed to try (https://hg.mozilla.org/try/rev/b4e2404d6d381e2254c18e1cf09093113079d9d6) makes sense to me. I'd go further and remove fennec_* and the beetmover kind -- if removing the APK handling is okay (across repositories) then removing this must be okay too --but that could also be follow-up. Can you commandeer the revision so we can get this landed?

Comment 6

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1586748 - Re-enable build-signing for N and Ngv tasks and clean up unused fennec manifests/kinds r=nalexander

Comment 7

[Johan Lorenzo [:jlorenzo]]

👍

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c100236e73c5
Re-enable build-signing for N and Ngv tasks and clean up unused fennec manifests/kinds r=mtabara

Comment 10

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/c100236e73c5

Comment 11

[Nick Alexander :nalexander [he/him]]

And we're back, folks: I see a BM-GV job and I see that https://maven.mozilla.org/?prefix=maven2/org/mozilla/geckoview/geckoview-nightly/71.0.20191011093010/ is looking happy and healthy.

Great work, Johan and Mihai!