[2025-04-03 17:26:18,971][  INFO][PID:2440976] Marking build as starting
[2025-04-03 17:26:19,228][  INFO][PID:2440976] Trying to allocate VM: ResallocHost, ticket_id=2211108, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-03 17:26:47,577][  INFO][PID:2440976] Allocated host ResallocHost, ticket_id=2211108, hostname=140.211.168.14, name=copr_osuosl_p09_prod_02713415_20250403_171948, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-03 17:26:47,580][  INFO][PID:2440976] Allocating ssh connection to builder
[2025-04-03 17:26:47,584][  INFO][PID:2440976] Checking that builder machine is OK
[2025-04-03 17:26:48,909][  INFO][PID:2440976] Running remote command: copr-builder-ready fedora-42-ppc64le
[2025-04-03 17:26:50,023][  INFO][PID:2440976] Red Hat subscription not needed for fedora-42-ppc64le
Builder is ready to be used

[2025-04-03 17:26:50,025][  INFO][PID:2440976] Filling build.info file with builder info
[2025-04-03 17:26:50,029][  INFO][PID:2440976] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "packit",
            "project_name": "osbuild-image-builder-cli-186",
            "project_dirname": "osbuild-image-builder-cli-186",
            "submitter": "packit",
            "ended_on": null,
            "started_on": 1743701210.0294082,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-42-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8858879-fedora-42-ppc64le",
            "build_id": 8858879,
            "package_name": "image-builder",
            "package_version": "15-1.20250403172553888496.pr186.4.g37cd637",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/packit/osbuild-image-builder-cli-186/image-builder",
            "git_hash": "f8567b2e97f970114627770c987b5987b74ed11f",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "packit/osbuild-image-builder-cli-186--packit",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/packit/osbuild-image-builder-cli-186",
            "result_dir": "08858879-image-builder",
            "built_packages": "",
            "id": 8858879,
            "mockchain_macros": {
                "copr_username": "packit",
                "copr_projectname": "osbuild-image-builder-cli-186",
                "vendor": "Fedora Project COPR (packit/osbuild-image-builder-cli-186)"
            }
        }
    ]
}
[2025-04-03 17:26:50,113][  INFO][PID:2440976] Sending fedora-messaging bus message in build.start
[2025-04-03 17:26:52,012][  INFO][PID:2440976] Sending fedora-messaging bus message in chroot.start
[2025-04-03 17:26:52,064][  INFO][PID:2440976] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8858879-fedora-42-ppc64le --chroot fedora-42-ppc64le --detached
[2025-04-03 17:26:53,477][  INFO][PID:2440976] Downloading the builder-live.log file, attempt 1
[2025-04-03 17:26:53,485][  INFO][PID:2440976] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log
[2025-04-03 17:26:58,539][  INFO][PID:2440976] Periodic builder liveness probe: alive
[2025-04-03 17:36:59,008][  INFO][PID:2440976] Periodic builder liveness probe: alive
[2025-04-03 17:46:59,247][  INFO][PID:2440976] Periodic builder liveness probe: alive
[2025-04-03 17:56:59,386][  INFO][PID:2440976] Periodic builder liveness probe: alive
[2025-04-03 18:06:59,605][  INFO][PID:2440976] Periodic builder liveness probe: alive
[2025-04-03 18:12:19,140][ ERROR][PID:2440976] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-03 18:12:29,153][WARNING][PID:2440976] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-03 18:12:39,158][  INFO][PID:2440976] Downloading the builder-live.log file, attempt 2
[2025-04-03 18:12:39,171][  INFO][PID:2440976] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log
[2025-04-03 18:13:39,252][ ERROR][PID:2440976] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-03 18:13:49,255][WARNING][PID:2440976] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-03 18:13:59,257][  INFO][PID:2440976] Downloading the builder-live.log file, attempt 3
[2025-04-03 18:13:59,264][  INFO][PID:2440976] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log
[2025-04-03 18:14:59,349][ ERROR][PID:2440976] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-03 18:15:09,429][WARNING][PID:2440976] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-03 18:15:19,431][  INFO][PID:2440976] Downloading the builder-live.log file, attempt 4
[2025-04-03 18:15:19,449][  INFO][PID:2440976] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log
[2025-04-03 18:16:19,527][ ERROR][PID:2440976] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-03 18:16:29,554][ ERROR][PID:2440976] Giving up for unstable SSH, failures: 1743703949.1536555, 1743704029.2557938, 1743704109.4294379, 1743704189.5541055
[2025-04-03 18:16:29,556][ ERROR][PID:2440976] Re-try request for task on 'ResallocHost, ticket_id=2211108, hostname=140.211.168.14, name=copr_osuosl_p09_prod_02713415_20250403_171948, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-04-03 18:16:29,557][  INFO][PID:2440976] Releasing VM back to pool
[2025-04-03 18:16:29,574][  INFO][PID:2440976] Retry #1 (on other host)
[2025-04-03 18:16:29,641][  INFO][PID:2440976] Trying to allocate VM: ResallocHost, ticket_id=2212385, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-03 18:16:35,692][  INFO][PID:2440976] Allocated host ResallocHost, ticket_id=2212385, hostname=2620:52:3:1:dead:beef:cafe:c28d, name=copr_p09_01_prod_02714442_20250403_175103, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-03 18:16:35,693][  INFO][PID:2440976] Allocating ssh connection to builder
[2025-04-03 18:16:35,694][  INFO][PID:2440976] Checking that builder machine is OK
[2025-04-03 18:16:36,711][  INFO][PID:2440976] Running remote command: copr-builder-ready fedora-42-ppc64le
[2025-04-03 18:16:37,267][  INFO][PID:2440976] Red Hat subscription not needed for fedora-42-ppc64le
Builder is ready to be used

[2025-04-03 18:16:37,272][  INFO][PID:2440976] Filling build.info file with builder info
[2025-04-03 18:16:37,286][  INFO][PID:2440976] Not re-notifying FE and msg buses for the new host.
[2025-04-03 18:16:37,287][  INFO][PID:2440976] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8858879-fedora-42-ppc64le --chroot fedora-42-ppc64le --detached
[2025-04-03 18:16:38,251][  INFO][PID:2440976] Downloading the builder-live.log file, attempt 1
[2025-04-03 18:16:38,260][  INFO][PID:2440976] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28d copr-rpmbuild-log
[2025-04-03 18:17:03,300][  INFO][PID:2440976] Periodic builder liveness probe: alive
[2025-04-03 18:22:50,805][  INFO][PID:2440976] Downloading results from builder
[2025-04-03 18:22:50,810][  INFO][PID:2440976] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c28d]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder started
[2025-04-03 18:22:50,811][  INFO][PID:2440976] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c28d]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/ &> /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/build-08858879.rsync.log
[2025-04-03 18:22:55,880][  INFO][PID:2440976] rsyncing finished.
[2025-04-03 18:22:55,881][  INFO][PID:2440976] Releasing VM back to pool
[2025-04-03 18:22:55,904][  INFO][PID:2440976] Searching for 'success' file in resultdir
[2025-04-03 18:22:55,905][  INFO][PID:2440976] Going to sign pkgs from source: 8858879-fedora-42-ppc64le in chroot: /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le
[2025-04-03 18:22:55,906][  INFO][PID:2440976] Calling '/bin/sign -u packit#osbuild-image-builder-cli-186@copr.fedorahosted.org -p' (attempt #1)
[2025-04-03 18:22:56,081][  INFO][PID:2440976] Calling '/bin/sign -4 -h sha256 -u packit#osbuild-image-builder-cli-186@copr.fedorahosted.org -r /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/image-builder-debuginfo-15-1.20250403172553888496.pr186.4.g37cd637.fc42.ppc64le.rpm' (attempt #1)
[2025-04-03 18:22:58,449][  INFO][PID:2440976] signed rpm: /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/image-builder-debuginfo-15-1.20250403172553888496.pr186.4.g37cd637.fc42.ppc64le.rpm
[2025-04-03 18:22:58,450][  INFO][PID:2440976] Calling '/bin/sign -4 -h sha256 -u packit#osbuild-image-builder-cli-186@copr.fedorahosted.org -r /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/image-builder-15-1.20250403172553888496.pr186.4.g37cd637.fc42.src.rpm' (attempt #1)
[2025-04-03 18:23:00,727][  INFO][PID:2440976] signed rpm: /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/image-builder-15-1.20250403172553888496.pr186.4.g37cd637.fc42.src.rpm
[2025-04-03 18:23:00,728][  INFO][PID:2440976] Calling '/bin/sign -4 -h sha256 -u packit#osbuild-image-builder-cli-186@copr.fedorahosted.org -r /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/image-builder-debugsource-15-1.20250403172553888496.pr186.4.g37cd637.fc42.ppc64le.rpm' (attempt #1)
[2025-04-03 18:23:02,386][  INFO][PID:2440976] signed rpm: /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/image-builder-debugsource-15-1.20250403172553888496.pr186.4.g37cd637.fc42.ppc64le.rpm
[2025-04-03 18:23:02,387][  INFO][PID:2440976] Calling '/bin/sign -4 -h sha256 -u packit#osbuild-image-builder-cli-186@copr.fedorahosted.org -r /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/image-builder-15-1.20250403172553888496.pr186.4.g37cd637.fc42.ppc64le.rpm' (attempt #1)
[2025-04-03 18:23:04,800][  INFO][PID:2440976] signed rpm: /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/image-builder-15-1.20250403172553888496.pr186.4.g37cd637.fc42.ppc64le.rpm
[2025-04-03 18:23:04,801][  INFO][PID:2440976] Sign done
[2025-04-03 18:23:04,801][  INFO][PID:2440976] Incremental createrepo run, adding 08858879-image-builder into https://download.copr.fedorainfracloud.org/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le, (auto-create-repo=True)
[2025-04-03 18:23:04,802][  INFO][PID:2440976] Running command 'copr-repo --batched /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le --add 08858879-image-builder --no-appstream-metadata' as PID 2916171
[2025-04-03 18:23:05,578][  INFO][PID:2440976] Finished after 0 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le --add 08858879-image-builder --no-appstream-metadata)
[2025-04-03 18:23:05,579][  INFO][PID:2440976] Getting build details
[2025-04-03 18:23:05,579][  INFO][PID:2440976] Listing built binary packages in /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder
[2025-04-03 18:23:05,580][  INFO][PID:2440976] Built packages:
['image-builder-debuginfo 15', 'image-builder-debugsource 15', 'image-builder 15']
[2025-04-03 18:23:05,580][  INFO][PID:2440976] build details: {'built_packages': 'image-builder-debuginfo 15\nimage-builder-debugsource 15\nimage-builder 15'}
[2025-04-03 18:23:05,581][  INFO][PID:2440976] Retrieving pubkey
[2025-04-03 18:23:05,581][  INFO][PID:2440976] Calling '/bin/sign -u packit#osbuild-image-builder-cli-186@copr.fedorahosted.org -p' (attempt #1)
[2025-04-03 18:23:05,932][  INFO][PID:2440976] Added pubkey for user packit project osbuild-image-builder-cli-186 into: /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/pubkey.gpg
[2025-04-03 18:23:05,933][  INFO][PID:2440976] Finished build: id=8858879 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186 chroot=fedora-42-ppc64le 
[2025-04-03 18:23:05,934][  INFO][PID:2440976] Worker succeeded build, took 388.64849615097046
[2025-04-03 18:23:05,935][  INFO][PID:2440976] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "packit",
            "project_name": "osbuild-image-builder-cli-186",
            "project_dirname": "osbuild-image-builder-cli-186",
            "submitter": "packit",
            "ended_on": 1743704585.9348342,
            "started_on": 1743704197.286338,
            "submitted_on": null,
            "status": 1,
            "chroot": "fedora-42-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8858879-fedora-42-ppc64le",
            "build_id": 8858879,
            "package_name": "image-builder",
            "package_version": "15-1.20250403172553888496.pr186.4.g37cd637",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/packit/osbuild-image-builder-cli-186/image-builder",
            "git_hash": "f8567b2e97f970114627770c987b5987b74ed11f",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "packit/osbuild-image-builder-cli-186--packit",
            "results": {
                "packages": [
                    {
                        "name": "image-builder",
                        "epoch": null,
                        "version": "15",
                        "release": "1.20250403172553888496.pr186.4.g37cd637.fc42",
                        "arch": "src"
                    },
                    {
                        "name": "image-builder-debuginfo",
                        "epoch": null,
                        "version": "15",
                        "release": "1.20250403172553888496.pr186.4.g37cd637.fc42",
                        "arch": "ppc64le"
                    },
                    {
                        "name": "image-builder-debugsource",
                        "epoch": null,
                        "version": "15",
                        "release": "1.20250403172553888496.pr186.4.g37cd637.fc42",
                        "arch": "ppc64le"
                    },
                    {
                        "name": "image-builder",
                        "epoch": null,
                        "version": "15",
                        "release": "1.20250403172553888496.pr186.4.g37cd637.fc42",
                        "arch": "ppc64le"
                    }
                ]
            },
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/packit/osbuild-image-builder-cli-186",
            "result_dir": "08858879-image-builder",
            "built_packages": "image-builder-debuginfo 15\nimage-builder-debugsource 15\nimage-builder 15",
            "id": 8858879,
            "mockchain_macros": {
                "copr_username": "packit",
                "copr_projectname": "osbuild-image-builder-cli-186",
                "vendor": "Fedora Project COPR (packit/osbuild-image-builder-cli-186)"
            }
        }
    ]
}
[2025-04-03 18:23:05,981][  INFO][PID:2440976] Sending fedora-messaging bus message in build.end
[2025-04-03 18:23:05,997][  INFO][PID:2440976] Compressing /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/builder-live.log by gzip
[2025-04-03 18:23:05,998][  INFO][PID:2440976] Running command 'gzip /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/builder-live.log' as PID 2916266
[2025-04-03 18:23:06,108][  INFO][PID:2440976] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/builder-live.log)
[2025-04-03 18:23:06,109][  INFO][PID:2440976] Compressing /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/backend.log by gzip
[2025-04-03 18:23:06,110][  INFO][PID:2440976] Running command 'gzip /var/lib/copr/public_html/results/packit/osbuild-image-builder-cli-186/fedora-42-ppc64le/08858879-image-builder/backend.log' as PID 2916274