[2026-01-06 23:22:28,178][ INFO][PID:3042498] Marking build as starting [2026-01-06 23:22:28,341][ INFO][PID:3042498] Checking that /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-9-ppc64le/repodata/repomd.xml exists [2026-01-06 23:22:28,345][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:22:28,348][ INFO][PID:3042498] VM allocation process starts [2026-01-06 23:22:28,376][ INFO][PID:3042498] Trying to allocate VM: ResallocHost, ticket_id=682499, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-06 23:22:31,399][ INFO][PID:3042498] Allocated host ResallocHost, ticket_id=682499, hostname=2620:52:6:1161:dead:beef:cafe:c154, name=vmhost_p09_03_prod_00973431_20260106_231539, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-06 23:22:31,400][ INFO][PID:3042498] Allocating ssh connection to builder [2026-01-06 23:22:31,400][ INFO][PID:3042498] Checking that builder machine is OK [2026-01-06 23:22:32,447][ INFO][PID:3042498] Running remote command: copr-builder-ready centos-stream-9-ppc64le [2026-01-06 23:22:32,476][ ERROR][PID:3042498] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c154 port 22: Permission denied [2026-01-06 23:22:42,476][ ERROR][PID:3042498] Re-try request for task on 'ResallocHost, ticket_id=682499, hostname=2620:52:6:1161:dead:beef:cafe:c154, name=vmhost_p09_03_prod_00973431_20260106_231539, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts [2026-01-06 23:22:42,477][ INFO][PID:3042498] Releasing VM back to pool [2026-01-06 23:22:42,488][ INFO][PID:3042498] Retry #1 (on other host) [2026-01-06 23:22:42,489][ INFO][PID:3042498] Checking that /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-9-ppc64le/repodata/repomd.xml exists [2026-01-06 23:22:42,489][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:22:42,490][ INFO][PID:3042498] VM allocation process starts [2026-01-06 23:22:42,502][ INFO][PID:3042498] Trying to allocate VM: ResallocHost, ticket_id=682516, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-06 23:22:45,520][ INFO][PID:3042498] Allocated host ResallocHost, ticket_id=682516, hostname=2620:52:6:1161:dead:beef:cafe:c18a, name=vmhost_p09_04_prod_00973194_20260106_225223, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-06 23:22:45,520][ INFO][PID:3042498] Allocating ssh connection to builder [2026-01-06 23:22:45,521][ INFO][PID:3042498] Checking that builder machine is OK [2026-01-06 23:23:45,564][ ERROR][PID:3042498] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c18a port 22: Connection timed out [2026-01-06 23:23:55,579][ ERROR][PID:3042498] Re-try request for task on 'ResallocHost, ticket_id=682516, hostname=2620:52:6:1161:dead:beef:cafe:c18a, name=vmhost_p09_04_prod_00973194_20260106_225223, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts [2026-01-06 23:23:55,619][ INFO][PID:3042498] Releasing VM back to pool [2026-01-06 23:23:55,659][ INFO][PID:3042498] Retry #2 (on other host) [2026-01-06 23:23:55,659][ INFO][PID:3042498] Checking that /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-9-ppc64le/repodata/repomd.xml exists [2026-01-06 23:23:55,693][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:23:55,694][ INFO][PID:3042498] VM allocation process starts [2026-01-06 23:23:55,704][ INFO][PID:3042498] Trying to allocate VM: ResallocHost, ticket_id=682518, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-06 23:23:58,716][ INFO][PID:3042498] Allocated host ResallocHost, ticket_id=682518, hostname=2620:52:6:1161:dead:beef:cafe:c18d, name=vmhost_p09_04_prod_00973452_20260106_231914, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-06 23:23:58,716][ INFO][PID:3042498] Allocating ssh connection to builder [2026-01-06 23:23:58,717][ INFO][PID:3042498] Checking that builder machine is OK [2026-01-06 23:23:59,598][ INFO][PID:3042498] Running remote command: copr-builder-ready centos-stream-9-ppc64le [2026-01-06 23:24:00,115][ INFO][PID:3042498] Red Hat subscription not needed for centos-stream-9-ppc64le Builder is ready to be used [2026-01-06 23:24:00,116][ INFO][PID:3042498] Filling build.info file with builder info [2026-01-06 23:24:00,151][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:24:00,152][ INFO][PID:3042498] Not re-notifying FE and msg buses for the new host. [2026-01-06 23:24:00,153][ INFO][PID:3042498] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9979891-centos-stream-9-ppc64le --chroot centos-stream-9-ppc64le --detached [2026-01-06 23:24:01,087][ INFO][PID:3042498] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18d' (ED25519) to the list of known hosts. [2026-01-06 23:24:01,088][ INFO][PID:3042498] Downloading the builder-live.log file, attempt 1 [2026-01-06 23:24:01,089][ INFO][PID:3042498] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18d copr-rpmbuild-log [2026-01-06 23:24:06,094][ INFO][PID:3042498] Periodic builder liveness probe: alive [2026-01-06 23:24:06,095][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:24:11,096][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:24:16,097][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:24:21,112][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:24:26,113][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:24:31,114][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:24:36,115][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:24:41,116][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:24:46,118][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:24:51,119][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:24:56,121][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:25:01,123][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:25:06,124][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:25:11,125][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:25:16,126][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:25:21,128][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:25:26,129][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:25:31,130][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:25:36,131][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:25:41,132][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:25:46,133][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:25:51,134][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:25:56,135][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:26:01,136][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:26:06,204][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:26:11,205][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:26:16,206][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:26:21,207][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:26:26,208][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:26:31,237][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:26:36,238][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:26:41,300][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:26:46,407][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:26:51,408][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:26:56,409][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:27:01,410][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:27:06,411][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:27:11,412][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:27:16,412][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:27:21,413][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:27:26,414][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:27:31,415][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:27:36,416][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:27:41,417][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:27:46,418][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:27:51,419][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:27:56,424][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:28:01,426][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:28:06,427][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:28:11,428][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:28:16,429][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:28:21,430][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:28:26,434][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:28:31,435][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:28:36,436][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:28:41,437][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:28:46,438][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:28:51,439][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:28:56,578][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:01,597][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:06,598][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:11,599][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:16,600][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:21,601][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:26,602][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:31,603][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:36,605][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:41,610][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:46,610][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:51,612][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:56,645][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:29:57,351][ ERROR][PID:3042498] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-06 23:30:01,963][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:30:06,966][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:30:07,352][WARNING][PID:3042498] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-06 23:30:11,969][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:30:16,970][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:30:17,352][ INFO][PID:3042498] Downloading the builder-live.log file, attempt 2 [2026-01-06 23:30:17,354][ INFO][PID:3042498] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18d copr-rpmbuild-log [2026-01-06 23:30:21,971][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:30:26,973][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:30:31,974][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:30:36,976][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:30:41,977][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:30:46,978][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:30:51,979][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:30:56,981][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:31:01,982][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:31:06,984][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:31:11,985][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:31:16,986][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:31:17,421][ ERROR][PID:3042498] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-06 23:31:21,991][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:31:27,116][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:31:27,438][WARNING][PID:3042498] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-06 23:31:32,118][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:31:37,121][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:31:37,439][ INFO][PID:3042498] Downloading the builder-live.log file, attempt 3 [2026-01-06 23:31:37,441][ INFO][PID:3042498] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18d copr-rpmbuild-log [2026-01-06 23:31:42,123][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:31:47,125][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:31:52,127][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:31:57,128][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:02,129][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:07,130][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:12,131][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:17,133][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:22,134][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:27,135][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:32,136][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:37,137][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:37,502][ ERROR][PID:3042498] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-06 23:32:42,240][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:47,241][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:47,515][WARNING][PID:3042498] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-06 23:32:52,242][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:57,243][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:32:57,515][ INFO][PID:3042498] Downloading the builder-live.log file, attempt 4 [2026-01-06 23:32:57,517][ INFO][PID:3042498] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18d copr-rpmbuild-log [2026-01-06 23:33:02,245][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:33:07,246][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:33:12,247][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:33:17,253][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:33:22,254][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:33:27,305][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:33:32,307][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:33:37,336][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:33:42,351][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:33:47,355][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:33:52,391][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:33:57,408][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:33:57,584][ ERROR][PID:3042498] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-06 23:34:02,415][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:34:07,428][ INFO][PID:3042498] Periodic builder liveness probe: alive [2026-01-06 23:34:07,428][ INFO][PID:3042498] Checking for cancel request [2026-01-06 23:34:07,604][ ERROR][PID:3042498] Giving up for unstable SSH, failures: 1767742207.35216, 1767742287.4387708, 1767742367.5151038, 1767742447.6040537 [2026-01-06 23:34:07,605][ ERROR][PID:3042498] Re-try request for task on 'ResallocHost, ticket_id=682518, hostname=2620:52:6:1161:dead:beef:cafe:c18d, name=vmhost_p09_04_prod_00973452_20260106_231914, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-06 23:34:07,605][ INFO][PID:3042498] Releasing VM back to pool [2026-01-06 23:34:07,615][ ERROR][PID:3042498] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c18d', '2620:52:6:1161:dead:beef:cafe:c18a', '2620:52:6:1161:dead:beef:cafe:c154'} [2026-01-06 23:34:07,616][WARNING][PID:3042498] Switching not-finished job state to 'failed' [2026-01-06 23:34:07,616][ INFO][PID:3042498] Worker failed build, took 607.4637351036072 [2026-01-06 23:34:07,617][ INFO][PID:3042498] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "rhcontainerbot", "project_name": "bootc", "project_dirname": "bootc", "submitter": "packit", "ended_on": 1767742447.6162963, "started_on": 1767741840.1525612, "submitted_on": null, "status": 0, "chroot": "centos-stream-9-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9979891-centos-stream-9-ppc64le", "build_id": 9979891, "package_name": "bootc", "package_version": "202601062318.g2a4476c6a8-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/bootc/bootc", "git_hash": "74b47ba47d9be29178f4e150cc80568533444b89", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "rhcontainerbot/bootc--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-9-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/rhcontainerbot/bootc", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc", "result_dir": "09979891-bootc", "built_packages": "", "id": 9979891, "mockchain_macros": { "copr_username": "rhcontainerbot", "copr_projectname": "bootc", "vendor": "Fedora Project COPR (rhcontainerbot/bootc)" } } ] } [2026-01-06 23:34:07,684][ INFO][PID:3042498] Sending fedora-messaging bus message in build.end [2026-01-06 23:34:08,255][ INFO][PID:3042498] Compressing /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-9-ppc64le/09979891-bootc/builder-live.log by gzip [2026-01-06 23:34:08,257][ INFO][PID:3042498] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-9-ppc64le/09979891-bootc/builder-live.log' as PID 3094458 [2026-01-06 23:34:08,258][ INFO][PID:3042498] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-9-ppc64le/09979891-bootc/builder-live.log) [2026-01-06 23:34:08,258][ INFO][PID:3042498] Compressing /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-9-ppc64le/09979891-bootc/backend.log by gzip [2026-01-06 23:34:08,259][ INFO][PID:3042498] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-9-ppc64le/09979891-bootc/backend.log' as PID 3094459