[2026-01-07 20:51:47,156][ INFO][PID:1405786] Marking build as starting [2026-01-07 20:51:47,801][ INFO][PID:1405786] Checking that /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/repodata/repomd.xml exists [2026-01-07 20:51:47,930][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:51:47,930][ INFO][PID:1405786] VM allocation process starts [2026-01-07 20:51:47,942][ INFO][PID:1405786] Trying to allocate VM: ResallocHost, ticket_id=697808, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-07 20:51:52,943][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:51:57,944][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:52:02,945][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:52:07,946][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:52:12,947][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:52:17,948][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:52:22,949][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:52:27,950][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:52:32,951][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:52:37,952][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:52:42,953][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:52:47,954][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:52:52,955][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:52:57,956][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:53:02,972][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:53:07,974][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:53:12,975][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:53:16,049][ INFO][PID:1405786] Allocated host ResallocHost, ticket_id=697808, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_01003622_20260107_204841, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-07 20:53:16,051][ INFO][PID:1405786] Allocating ssh connection to builder [2026-01-07 20:53:16,052][ INFO][PID:1405786] Checking that builder machine is OK [2026-01-07 20:53:17,003][ INFO][PID:1405786] Running remote command: copr-builder-ready centos-stream-10-ppc64le [2026-01-07 20:53:17,528][ INFO][PID:1405786] Red Hat subscription not needed for centos-stream-10-ppc64le Builder is ready to be used [2026-01-07 20:53:17,528][ INFO][PID:1405786] Filling build.info file with builder info [2026-01-07 20:53:17,615][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:53:17,616][ INFO][PID:1405786] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "keithbowes", "project_name": "deponejo", "project_dirname": "deponejo", "submitter": "keithbowes", "ended_on": null, "started_on": 1767819197.6163726, "submitted_on": null, "status": 3, "chroot": "centos-stream-10-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9982860-centos-stream-10-ppc64le", "build_id": 9982860, "package_name": "socat", "package_version": "1:1.8.1.0-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/keithbowes/deponejo/socat", "git_hash": "975bd8f711ebf44f63031f44df6b54c5434d2efd", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "keithbowes/deponejo--keithbowes", "results": null, "appstream": true, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/keithbowes/deponejo/centos-stream-10-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/keithbowes/deponejo", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/keithbowes/deponejo", "result_dir": "09982860-socat", "built_packages": "", "id": 9982860, "mockchain_macros": { "copr_username": "keithbowes", "copr_projectname": "deponejo", "vendor": "Fedora Project COPR (keithbowes/deponejo)" } } ] } [2026-01-07 20:53:17,700][ INFO][PID:1405786] Sending fedora-messaging bus message in build.start [2026-01-07 20:53:18,315][ INFO][PID:1405786] Sending fedora-messaging bus message in chroot.start [2026-01-07 20:53:18,346][ INFO][PID:1405786] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9982860-centos-stream-10-ppc64le --chroot centos-stream-10-ppc64le --detached [2026-01-07 20:53:19,351][ INFO][PID:1405786] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c191' (ED25519) to the list of known hosts. [2026-01-07 20:53:19,352][ INFO][PID:1405786] Downloading the builder-live.log file, attempt 1 [2026-01-07 20:53:19,354][ INFO][PID:1405786] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c191 copr-rpmbuild-log [2026-01-07 20:53:24,360][ INFO][PID:1405786] Periodic builder liveness probe: alive [2026-01-07 20:53:24,361][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:53:29,362][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:53:34,363][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:53:39,364][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:53:44,365][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:53:49,368][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:53:54,385][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:53:59,387][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:54:04,389][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:54:09,390][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:54:14,394][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:54:19,396][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:54:24,401][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:54:29,402][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:54:34,404][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:54:39,406][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:54:44,408][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:54:49,409][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:54:54,410][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:54:59,412][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:55:04,413][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:55:09,414][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:55:14,415][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:55:19,417][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:55:24,418][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:55:29,419][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:55:34,420][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:55:39,421][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:55:44,422][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:55:49,423][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:55:54,424][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:55:59,425][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:56:04,426][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:56:09,428][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:56:14,429][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:56:19,430][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:56:24,369][ ERROR][PID:1405786] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 20:56:24,431][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:56:29,432][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:56:34,371][WARNING][PID:1405786] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-07 20:56:34,433][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:56:39,434][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:56:44,372][ INFO][PID:1405786] Downloading the builder-live.log file, attempt 2 [2026-01-07 20:56:44,373][ INFO][PID:1405786] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c191 copr-rpmbuild-log [2026-01-07 20:56:44,435][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:56:49,436][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:56:54,437][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:56:59,438][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:57:04,439][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:57:09,440][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:57:14,441][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:57:19,442][ INFO][PID:1405786] Checking for cancel request [2026-01-07 20:57:19,825][ INFO][PID:1405786] Downloading results from builder [2026-01-07 20:57:19,826][ INFO][PID:1405786] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat started [2026-01-07 20:57:19,827][ INFO][PID:1405786] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat/ &> /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat/build-09982860.rsync.log [2026-01-07 20:58:19,904][ INFO][PID:1405786] rsyncing finished. [2026-01-07 20:58:19,904][ ERROR][PID:1405786] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-07 20:58:19,905][ ERROR][PID:1405786] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-07 20:58:29,905][ INFO][PID:1405786] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat started [2026-01-07 20:58:29,906][ INFO][PID:1405786] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat/ &> /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat/build-09982860.rsync.log [2026-01-07 20:59:29,991][ INFO][PID:1405786] rsyncing finished. [2026-01-07 20:59:29,992][ ERROR][PID:1405786] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-07 20:59:29,992][ ERROR][PID:1405786] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-07 20:59:39,993][ INFO][PID:1405786] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat started [2026-01-07 20:59:39,994][ INFO][PID:1405786] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c191]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat/ &> /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat/build-09982860.rsync.log [2026-01-07 21:00:40,073][ INFO][PID:1405786] rsyncing finished. [2026-01-07 21:00:40,074][ ERROR][PID:1405786] Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-07 21:00:40,074][ ERROR][PID:1405786] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2026-01-07 21:00:50,075][ ERROR][PID:1405786] Re-try request for task on 'ResallocHost, ticket_id=697808, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_01003622_20260107_204841, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 3 SSH attempts [2026-01-07 21:00:50,076][ INFO][PID:1405786] Releasing VM back to pool [2026-01-07 21:00:50,084][ INFO][PID:1405786] Retry #1 (on other host) [2026-01-07 21:00:50,085][ INFO][PID:1405786] Checking that /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/repodata/repomd.xml exists [2026-01-07 21:00:50,085][ INFO][PID:1405786] Checking for cancel request [2026-01-07 21:00:50,086][ INFO][PID:1405786] VM allocation process starts [2026-01-07 21:00:50,130][ INFO][PID:1405786] Trying to allocate VM: ResallocHost, ticket_id=697917, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-07 21:00:53,171][ INFO][PID:1405786] Allocated host ResallocHost, ticket_id=697917, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_01003622_20260107_204841, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-07 21:00:53,809][ INFO][PID:1405786] Allocating ssh connection to builder [2026-01-07 21:00:53,809][ INFO][PID:1405786] Checking that builder machine is OK [2026-01-07 21:01:53,873][ ERROR][PID:1405786] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c191 port 22: Connection timed out [2026-01-07 21:02:03,874][ ERROR][PID:1405786] Re-try request for task on 'ResallocHost, ticket_id=697917, hostname=2620:52:6:1161:dead:beef:cafe:c191, name=vmhost_p09_04_prod_01003622_20260107_204841, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts [2026-01-07 21:02:03,874][ INFO][PID:1405786] Releasing VM back to pool [2026-01-07 21:02:03,883][ INFO][PID:1405786] Retry #2 (on other host) [2026-01-07 21:02:03,883][ INFO][PID:1405786] Checking that /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/repodata/repomd.xml exists [2026-01-07 21:02:03,884][ INFO][PID:1405786] Checking for cancel request [2026-01-07 21:02:03,884][WARNING][PID:1405786] Canceling the build early [2026-01-07 21:02:03,885][ ERROR][PID:1405786] Build was canceled [2026-01-07 21:02:03,885][WARNING][PID:1405786] Switching not-finished job state to 'failed' [2026-01-07 21:02:03,886][ INFO][PID:1405786] Worker failed build, took 526.2692852020264 [2026-01-07 21:02:03,886][ INFO][PID:1405786] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "keithbowes", "project_name": "deponejo", "project_dirname": "deponejo", "submitter": "keithbowes", "ended_on": 1767819723.8856578, "started_on": 1767819197.6163726, "submitted_on": null, "status": 0, "chroot": "centos-stream-10-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9982860-centos-stream-10-ppc64le", "build_id": 9982860, "package_name": "socat", "package_version": "1:1.8.1.0-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/keithbowes/deponejo/socat", "git_hash": "975bd8f711ebf44f63031f44df6b54c5434d2efd", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "keithbowes/deponejo--keithbowes", "results": null, "appstream": true, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/keithbowes/deponejo/centos-stream-10-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/keithbowes/deponejo", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/keithbowes/deponejo", "result_dir": "09982860-socat", "built_packages": "", "id": 9982860, "mockchain_macros": { "copr_username": "keithbowes", "copr_projectname": "deponejo", "vendor": "Fedora Project COPR (keithbowes/deponejo)" } } ] } [2026-01-07 21:02:03,991][ INFO][PID:1405786] Sending fedora-messaging bus message in build.end [2026-01-07 21:02:04,020][ INFO][PID:1405786] Compressing /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat/builder-live.log by gzip [2026-01-07 21:02:04,022][ INFO][PID:1405786] Running command 'gzip /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat/builder-live.log' as PID 1462175 [2026-01-07 21:02:04,029][ INFO][PID:1405786] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat/builder-live.log) [2026-01-07 21:02:04,029][ INFO][PID:1405786] Compressing /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat/backend.log by gzip [2026-01-07 21:02:04,031][ INFO][PID:1405786] Running command 'gzip /var/lib/copr/public_html/results/keithbowes/deponejo/centos-stream-10-ppc64le/09982860-socat/backend.log' as PID 1462177