[2026-01-07 14:33:09,305][ INFO][PID:3264968] Marking build as starting [2026-01-07 14:33:09,630][ INFO][PID:3264968] Checking that /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/repodata/repomd.xml exists [2026-01-07 14:33:09,637][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:33:09,638][ INFO][PID:3264968] VM allocation process starts [2026-01-07 14:33:09,650][ INFO][PID:3264968] Trying to allocate VM: ResallocHost, ticket_id=692088, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-07 14:33:12,701][ INFO][PID:3264968] Allocated host ResallocHost, ticket_id=692088, hostname=2620:52:6:1161:dead:beef:cafe:c148, name=vmhost_p09_03_prod_00991181_20260107_142626, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-07 14:33:12,701][ INFO][PID:3264968] Allocating ssh connection to builder [2026-01-07 14:33:12,702][ INFO][PID:3264968] Checking that builder machine is OK [2026-01-07 14:33:13,626][ INFO][PID:3264968] Running remote command: copr-builder-ready epel-10-ppc64le [2026-01-07 14:33:14,120][ INFO][PID:3264968] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-01-07 14:33:14,121][ INFO][PID:3264968] Filling build.info file with builder info [2026-01-07 14:33:14,121][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:33:14,122][ INFO][PID:3264968] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "osalbahr", "project_name": "mirafetch", "project_dirname": "mirafetch", "submitter": "https://src.fedoraproject.org/user/decathorpe", "ended_on": null, "started_on": 1767796394.122379, "submitted_on": null, "status": 3, "chroot": "epel-10-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9981874-epel-10-ppc64le", "build_id": 9981874, "package_name": "cargo2rpm", "package_version": "0.3.2-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osalbahr/mirafetch/cargo2rpm", "git_hash": "f1b2a1f835f643a84a778378d5da44b83ffe1576", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "osalbahr/mirafetch--https://src.fedoraproject.org/user/decathorpe", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/osalbahr/mirafetch/epel-10-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/osalbahr/mirafetch", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osalbahr/mirafetch", "result_dir": "09981874-cargo2rpm", "built_packages": "", "id": 9981874, "mockchain_macros": { "copr_username": "osalbahr", "copr_projectname": "mirafetch", "vendor": "Fedora Project COPR (osalbahr/mirafetch)" } } ] } [2026-01-07 14:33:14,507][ INFO][PID:3264968] Sending fedora-messaging bus message in build.start [2026-01-07 14:33:15,107][ INFO][PID:3264968] Sending fedora-messaging bus message in chroot.start [2026-01-07 14:33:15,137][ INFO][PID:3264968] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9981874-epel-10-ppc64le --chroot epel-10-ppc64le --detached [2026-01-07 14:33:16,097][ INFO][PID:3264968] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c148' (ED25519) to the list of known hosts. [2026-01-07 14:33:16,098][ INFO][PID:3264968] Downloading the builder-live.log file, attempt 1 [2026-01-07 14:33:16,099][ INFO][PID:3264968] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c148 copr-rpmbuild-log [2026-01-07 14:33:21,146][ INFO][PID:3264968] Periodic builder liveness probe: alive [2026-01-07 14:33:21,146][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:33:26,148][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:33:31,149][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:33:36,150][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:33:41,151][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:33:46,153][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:33:51,153][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:33:56,154][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:34:01,156][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:34:06,157][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:34:11,158][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:34:16,159][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:34:21,160][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:34:26,161][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:34:31,162][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:34:36,163][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:34:41,164][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:34:46,165][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:34:51,400][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:34:56,401][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:35:01,402][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:35:06,403][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:35:11,404][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:35:16,405][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:35:21,406][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:35:26,407][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:35:31,408][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:35:36,409][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:35:41,411][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:35:46,412][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:35:51,413][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:35:56,415][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:36:01,416][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:36:06,417][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:36:11,419][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:36:16,422][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:36:21,424][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:36:26,425][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:36:31,426][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:36:36,427][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:36:40,372][ ERROR][PID:3264968] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 14:36:41,429][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:36:46,431][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:36:50,374][WARNING][PID:3264968] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-07 14:36:51,434][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:36:56,436][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:37:00,375][ INFO][PID:3264968] Downloading the builder-live.log file, attempt 2 [2026-01-07 14:37:00,377][ INFO][PID:3264968] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c148 copr-rpmbuild-log [2026-01-07 14:37:01,440][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:37:06,442][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:37:11,446][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:37:16,448][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:37:21,451][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:37:26,452][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:37:31,456][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:37:36,458][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:37:41,464][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:37:46,470][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:37:51,483][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:37:56,486][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:38:00,450][ ERROR][PID:3264968] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 14:38:01,495][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:38:06,506][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:38:10,494][WARNING][PID:3264968] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-07 14:38:11,510][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:38:16,517][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:38:20,496][ INFO][PID:3264968] Downloading the builder-live.log file, attempt 3 [2026-01-07 14:38:20,499][ INFO][PID:3264968] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c148 copr-rpmbuild-log [2026-01-07 14:38:21,521][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:38:26,524][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:38:31,527][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:38:36,547][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:38:41,550][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:38:46,551][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:38:51,554][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:38:56,556][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:39:01,557][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:39:06,559][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:39:11,612][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:39:16,614][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:39:20,579][ ERROR][PID:3264968] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 14:39:21,618][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:39:26,622][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:39:30,580][WARNING][PID:3264968] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-07 14:39:31,623][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:39:36,624][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:39:40,630][ INFO][PID:3264968] Downloading the builder-live.log file, attempt 4 [2026-01-07 14:39:40,631][ INFO][PID:3264968] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c148 copr-rpmbuild-log [2026-01-07 14:39:41,656][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:39:46,657][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:39:51,658][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:39:56,660][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:01,661][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:06,663][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:11,665][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:16,666][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:21,668][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:26,669][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:31,671][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:36,672][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:40,700][ ERROR][PID:3264968] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 14:40:41,674][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:46,675][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:50,701][ ERROR][PID:3264968] Giving up for unstable SSH, failures: 1767796610.3742, 1767796690.494638, 1767796770.5805051, 1767796850.701042 [2026-01-07 14:40:50,702][ ERROR][PID:3264968] Re-try request for task on 'ResallocHost, ticket_id=692088, hostname=2620:52:6:1161:dead:beef:cafe:c148, name=vmhost_p09_03_prod_00991181_20260107_142626, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-07 14:40:50,702][ INFO][PID:3264968] Releasing VM back to pool [2026-01-07 14:40:50,712][ INFO][PID:3264968] Retry #1 (on other host) [2026-01-07 14:40:50,712][ INFO][PID:3264968] Checking that /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/repodata/repomd.xml exists [2026-01-07 14:40:50,721][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:50,722][ INFO][PID:3264968] VM allocation process starts [2026-01-07 14:40:50,773][ INFO][PID:3264968] Trying to allocate VM: ResallocHost, ticket_id=692268, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-07 14:40:53,817][ INFO][PID:3264968] Allocated host ResallocHost, ticket_id=692268, hostname=2620:52:6:1161:dead:beef:cafe:c18e, name=vmhost_p09_04_prod_00991694_20260107_143408, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-07 14:40:53,818][ INFO][PID:3264968] Allocating ssh connection to builder [2026-01-07 14:40:53,819][ INFO][PID:3264968] Checking that builder machine is OK [2026-01-07 14:40:54,888][ INFO][PID:3264968] Running remote command: copr-builder-ready epel-10-ppc64le [2026-01-07 14:40:55,435][ INFO][PID:3264968] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-01-07 14:40:55,438][ INFO][PID:3264968] Filling build.info file with builder info [2026-01-07 14:40:55,461][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:40:55,462][ INFO][PID:3264968] Not re-notifying FE and msg buses for the new host. [2026-01-07 14:40:55,463][ INFO][PID:3264968] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9981874-epel-10-ppc64le --chroot epel-10-ppc64le --detached [2026-01-07 14:40:56,466][ INFO][PID:3264968] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18e' (ED25519) to the list of known hosts. [2026-01-07 14:40:56,467][ INFO][PID:3264968] Downloading the builder-live.log file, attempt 1 [2026-01-07 14:40:56,469][ INFO][PID:3264968] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18e copr-rpmbuild-log [2026-01-07 14:41:01,468][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:41:06,469][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:41:11,470][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:41:16,473][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:41:21,474][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:41:26,476][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:41:31,478][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:41:36,479][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:41:41,480][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:41:46,482][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:41:51,483][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:41:56,484][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:42:01,485][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:42:06,486][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:42:11,487][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:42:16,488][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:42:21,489][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:42:26,493][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:42:31,495][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:42:36,496][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:42:41,497][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:42:46,499][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:42:51,500][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:42:56,501][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:43:01,502][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:43:06,503][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:43:11,504][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:43:16,505][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:43:21,514][ INFO][PID:3264968] Periodic builder liveness probe: alive [2026-01-07 14:43:21,515][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:43:26,518][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:43:31,521][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:43:36,523][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:43:41,524][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:43:46,524][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:43:51,546][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:43:56,550][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:44:01,551][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:44:06,552][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:44:11,553][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:44:16,554][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:44:21,555][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:44:26,556][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:44:31,557][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:44:36,558][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:44:41,559][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:44:46,560][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:44:51,561][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:44:56,562][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:45:01,563][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:45:06,564][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:45:11,565][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:45:16,567][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:45:21,568][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:45:26,569][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:45:31,570][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:45:36,571][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:45:41,572][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:45:46,573][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:45:51,574][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:45:56,575][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:46:01,576][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:46:06,577][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:46:11,578][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:46:16,579][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:46:21,581][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:46:26,582][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:46:29,369][ ERROR][PID:3264968] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 14:46:31,583][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:46:36,584][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:46:39,370][WARNING][PID:3264968] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-07 14:46:41,585][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:46:46,587][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:46:49,375][ INFO][PID:3264968] Downloading the builder-live.log file, attempt 2 [2026-01-07 14:46:49,377][ INFO][PID:3264968] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18e copr-rpmbuild-log [2026-01-07 14:46:51,589][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:46:56,590][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:01,591][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:06,593][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:11,594][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:16,595][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:21,596][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:26,601][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:31,602][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:36,604][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:41,606][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:46,613][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:49,444][ ERROR][PID:3264968] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 14:47:51,615][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:56,617][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:47:59,490][WARNING][PID:3264968] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-07 14:48:01,620][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:48:06,621][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:48:09,494][ INFO][PID:3264968] Downloading the builder-live.log file, attempt 3 [2026-01-07 14:48:09,496][ INFO][PID:3264968] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18e copr-rpmbuild-log [2026-01-07 14:48:11,622][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:48:16,623][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:48:21,624][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:48:26,626][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:48:31,629][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:48:36,631][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:48:41,632][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:48:46,633][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:48:51,634][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:48:56,636][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:49:01,640][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:49:06,642][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:49:09,566][ ERROR][PID:3264968] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 14:49:11,643][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:49:16,644][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:49:19,577][WARNING][PID:3264968] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-07 14:49:21,649][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:49:26,650][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:49:29,582][ INFO][PID:3264968] Downloading the builder-live.log file, attempt 4 [2026-01-07 14:49:29,588][ INFO][PID:3264968] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18e copr-rpmbuild-log [2026-01-07 14:49:31,654][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:49:36,656][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:49:41,668][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:49:46,669][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:49:51,678][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:49:56,685][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:50:01,689][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:50:06,690][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:50:11,695][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:50:16,698][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:50:21,700][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:50:26,701][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:50:29,673][ ERROR][PID:3264968] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-07 14:50:31,702][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:50:36,706][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:50:39,678][ ERROR][PID:3264968] Giving up for unstable SSH, failures: 1767797199.3707426, 1767797279.4908917, 1767797359.577076, 1767797439.6787019 [2026-01-07 14:50:39,679][ ERROR][PID:3264968] Re-try request for task on 'ResallocHost, ticket_id=692268, hostname=2620:52:6:1161:dead:beef:cafe:c18e, name=vmhost_p09_04_prod_00991694_20260107_143408, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-07 14:50:39,680][ INFO][PID:3264968] Releasing VM back to pool [2026-01-07 14:50:39,691][ INFO][PID:3264968] Retry #2 (on other host) [2026-01-07 14:50:39,691][ INFO][PID:3264968] Checking that /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/repodata/repomd.xml exists [2026-01-07 14:50:39,692][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:50:39,692][ INFO][PID:3264968] VM allocation process starts [2026-01-07 14:50:39,704][ INFO][PID:3264968] Trying to allocate VM: ResallocHost, ticket_id=692569, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-07 14:50:42,720][ INFO][PID:3264968] Allocated host ResallocHost, ticket_id=692569, hostname=2620:52:6:1161:dead:beef:cafe:c14d, name=vmhost_p09_03_prod_00992244_20260107_144209, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9'] [2026-01-07 14:50:42,720][ INFO][PID:3264968] Allocating ssh connection to builder [2026-01-07 14:50:42,721][ INFO][PID:3264968] Checking that builder machine is OK [2026-01-07 14:50:43,636][ INFO][PID:3264968] Running remote command: copr-builder-ready epel-10-ppc64le [2026-01-07 14:50:44,173][ INFO][PID:3264968] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2026-01-07 14:50:44,173][ INFO][PID:3264968] Filling build.info file with builder info [2026-01-07 14:50:44,202][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:50:44,203][ INFO][PID:3264968] Not re-notifying FE and msg buses for the new host. [2026-01-07 14:50:44,204][ INFO][PID:3264968] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9981874-epel-10-ppc64le --chroot epel-10-ppc64le --detached [2026-01-07 14:50:45,089][ INFO][PID:3264968] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c14d' (ED25519) to the list of known hosts. [2026-01-07 14:50:45,090][ INFO][PID:3264968] Downloading the builder-live.log file, attempt 1 [2026-01-07 14:50:45,091][ INFO][PID:3264968] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14d copr-rpmbuild-log [2026-01-07 14:50:50,090][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:50:55,091][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:51:00,092][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:51:05,094][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:51:10,095][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:51:15,096][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:51:20,097][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:51:25,098][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:51:30,099][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:51:35,105][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:51:40,109][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:51:45,110][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:51:50,116][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:51:55,120][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:52:00,121][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:52:05,124][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:52:10,125][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:52:15,128][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:52:20,130][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:52:25,135][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:52:30,140][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:52:35,143][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:52:40,147][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:52:45,150][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:52:50,156][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:52:55,160][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:53:00,163][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:53:05,167][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:53:10,173][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:53:15,175][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:53:20,179][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:53:25,223][ INFO][PID:3264968] Periodic builder liveness probe: alive [2026-01-07 14:53:25,228][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:53:30,240][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:53:35,311][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:53:40,315][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:53:45,318][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:53:50,321][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:53:55,326][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:54:00,334][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:54:05,345][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:54:10,349][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:54:15,353][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:54:20,356][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:54:25,362][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:54:30,367][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:54:35,372][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:54:40,375][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:54:45,377][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:54:50,380][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:54:55,383][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:55:00,385][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:55:05,387][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:55:10,389][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:55:15,390][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:55:20,391][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:55:25,394][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:55:30,397][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:55:35,399][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:55:40,401][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:55:45,404][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:55:50,407][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:55:55,410][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:56:00,411][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:56:05,412][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:56:10,413][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:56:15,414][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:56:20,416][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:56:25,417][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:56:30,435][ INFO][PID:3264968] Checking for cancel request [2026-01-07 14:56:34,918][ INFO][PID:3264968] Downloading results from builder [2026-01-07 14:56:34,919][ INFO][PID:3264968] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c14d]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm started [2026-01-07 14:56:34,920][ INFO][PID:3264968] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c14d]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm/ &> /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm/build-09981874.rsync.log [2026-01-07 14:56:35,510][ INFO][PID:3264968] rsyncing finished. [2026-01-07 14:56:35,511][ INFO][PID:3264968] Releasing VM back to pool [2026-01-07 14:56:35,522][ INFO][PID:3264968] Searching for 'success' file in resultdir [2026-01-07 14:56:35,523][ INFO][PID:3264968] Going to sign pkgs from source: 9981874-epel-10-ppc64le in chroot: /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le [2026-01-07 14:56:35,523][ INFO][PID:3264968] Calling '/bin/sign -u osalbahr#mirafetch@copr.fedorahosted.org -p' (attempt #1) [2026-01-07 14:56:35,784][ INFO][PID:3264968] Calling '/bin/sign -4 -h sha256 -u osalbahr#mirafetch@copr.fedorahosted.org -r /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm/cargo2rpm-0.3.2-1.el10.src.rpm' (attempt #1) [2026-01-07 14:56:36,583][ INFO][PID:3264968] signed rpm: /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm/cargo2rpm-0.3.2-1.el10.src.rpm [2026-01-07 14:56:36,584][ INFO][PID:3264968] Calling '/bin/sign -4 -h sha256 -u osalbahr#mirafetch@copr.fedorahosted.org -r /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm/cargo2rpm-0.3.2-1.el10.noarch.rpm' (attempt #1) [2026-01-07 14:56:37,392][ INFO][PID:3264968] signed rpm: /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm/cargo2rpm-0.3.2-1.el10.noarch.rpm [2026-01-07 14:56:37,392][ INFO][PID:3264968] Sign done [2026-01-07 14:56:37,393][ INFO][PID:3264968] Incremental createrepo run, adding 09981874-cargo2rpm into https://download.copr.fedorainfracloud.org/results/osalbahr/mirafetch/epel-10-ppc64le, (auto-create-repo=True) [2026-01-07 14:56:37,394][ INFO][PID:3264968] Running command 'copr-repo --batched /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le --add 09981874-cargo2rpm --no-appstream-metadata' as PID 3467376 [2026-01-07 14:56:38,079][ INFO][PID:3264968] Finished after 0 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le --add 09981874-cargo2rpm --no-appstream-metadata) [2026-01-07 14:56:38,080][ INFO][PID:3264968] Getting build details [2026-01-07 14:56:38,082][ INFO][PID:3264968] Listing built binary packages in /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm [2026-01-07 14:56:38,083][ INFO][PID:3264968] Built packages: ['cargo2rpm 0.3.2'] [2026-01-07 14:56:38,083][ INFO][PID:3264968] build details: {'built_packages': 'cargo2rpm 0.3.2'} [2026-01-07 14:56:38,084][ INFO][PID:3264968] Retrieving pubkey [2026-01-07 14:56:38,084][ INFO][PID:3264968] Calling '/bin/sign -u osalbahr#mirafetch@copr.fedorahosted.org -p' (attempt #1) [2026-01-07 14:56:38,357][ INFO][PID:3264968] Added pubkey for user osalbahr project mirafetch into: /var/lib/copr/public_html/results/osalbahr/mirafetch/pubkey.gpg [2026-01-07 14:56:38,358][ INFO][PID:3264968] Finished build: id=9981874 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/osalbahr/mirafetch chroot=epel-10-ppc64le [2026-01-07 14:56:38,359][ INFO][PID:3264968] Worker succeeded build, took 354.15611577033997 [2026-01-07 14:56:38,360][ INFO][PID:3264968] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "osalbahr", "project_name": "mirafetch", "project_dirname": "mirafetch", "submitter": "https://src.fedoraproject.org/user/decathorpe", "ended_on": 1767797798.3595798, "started_on": 1767797444.203464, "submitted_on": null, "status": 1, "chroot": "epel-10-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9981874-epel-10-ppc64le", "build_id": 9981874, "package_name": "cargo2rpm", "package_version": "0.3.2-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osalbahr/mirafetch/cargo2rpm", "git_hash": "f1b2a1f835f643a84a778378d5da44b83ffe1576", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "osalbahr/mirafetch--https://src.fedoraproject.org/user/decathorpe", "results": { "packages": [ { "name": "cargo2rpm", "epoch": null, "version": "0.3.2", "release": "1.el10", "arch": "src" }, { "name": "cargo2rpm", "epoch": null, "version": "0.3.2", "release": "1.el10", "arch": "noarch" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/osalbahr/mirafetch/epel-10-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/osalbahr/mirafetch", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osalbahr/mirafetch", "result_dir": "09981874-cargo2rpm", "built_packages": "cargo2rpm 0.3.2", "id": 9981874, "mockchain_macros": { "copr_username": "osalbahr", "copr_projectname": "mirafetch", "vendor": "Fedora Project COPR (osalbahr/mirafetch)" } } ] } [2026-01-07 14:56:38,503][ INFO][PID:3264968] Sending fedora-messaging bus message in build.end [2026-01-07 14:56:38,538][ INFO][PID:3264968] Compressing /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm/builder-live.log by gzip [2026-01-07 14:56:38,540][ INFO][PID:3264968] Running command 'gzip /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm/builder-live.log' as PID 3467550 [2026-01-07 14:56:38,567][ INFO][PID:3264968] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm/builder-live.log) [2026-01-07 14:56:38,568][ INFO][PID:3264968] Compressing /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm/backend.log by gzip [2026-01-07 14:56:38,570][ INFO][PID:3264968] Running command 'gzip /var/lib/copr/public_html/results/osalbahr/mirafetch/epel-10-ppc64le/09981874-cargo2rpm/backend.log' as PID 3467555