[2026-01-04 16:43:49,123][ INFO][PID:4122915] Marking build as starting [2026-01-04 16:43:49,431][ INFO][PID:4122915] Checking that /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/repodata/repomd.xml exists [2026-01-04 16:43:49,488][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:43:49,489][ INFO][PID:4122915] VM allocation process starts [2026-01-04 16:43:49,499][ INFO][PID:4122915] Trying to allocate VM: ResallocHost, ticket_id=650916, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-04 16:43:52,516][ INFO][PID:4122915] Allocated host ResallocHost, ticket_id=650916, hostname=2620:52:6:1161:dead:beef:cafe:c150, name=vmhost_p09_03_prod_00912904_20260104_154010, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-04 16:43:52,516][ INFO][PID:4122915] Allocating ssh connection to builder [2026-01-04 16:43:52,517][ INFO][PID:4122915] Checking that builder machine is OK [2026-01-04 16:43:53,485][ INFO][PID:4122915] Running remote command: copr-builder-ready fedora-rawhide-ppc64le [2026-01-04 16:43:54,027][ INFO][PID:4122915] Red Hat subscription not needed for fedora-rawhide-ppc64le Builder is ready to be used [2026-01-04 16:43:54,028][ INFO][PID:4122915] Filling build.info file with builder info [2026-01-04 16:43:54,029][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:43:54,029][ INFO][PID:4122915] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "topazus", "project_name": "test-review", "project_dirname": "test-review", "submitter": "topazus", "ended_on": null, "started_on": 1767545034.0297728, "submitted_on": null, "status": 3, "chroot": "fedora-rawhide-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9974088-fedora-rawhide-ppc64le", "build_id": 9974088, "package_name": "dtk6declarative", "package_version": "6.0.47-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/topazus/test-review/dtk6declarative", "git_hash": "b7740b236301f0a22b844ca5f9ad5751ed0fab54", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "topazus/test-review--topazus", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test-review/fedora-rawhide-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test-review/fedora-rawhide-ppc64le/", "id": "copr_topazus_test_review", "name": "Additional repo copr_topazus_test_review" } ], "background": false, "fedora_review": true, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/topazus/test-review", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/topazus/test-review", "result_dir": "09974088-dtk6declarative", "built_packages": "", "id": 9974088, "mockchain_macros": { "copr_username": "topazus", "copr_projectname": "test-review", "vendor": "Fedora Project COPR (topazus/test-review)" } } ] } [2026-01-04 16:43:54,086][ INFO][PID:4122915] Sending fedora-messaging bus message in build.start [2026-01-04 16:43:54,584][ INFO][PID:4122915] Sending fedora-messaging bus message in chroot.start [2026-01-04 16:43:54,611][ INFO][PID:4122915] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9974088-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached [2026-01-04 16:43:55,636][ INFO][PID:4122915] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c150' (ED25519) to the list of known hosts. [2026-01-04 16:43:55,637][ INFO][PID:4122915] Downloading the builder-live.log file, attempt 1 [2026-01-04 16:43:55,639][ INFO][PID:4122915] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c150 copr-rpmbuild-log [2026-01-04 16:44:00,643][ INFO][PID:4122915] Periodic builder liveness probe: alive [2026-01-04 16:44:00,644][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:44:05,645][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:44:10,646][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:44:15,646][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:44:20,647][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:44:25,648][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:44:30,649][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:44:35,650][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:44:40,651][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:44:45,652][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:44:50,653][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:44:55,654][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:44:55,704][ ERROR][PID:4122915] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-04 16:45:00,655][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:45:05,656][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:45:05,704][WARNING][PID:4122915] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-04 16:45:10,657][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:45:15,658][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:45:15,705][ INFO][PID:4122915] Downloading the builder-live.log file, attempt 2 [2026-01-04 16:45:15,706][ INFO][PID:4122915] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c150 copr-rpmbuild-log [2026-01-04 16:45:20,659][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:45:25,660][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:45:30,747][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:45:35,748][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:45:40,749][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:45:45,750][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:45:50,751][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:45:55,752][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:46:00,753][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:46:05,754][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:46:10,755][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:46:15,755][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:46:15,773][ ERROR][PID:4122915] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-04 16:46:20,757][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:46:25,758][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:46:25,921][WARNING][PID:4122915] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-04 16:46:30,759][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:46:35,889][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:46:35,921][ INFO][PID:4122915] Downloading the builder-live.log file, attempt 3 [2026-01-04 16:46:36,313][ INFO][PID:4122915] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c150 copr-rpmbuild-log [2026-01-04 16:46:41,312][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:46:46,313][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:46:51,314][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:46:56,315][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:47:01,316][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:47:06,317][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:47:11,318][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:47:16,319][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:47:21,399][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:47:26,416][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:47:31,450][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:47:36,451][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:47:41,452][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:47:46,563][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:47:51,688][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:47:56,693][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:48:01,693][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:48:06,915][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:48:12,143][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:48:17,144][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:48:22,145][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:48:27,146][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:48:32,147][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:48:37,148][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:48:42,149][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:48:47,149][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:48:52,151][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:48:57,153][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:49:02,154][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:49:07,155][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:49:12,256][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:49:17,258][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:49:22,280][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:49:27,281][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:49:32,282][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:49:37,283][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:49:42,617][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:49:47,618][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:49:52,619][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:49:57,621][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:50:02,622][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:50:07,623][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:50:12,624][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:50:17,625][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:50:22,627][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:50:27,628][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:50:32,647][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:50:37,648][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:50:42,649][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:50:47,650][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:50:52,651][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:50:57,652][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:51:02,653][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:51:07,653][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:51:12,654][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:51:17,655][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:51:22,656][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:51:27,657][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:51:32,715][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:51:38,219][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:51:43,220][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:51:48,221][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:51:53,221][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:51:58,222][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:52:03,223][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:52:08,224][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:52:13,225][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:52:18,226][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:52:23,227][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:52:28,228][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:52:33,229][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:52:38,230][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:52:43,230][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:52:48,231][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:52:53,232][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:52:58,233][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:53:03,239][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:53:08,240][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:53:13,241][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:53:18,242][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:53:23,242][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:53:28,275][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:53:33,308][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:53:38,309][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:53:43,347][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:53:48,349][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:53:53,371][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:53:58,481][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:54:03,487][ INFO][PID:4122915] Periodic builder liveness probe: alive [2026-01-04 16:54:03,667][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:54:08,667][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:54:13,668][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:54:18,669][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:54:23,670][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:54:28,671][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:54:33,672][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:54:38,673][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:54:43,674][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:54:48,675][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:54:53,693][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:54:58,694][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:55:03,695][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:55:08,696][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:55:13,697][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:55:18,698][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:55:23,775][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:55:28,783][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:55:33,784][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:55:38,856][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:55:43,899][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:55:48,901][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:55:53,944][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:55:59,165][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:56:04,171][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:56:09,220][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:56:14,321][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:56:19,322][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:56:24,323][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:56:29,656][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:56:34,657][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:56:39,659][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:56:44,660][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:56:49,661][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:56:54,662][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:56:59,663][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:57:04,665][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:57:09,696][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:57:14,698][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:57:19,708][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:57:24,710][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:57:29,717][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:57:34,780][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:57:39,781][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:57:44,782][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:57:49,783][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:57:54,784][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:57:59,785][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:58:04,786][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:58:09,787][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:58:14,798][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:58:19,818][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:58:24,819][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:58:29,820][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:58:34,821][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:58:39,927][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:58:44,928][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:58:49,929][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:58:54,972][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:58:59,973][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:59:04,974][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:59:10,521][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:59:15,522][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:59:20,523][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:59:25,524][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:59:30,525][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:59:35,526][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:59:40,527][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:59:45,528][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:59:50,793][ INFO][PID:4122915] Checking for cancel request [2026-01-04 16:59:55,795][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:00:01,158][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:00:06,159][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:00:11,160][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:00:16,744][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:00:21,745][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:00:26,746][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:00:31,747][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:00:36,930][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:00:42,221][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:00:47,223][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:00:52,225][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:00:57,227][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:01:02,229][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:01:07,230][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:01:12,232][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:01:17,371][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:01:22,378][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:01:27,380][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:01:32,599][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:01:37,600][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:01:42,601][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:01:47,602][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:01:52,640][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:01:57,641][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:02:02,642][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:02:07,643][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:02:12,644][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:02:17,645][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:02:22,646][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:02:27,647][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:02:32,647][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:02:37,648][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:02:42,649][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:02:47,651][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:02:52,651][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:02:57,652][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:03:02,653][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:03:07,654][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:03:12,655][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:03:17,656][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:03:22,657][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:03:27,658][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:03:32,659][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:03:37,660][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:03:42,661][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:03:47,662][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:03:52,663][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:03:57,664][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:04:02,665][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:04:07,670][ INFO][PID:4122915] Periodic builder liveness probe: alive [2026-01-04 17:04:07,670][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:04:12,671][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:04:17,672][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:04:22,673][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:04:27,674][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:04:32,675][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:04:37,676][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:04:42,676][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:04:47,677][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:04:52,678][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:04:57,679][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:05:02,680][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:05:07,681][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:05:12,681][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:05:17,682][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:05:22,683][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:05:27,684][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:05:32,685][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:05:37,686][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:05:42,687][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:05:47,688][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:05:52,689][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:05:57,690][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:06:02,691][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:06:07,692][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:06:12,693][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:06:17,693][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:06:22,694][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:06:27,695][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:06:32,696][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:06:37,697][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:06:42,698][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:06:47,699][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:06:52,700][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:06:57,701][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:07:02,702][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:07:07,703][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:07:12,704][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:07:17,705][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:07:22,706][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:07:27,707][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:07:32,709][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:07:37,710][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:07:42,712][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:07:47,713][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:07:52,714][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:07:57,715][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:08:02,716][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:08:07,717][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:08:12,718][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:08:17,719][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:08:22,720][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:08:27,721][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:08:32,722][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:08:37,723][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:08:42,724][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:08:47,725][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:08:52,726][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:08:57,727][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:09:02,728][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:09:07,729][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:09:12,730][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:09:17,731][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:09:22,732][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:09:27,733][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:09:32,734][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:09:37,735][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:09:42,736][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:09:47,737][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:09:52,738][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:09:57,739][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:10:02,740][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:10:07,741][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:10:12,742][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:10:17,743][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:10:22,744][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:10:27,745][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:10:32,746][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:10:37,746][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:10:42,747][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:10:47,748][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:10:52,749][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:10:57,750][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:11:02,751][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:11:07,752][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:11:12,753][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:11:17,754][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:11:22,755][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:11:27,756][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:11:32,757][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:11:37,758][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:11:42,758][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:11:47,759][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:11:52,760][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:11:57,761][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:12:02,762][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:12:07,763][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:12:12,764][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:12:17,765][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:12:22,766][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:12:27,766][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:12:32,767][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:12:37,768][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:12:42,769][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:12:47,770][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:12:52,771][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:12:57,772][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:13:02,773][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:13:07,774][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:13:12,774][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:13:17,775][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:13:22,776][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:13:27,777][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:13:32,778][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:13:37,779][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:13:42,780][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:13:47,781][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:13:52,782][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:13:57,782][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:14:02,783][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:14:07,789][ INFO][PID:4122915] Periodic builder liveness probe: dead [2026-01-04 17:14:07,790][ INFO][PID:4122915] calling cancel callback, and waiting [2026-01-04 17:14:08,979][WARNING][PID:4122915] Can't cancel build out: err: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c150' (ED25519) to the list of known hosts. Traceback (most recent call last): File "/usr/bin/copr-rpmbuild-cancel", line 85, in _main() ~~~~~^^ File "/usr/bin/copr-rpmbuild-cancel", line 69, in _main with open(pidfile, "r") as pidfd: ~~~~^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/copr-rpmbuild/pid' [2026-01-04 17:14:10,828][ ERROR][PID:4122915] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-04 17:14:20,829][WARNING][PID:4122915] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-04 17:14:30,829][ INFO][PID:4122915] Downloading the builder-live.log file, attempt 4 [2026-01-04 17:14:30,866][ INFO][PID:4122915] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c150 copr-rpmbuild-log [2026-01-04 17:16:32,072][ ERROR][PID:4122915] Re-try request for task on 'ResallocHost, ticket_id=650916, hostname=2620:52:6:1161:dead:beef:cafe:c150, name=vmhost_p09_03_prod_00912904_20260104_154010, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED [2026-01-04 17:16:32,072][ INFO][PID:4122915] Releasing VM back to pool [2026-01-04 17:16:32,080][ INFO][PID:4122915] Retry #1 (on other host) [2026-01-04 17:16:32,081][ INFO][PID:4122915] Checking that /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/repodata/repomd.xml exists [2026-01-04 17:16:32,087][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:16:32,087][ INFO][PID:4122915] VM allocation process starts [2026-01-04 17:16:32,128][ INFO][PID:4122915] Trying to allocate VM: ResallocHost, ticket_id=651061, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-04 17:16:35,165][ INFO][PID:4122915] Allocated host ResallocHost, ticket_id=651061, hostname=2620:52:6:1161:dead:beef:cafe:c18e, name=vmhost_p09_04_prod_00913197_20260104_163122, requested_tags=['copr_builder', 'arch_ppc64le'] [2026-01-04 17:16:35,166][ INFO][PID:4122915] Allocating ssh connection to builder [2026-01-04 17:16:35,166][ INFO][PID:4122915] Checking that builder machine is OK [2026-01-04 17:16:36,079][ INFO][PID:4122915] Running remote command: copr-builder-ready fedora-rawhide-ppc64le [2026-01-04 17:16:36,576][ INFO][PID:4122915] Red Hat subscription not needed for fedora-rawhide-ppc64le Builder is ready to be used [2026-01-04 17:16:36,577][ INFO][PID:4122915] Filling build.info file with builder info [2026-01-04 17:16:36,599][ INFO][PID:4122915] Checking for cancel request [2026-01-04 17:16:36,600][ INFO][PID:4122915] Not re-notifying FE and msg buses for the new host. [2026-01-04 17:16:36,600][ INFO][PID:4122915] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9974088-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached [2026-01-04 17:16:37,575][ INFO][PID:4122915] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18e' (ED25519) to the list of known hosts. [2026-01-04 17:16:37,668][ INFO][PID:4122915] Downloading the builder-live.log file, attempt 1 [2026-01-04 17:16:37,669][ INFO][PID:4122915] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18e copr-rpmbuild-log [2026-01-04 17:16:42,668][ INFO][PID:4122915] calling cancel callback, and waiting [2026-01-04 17:16:46,340][ INFO][PID:4122915] Cancel request succeeded out: success err: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18e' (ED25519) to the list of known hosts. [2026-01-04 17:16:46,340][ INFO][PID:4122915] Downloading results from builder [2026-01-04 17:16:46,341][ INFO][PID:4122915] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18e]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09974088-dtk6declarative started [2026-01-04 17:16:46,341][ INFO][PID:4122915] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18e]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09974088-dtk6declarative/ &> /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09974088-dtk6declarative/build-09974088.rsync.log [2026-01-04 17:16:47,585][ INFO][PID:4122915] rsyncing finished. [2026-01-04 17:16:47,586][ INFO][PID:4122915] Releasing VM back to pool [2026-01-04 17:16:47,594][ INFO][PID:4122915] Searching for 'success' file in resultdir [2026-01-04 17:16:47,594][ ERROR][PID:4122915] Build failed: Backend process error: No success file => build failure [2026-01-04 17:16:47,595][ INFO][PID:4122915] Finished build: id=9974088 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/topazus/test-review chroot=fedora-rawhide-ppc64le [2026-01-04 17:16:47,596][ ERROR][PID:4122915] Backend process error: No success file => build failure [2026-01-04 17:16:47,596][ INFO][PID:4122915] Worker failed build, took 10.996581315994263 [2026-01-04 17:16:47,597][ INFO][PID:4122915] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "topazus", "project_name": "test-review", "project_dirname": "test-review", "submitter": "topazus", "ended_on": 1767547007.5967536, "started_on": 1767546996.6001723, "submitted_on": null, "status": 0, "chroot": "fedora-rawhide-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9974088-fedora-rawhide-ppc64le", "build_id": 9974088, "package_name": "dtk6declarative", "package_version": "6.0.47-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/topazus/test-review/dtk6declarative", "git_hash": "b7740b236301f0a22b844ca5f9ad5751ed0fab54", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "topazus/test-review--topazus", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test-review/fedora-rawhide-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test-review/fedora-rawhide-ppc64le/", "id": "copr_topazus_test_review", "name": "Additional repo copr_topazus_test_review" } ], "background": false, "fedora_review": true, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/topazus/test-review", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/topazus/test-review", "result_dir": "09974088-dtk6declarative", "built_packages": "", "id": 9974088, "mockchain_macros": { "copr_username": "topazus", "copr_projectname": "test-review", "vendor": "Fedora Project COPR (topazus/test-review)" } } ] } [2026-01-04 17:16:47,650][ INFO][PID:4122915] Sending fedora-messaging bus message in build.end [2026-01-04 17:16:47,678][ INFO][PID:4122915] Compressing /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09974088-dtk6declarative/builder-live.log by gzip [2026-01-04 17:16:47,679][ INFO][PID:4122915] Running command 'gzip /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09974088-dtk6declarative/builder-live.log' as PID 39716 [2026-01-04 17:16:47,680][ INFO][PID:4122915] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09974088-dtk6declarative/builder-live.log) [2026-01-04 17:16:47,680][ INFO][PID:4122915] Compressing /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09974088-dtk6declarative/backend.log by gzip [2026-01-04 17:16:47,681][ INFO][PID:4122915] Running command 'gzip /var/lib/copr/public_html/results/topazus/test-review/fedora-rawhide-ppc64le/09974088-dtk6declarative/backend.log' as PID 39717