[2025-12-28 04:42:08,497][ INFO][PID:642732] Marking build as starting [2025-12-28 04:42:08,621][ INFO][PID:642732] Checking that /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/repodata/repomd.xml exists [2025-12-28 04:42:08,630][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:42:08,631][ INFO][PID:642732] VM allocation process starts [2025-12-28 04:42:08,666][ INFO][PID:642732] Trying to allocate VM: ResallocHost, ticket_id=541489, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-28 04:42:11,708][ INFO][PID:642732] Allocated host ResallocHost, ticket_id=541489, hostname=2620:52:6:1161:dead:beef:cafe:c18d, name=vmhost_p09_04_prod_00796037_20251228_041201, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-12-28 04:42:11,709][ INFO][PID:642732] Allocating ssh connection to builder [2025-12-28 04:42:11,709][ INFO][PID:642732] Checking that builder machine is OK [2025-12-28 04:42:12,698][ INFO][PID:642732] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-12-28 04:42:13,209][ INFO][PID:642732] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-12-28 04:42:13,209][ INFO][PID:642732] Filling build.info file with builder info [2025-12-28 04:42:13,210][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:42:13,211][ INFO][PID:642732] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "topazus", "project_name": "test-review", "project_dirname": "test-review", "submitter": "topazus", "ended_on": null, "started_on": 1766896933.2116492, "submitted_on": null, "status": 3, "chroot": "fedora-41-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9954231-fedora-41-ppc64le", "build_id": 9954231, "package_name": "python-uncertainties", "package_version": "3.2.3-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/topazus/test-review/python-uncertainties", "git_hash": "e2c52bd0e495bdd5a1773f1ee31bbb87cb3657fa", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "topazus/test-review--topazus", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test-review/fedora-41-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test-review/fedora-41-ppc64le/", "id": "copr_topazus_test_review", "name": "Additional repo copr_topazus_test_review" } ], "background": false, "fedora_review": true, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/topazus/test-review", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/topazus/test-review", "result_dir": "09954231-python-uncertainties", "built_packages": "", "id": 9954231, "mockchain_macros": { "copr_username": "topazus", "copr_projectname": "test-review", "vendor": "Fedora Project COPR (topazus/test-review)" } } ] } [2025-12-28 04:42:13,309][ INFO][PID:642732] Sending fedora-messaging bus message in build.start [2025-12-28 04:42:13,924][ INFO][PID:642732] Sending fedora-messaging bus message in chroot.start [2025-12-28 04:42:13,953][ INFO][PID:642732] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9954231-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-12-28 04:42:14,847][ INFO][PID:642732] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18d' (ED25519) to the list of known hosts. [2025-12-28 04:42:14,848][ INFO][PID:642732] Downloading the builder-live.log file, attempt 1 [2025-12-28 04:42:14,849][ INFO][PID:642732] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18d copr-rpmbuild-log [2025-12-28 04:42:19,853][ INFO][PID:642732] Periodic builder liveness probe: alive [2025-12-28 04:42:19,853][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:42:24,854][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:42:29,855][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:42:34,856][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:42:39,857][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:42:44,858][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:42:49,864][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:42:54,865][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:42:59,866][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:43:04,867][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:43:09,868][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:43:14,869][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:43:19,870][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:43:24,871][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:43:29,873][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:43:34,874][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:43:39,875][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:43:44,876][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:43:49,877][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:43:54,878][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:43:59,879][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:44:04,880][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:44:09,881][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:44:14,882][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:44:19,883][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:44:24,884][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:44:29,886][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:44:34,887][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:44:39,889][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:44:44,891][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:44:49,892][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:44:54,893][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:44:59,894][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:45:04,895][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:45:09,896][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:45:14,897][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:45:19,898][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:45:24,900][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:45:29,901][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:45:34,902][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:45:39,903][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:45:44,904][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:45:49,905][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:45:54,906][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:45:59,907][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:46:04,908][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:46:09,909][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:46:14,910][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:46:19,910][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:46:24,911][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:46:29,912][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:46:34,913][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:46:39,914][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:46:44,915][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:46:49,916][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:46:54,917][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:46:59,918][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:47:04,919][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:47:09,919][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:47:14,920][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:47:19,921][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:47:25,086][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:47:30,087][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:47:35,088][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:47:40,090][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:47:45,091][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:47:50,092][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:47:55,093][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:48:00,094][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:48:05,095][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:48:10,096][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:48:15,096][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:48:20,097][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:48:25,098][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:48:30,099][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:48:35,100][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:48:40,101][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:48:45,102][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:48:50,103][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:48:55,103][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:49:00,104][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:49:05,105][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:49:10,106][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:49:15,107][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:49:20,108][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:49:25,109][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:49:30,110][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:49:35,111][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:49:40,112][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:49:45,278][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:49:50,279][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:49:55,280][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:50:00,310][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:50:05,405][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:50:10,407][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:50:15,408][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:50:20,409][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:50:25,411][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:50:30,442][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:50:35,443][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:50:40,444][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:50:45,445][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:50:50,639][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:50:55,640][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:51:00,641][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:51:05,711][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:51:10,713][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:51:15,768][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:51:20,770][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:51:25,801][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:51:30,836][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:51:35,910][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:51:40,911][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:51:45,912][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:51:50,914][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:51:55,932][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:52:00,933][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:52:05,946][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:52:10,947][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:52:15,947][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:52:20,953][ INFO][PID:642732] Periodic builder liveness probe: alive [2025-12-28 04:52:20,954][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:52:25,954][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:52:30,978][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:52:36,057][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:52:41,106][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:52:46,130][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:52:51,131][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:52:56,173][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:53:01,206][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:53:06,207][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:53:11,250][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:53:16,278][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:53:21,285][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:53:26,310][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:53:31,340][ INFO][PID:642732] Checking for cancel request [2025-12-28 04:53:34,189][ INFO][PID:642732] Downloading results from builder [2025-12-28 04:53:34,190][ INFO][PID:642732] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18d]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties started [2025-12-28 04:53:34,190][ INFO][PID:642732] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18d]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties/ &> /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties/build-09954231.rsync.log [2025-12-28 04:54:34,263][ INFO][PID:642732] rsyncing finished. [2025-12-28 04:54:34,264][ ERROR][PID:642732] Failed to download data from builder due to rsync error, see the rsync log file for details. [2025-12-28 04:54:34,264][ ERROR][PID:642732] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2025-12-28 04:54:44,265][ INFO][PID:642732] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18d]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties started [2025-12-28 04:54:44,266][ INFO][PID:642732] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18d]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties/ &> /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties/build-09954231.rsync.log [2025-12-28 04:55:44,356][ INFO][PID:642732] rsyncing finished. [2025-12-28 04:55:44,357][ ERROR][PID:642732] Failed to download data from builder due to rsync error, see the rsync log file for details. [2025-12-28 04:55:44,357][ ERROR][PID:642732] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details. [2025-12-28 04:55:54,358][ INFO][PID:642732] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18d]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties started [2025-12-28 04:55:54,359][ INFO][PID:642732] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c18d]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties/ &> /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties/build-09954231.rsync.log [2025-12-28 04:55:55,514][ INFO][PID:642732] rsyncing finished. [2025-12-28 04:55:55,514][ INFO][PID:642732] Releasing VM back to pool [2025-12-28 04:55:55,523][ INFO][PID:642732] Searching for 'success' file in resultdir [2025-12-28 04:55:55,524][ ERROR][PID:642732] Build failed: Backend process error: No success file => build failure [2025-12-28 04:55:55,525][ INFO][PID:642732] Finished build: id=9954231 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/topazus/test-review chroot=fedora-41-ppc64le [2025-12-28 04:55:55,526][ ERROR][PID:642732] Backend process error: No success file => build failure [2025-12-28 04:55:55,527][ INFO][PID:642732] Worker failed build, took 822.3155202865601 [2025-12-28 04:55:55,527][ INFO][PID:642732] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "topazus", "project_name": "test-review", "project_dirname": "test-review", "submitter": "topazus", "ended_on": 1766897755.5271695, "started_on": 1766896933.2116492, "submitted_on": null, "status": 0, "chroot": "fedora-41-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9954231-fedora-41-ppc64le", "build_id": 9954231, "package_name": "python-uncertainties", "package_version": "3.2.3-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/topazus/test-review/python-uncertainties", "git_hash": "e2c52bd0e495bdd5a1773f1ee31bbb87cb3657fa", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "topazus/test-review--topazus", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test-review/fedora-41-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test-review/fedora-41-ppc64le/", "id": "copr_topazus_test_review", "name": "Additional repo copr_topazus_test_review" } ], "background": false, "fedora_review": true, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/topazus/test-review", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/topazus/test-review", "result_dir": "09954231-python-uncertainties", "built_packages": "", "id": 9954231, "mockchain_macros": { "copr_username": "topazus", "copr_projectname": "test-review", "vendor": "Fedora Project COPR (topazus/test-review)" } } ] } [2025-12-28 04:55:55,593][ INFO][PID:642732] Sending fedora-messaging bus message in build.end [2025-12-28 04:55:55,621][ INFO][PID:642732] Compressing /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties/builder-live.log by gzip [2025-12-28 04:55:55,622][ INFO][PID:642732] Running command 'gzip /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties/builder-live.log' as PID 712877 [2025-12-28 04:55:55,658][ INFO][PID:642732] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties/builder-live.log) [2025-12-28 04:55:55,658][ INFO][PID:642732] Compressing /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties/backend.log by gzip [2025-12-28 04:55:55,659][ INFO][PID:642732] Running command 'gzip /var/lib/copr/public_html/results/topazus/test-review/fedora-41-ppc64le/09954231-python-uncertainties/backend.log' as PID 712879