[2025-03-26 11:49:44,328][ INFO][PID:1162760] Marking build as starting [2025-03-26 11:49:44,417][ INFO][PID:1162760] Trying to allocate VM: ResallocHost, ticket_id=2052552, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-03-26 11:49:47,474][ INFO][PID:1162760] Allocated host ResallocHost, ticket_id=2052552, hostname=2620:52:3:1:dead:beef:cafe:c28d, name=copr_p09_01_prod_02338412_20250326_113911, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-03-26 11:49:47,475][ INFO][PID:1162760] Allocating ssh connection to builder [2025-03-26 11:49:47,476][ INFO][PID:1162760] Checking that builder machine is OK [2025-03-26 11:49:48,527][ INFO][PID:1162760] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-03-26 11:49:49,039][ INFO][PID:1162760] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-03-26 11:49:49,041][ INFO][PID:1162760] Filling build.info file with builder info [2025-03-26 11:49:49,056][ INFO][PID:1162760] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "psimovec", "project_name": "openblas-0.3.23", "project_dirname": "openblas-0.3.23", "submitter": "https://src.fedoraproject.org/user/somlo", "ended_on": null, "started_on": 1742989789.0565107, "submitted_on": null, "status": 3, "chroot": "fedora-41-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "8823049-fedora-41-ppc64le", "build_id": 8823049, "package_name": "yosys", "package_version": "0.51-2.20250313gitc4b5190", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/psimovec/openblas-0.3.23/yosys", "git_hash": "735c95b712938dee10462febbeb3738e20bc103b", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "psimovec/openblas-0.3.23--https://src.fedoraproject.org/user/somlo", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/", "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch", "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch" } ], "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/psimovec/openblas-0.3.23", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/psimovec/openblas-0.3.23", "result_dir": "08823049-yosys", "built_packages": "", "id": 8823049, "mockchain_macros": { "copr_username": "psimovec", "copr_projectname": "openblas-0.3.23", "vendor": "Fedora Project COPR (psimovec/openblas-0.3.23)" } } ] } [2025-03-26 11:49:49,102][ INFO][PID:1162760] Sending fedora-messaging bus message in build.start [2025-03-26 11:49:49,910][ INFO][PID:1162760] Sending fedora-messaging bus message in chroot.start [2025-03-26 11:49:49,935][ INFO][PID:1162760] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8823049-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-03-26 11:49:51,174][ INFO][PID:1162760] Downloading the builder-live.log file, attempt 1 [2025-03-26 11:49:51,176][ INFO][PID:1162760] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28d copr-rpmbuild-log [2025-03-26 11:49:56,183][ INFO][PID:1162760] Periodic builder liveness probe: alive [2025-03-26 11:59:56,276][ INFO][PID:1162760] Periodic builder liveness probe: alive [2025-03-26 12:09:56,382][ INFO][PID:1162760] Periodic builder liveness probe: alive [2025-03-26 12:13:00,140][ ERROR][PID:1162760] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-03-26 12:13:10,142][WARNING][PID:1162760] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-03-26 12:13:20,144][ INFO][PID:1162760] Downloading the builder-live.log file, attempt 2 [2025-03-26 12:13:20,146][ INFO][PID:1162760] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28d copr-rpmbuild-log [2025-03-26 12:14:20,217][ ERROR][PID:1162760] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-03-26 12:14:30,219][WARNING][PID:1162760] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-03-26 12:14:40,220][ INFO][PID:1162760] Downloading the builder-live.log file, attempt 3 [2025-03-26 12:14:40,223][ INFO][PID:1162760] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28d copr-rpmbuild-log [2025-03-26 12:15:40,293][ ERROR][PID:1162760] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-03-26 12:15:50,295][WARNING][PID:1162760] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-03-26 12:16:00,296][ INFO][PID:1162760] Downloading the builder-live.log file, attempt 4 [2025-03-26 12:16:00,298][ INFO][PID:1162760] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c28d copr-rpmbuild-log [2025-03-26 12:17:00,367][ ERROR][PID:1162760] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-03-26 12:17:10,373][ ERROR][PID:1162760] Giving up for unstable SSH, failures: 1742991190.1424286, 1742991270.2191825, 1742991350.2955258, 1742991430.373278 [2025-03-26 12:17:10,374][ ERROR][PID:1162760] Re-try request for task on 'ResallocHost, ticket_id=2052552, hostname=2620:52:3:1:dead:beef:cafe:c28d, name=copr_p09_01_prod_02338412_20250326_113911, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-03-26 12:17:10,375][ INFO][PID:1162760] Releasing VM back to pool [2025-03-26 12:17:10,385][ INFO][PID:1162760] Retry #1 (on other host) [2025-03-26 12:17:10,432][ INFO][PID:1162760] Trying to allocate VM: ResallocHost, ticket_id=2053031, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-03-26 12:17:13,460][ INFO][PID:1162760] Allocated host ResallocHost, ticket_id=2053031, hostname=2620:52:3:1:dead:beef:cafe:c2a2, name=copr_p09_01_prod_02339056_20250326_120107, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-03-26 12:17:13,460][ INFO][PID:1162760] Allocating ssh connection to builder [2025-03-26 12:17:13,461][ INFO][PID:1162760] Checking that builder machine is OK [2025-03-26 12:17:14,172][ INFO][PID:1162760] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-03-26 12:17:14,572][ INFO][PID:1162760] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-03-26 12:17:14,572][ INFO][PID:1162760] Filling build.info file with builder info [2025-03-26 12:17:14,578][ INFO][PID:1162760] Not re-notifying FE and msg buses for the new host. [2025-03-26 12:17:14,578][ INFO][PID:1162760] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8823049-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-03-26 12:17:15,326][ INFO][PID:1162760] Downloading the builder-live.log file, attempt 1 [2025-03-26 12:17:15,328][ INFO][PID:1162760] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c2a2 copr-rpmbuild-log [2025-03-26 12:20:00,354][ INFO][PID:1162760] Periodic builder liveness probe: alive [2025-03-26 12:30:00,432][ INFO][PID:1162760] Periodic builder liveness probe: alive [2025-03-26 12:40:00,538][ INFO][PID:1162760] Periodic builder liveness probe: alive [2025-03-26 12:50:00,616][ INFO][PID:1162760] Periodic builder liveness probe: alive [2025-03-26 13:00:00,701][ INFO][PID:1162760] Periodic builder liveness probe: alive [2025-03-26 13:10:00,814][ INFO][PID:1162760] Periodic builder liveness probe: alive [2025-03-26 13:20:00,947][ INFO][PID:1162760] Periodic builder liveness probe: alive [2025-03-26 13:30:01,040][ INFO][PID:1162760] Periodic builder liveness probe: alive [2025-03-26 13:40:01,115][ INFO][PID:1162760] Periodic builder liveness probe: alive [2025-03-26 13:44:16,390][ INFO][PID:1162760] Downloading results from builder [2025-03-26 13:44:16,390][ INFO][PID:1162760] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a2]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys started [2025-03-26 13:44:16,391][ INFO][PID:1162760] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a2]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/ &> /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/build-08823049.rsync.log [2025-03-26 13:44:18,329][ INFO][PID:1162760] rsyncing finished. [2025-03-26 13:44:18,330][ INFO][PID:1162760] Releasing VM back to pool [2025-03-26 13:44:18,339][ INFO][PID:1162760] Searching for 'success' file in resultdir [2025-03-26 13:44:18,339][ INFO][PID:1162760] Going to sign pkgs from source: 8823049-fedora-41-ppc64le in chroot: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le [2025-03-26 13:44:18,340][ INFO][PID:1162760] Calling '/bin/sign -u psimovec#openblas-0.3.23@copr.fedorahosted.org -p' (attempt #1) [2025-03-26 13:44:18,655][ INFO][PID:1162760] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-debuginfo-0.51-2.20250313gitc4b5190.fc43.ppc64le.rpm' (attempt #1) [2025-03-26 13:44:20,834][ INFO][PID:1162760] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-debuginfo-0.51-2.20250313gitc4b5190.fc43.ppc64le.rpm [2025-03-26 13:44:20,835][ INFO][PID:1162760] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-share-0.51-2.20250313gitc4b5190.fc43.noarch.rpm' (attempt #1) [2025-03-26 13:44:21,730][ INFO][PID:1162760] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-share-0.51-2.20250313gitc4b5190.fc43.noarch.rpm [2025-03-26 13:44:21,731][ INFO][PID:1162760] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-0.51-2.20250313gitc4b5190.fc43.ppc64le.rpm' (attempt #1) [2025-03-26 13:44:22,739][ INFO][PID:1162760] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-0.51-2.20250313gitc4b5190.fc43.ppc64le.rpm [2025-03-26 13:44:22,739][ INFO][PID:1162760] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-debugsource-0.51-2.20250313gitc4b5190.fc43.ppc64le.rpm' (attempt #1) [2025-03-26 13:44:23,645][ INFO][PID:1162760] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-debugsource-0.51-2.20250313gitc4b5190.fc43.ppc64le.rpm [2025-03-26 13:44:23,646][ INFO][PID:1162760] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-devel-0.51-2.20250313gitc4b5190.fc43.ppc64le.rpm' (attempt #1) [2025-03-26 13:44:24,532][ INFO][PID:1162760] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-devel-0.51-2.20250313gitc4b5190.fc43.ppc64le.rpm [2025-03-26 13:44:24,533][ INFO][PID:1162760] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-doc-0.51-2.20250313gitc4b5190.fc43.ppc64le.rpm' (attempt #1) [2025-03-26 13:44:25,469][ INFO][PID:1162760] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-doc-0.51-2.20250313gitc4b5190.fc43.ppc64le.rpm [2025-03-26 13:44:25,469][ INFO][PID:1162760] Calling '/bin/sign -4 -h sha256 -u psimovec#openblas-0.3.23@copr.fedorahosted.org -r /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-0.51-2.20250313gitc4b5190.fc43.src.rpm' (attempt #1) [2025-03-26 13:44:26,462][ INFO][PID:1162760] signed rpm: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/yosys-0.51-2.20250313gitc4b5190.fc43.src.rpm [2025-03-26 13:44:26,463][ INFO][PID:1162760] Sign done [2025-03-26 13:44:26,464][ INFO][PID:1162760] Incremental createrepo run, adding 08823049-yosys into https://download.copr.fedorainfracloud.org/results/psimovec/openblas-0.3.23/fedora-41-ppc64le, (auto-create-repo=True) [2025-03-26 13:44:26,466][ INFO][PID:1162760] Running command 'copr-repo --batched /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le --add 08823049-yosys --no-appstream-metadata' as PID 1957644 [2025-03-26 13:44:37,592][ INFO][PID:1162760] Finished after 11 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le --add 08823049-yosys --no-appstream-metadata) [2025-03-26 13:44:37,593][ INFO][PID:1162760] Getting build details [2025-03-26 13:44:37,594][ INFO][PID:1162760] Listing built binary packages in /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys [2025-03-26 13:44:37,595][ INFO][PID:1162760] Built packages: ['yosys-debuginfo 0.51', 'yosys 0.51', 'yosys-share 0.51', 'yosys-doc 0.51', 'yosys-devel 0.51', 'yosys-debugsource 0.51'] [2025-03-26 13:44:37,595][ INFO][PID:1162760] build details: {'built_packages': 'yosys-debuginfo 0.51\nyosys 0.51\nyosys-share 0.51\nyosys-doc 0.51\nyosys-devel 0.51\nyosys-debugsource 0.51'} [2025-03-26 13:44:37,596][ INFO][PID:1162760] Retrieving pubkey [2025-03-26 13:44:37,596][ INFO][PID:1162760] Calling '/bin/sign -u psimovec#openblas-0.3.23@copr.fedorahosted.org -p' (attempt #1) [2025-03-26 13:44:37,905][ INFO][PID:1162760] Added pubkey for user psimovec project openblas-0.3.23 into: /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/pubkey.gpg [2025-03-26 13:44:37,906][ INFO][PID:1162760] Finished build: id=8823049 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/psimovec/openblas-0.3.23 chroot=fedora-41-ppc64le [2025-03-26 13:44:37,908][ INFO][PID:1162760] Worker succeeded build, took 5243.330302238464 [2025-03-26 13:44:37,909][ INFO][PID:1162760] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "psimovec", "project_name": "openblas-0.3.23", "project_dirname": "openblas-0.3.23", "submitter": "https://src.fedoraproject.org/user/somlo", "ended_on": 1742996677.908382, "started_on": 1742991434.5780797, "submitted_on": null, "status": 1, "chroot": "fedora-41-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "8823049-fedora-41-ppc64le", "build_id": 8823049, "package_name": "yosys", "package_version": "0.51-2.20250313gitc4b5190", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/psimovec/openblas-0.3.23/yosys", "git_hash": "735c95b712938dee10462febbeb3738e20bc103b", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "psimovec/openblas-0.3.23--https://src.fedoraproject.org/user/somlo", "results": { "packages": [ { "name": "yosys-debuginfo", "epoch": null, "version": "0.51", "release": "2.20250313gitc4b5190.fc43", "arch": "ppc64le" }, { "name": "yosys", "epoch": null, "version": "0.51", "release": "2.20250313gitc4b5190.fc43", "arch": "ppc64le" }, { "name": "yosys-share", "epoch": null, "version": "0.51", "release": "2.20250313gitc4b5190.fc43", "arch": "noarch" }, { "name": "yosys-doc", "epoch": null, "version": "0.51", "release": "2.20250313gitc4b5190.fc43", "arch": "ppc64le" }, { "name": "yosys-devel", "epoch": null, "version": "0.51", "release": "2.20250313gitc4b5190.fc43", "arch": "ppc64le" }, { "name": "yosys-debugsource", "epoch": null, "version": "0.51", "release": "2.20250313gitc4b5190.fc43", "arch": "ppc64le" }, { "name": "yosys", "epoch": null, "version": "0.51", "release": "2.20250313gitc4b5190.fc43", "arch": "src" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/", "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch", "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch" } ], "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/psimovec/openblas-0.3.23", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/psimovec/openblas-0.3.23", "result_dir": "08823049-yosys", "built_packages": "yosys-debuginfo 0.51\nyosys 0.51\nyosys-share 0.51\nyosys-doc 0.51\nyosys-devel 0.51\nyosys-debugsource 0.51", "id": 8823049, "mockchain_macros": { "copr_username": "psimovec", "copr_projectname": "openblas-0.3.23", "vendor": "Fedora Project COPR (psimovec/openblas-0.3.23)" } } ] } [2025-03-26 13:44:37,949][ INFO][PID:1162760] Sending fedora-messaging bus message in build.end [2025-03-26 13:44:37,993][ INFO][PID:1162760] Compressing /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/builder-live.log by gzip [2025-03-26 13:44:37,994][ INFO][PID:1162760] Running command 'gzip /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/builder-live.log' as PID 1959137 [2025-03-26 13:44:38,127][ INFO][PID:1162760] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/builder-live.log) [2025-03-26 13:44:38,128][ INFO][PID:1162760] Compressing /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/backend.log by gzip [2025-03-26 13:44:38,129][ INFO][PID:1162760] Running command 'gzip /var/lib/copr/public_html/results/psimovec/openblas-0.3.23/fedora-41-ppc64le/08823049-yosys/backend.log' as PID 1959149