[2025-01-17 19:47:12,070][ INFO][PID:1981807] Marking build as starting [2025-01-17 19:47:12,306][ INFO][PID:1981807] Trying to allocate VM [2025-01-17 19:47:24,391][ INFO][PID:1981807] Allocated host ResallocHost, ticket_id=745835, hostname=140.211.168.104, name=copr_osuosl_p09_prod_00626430_20250117_191130 [2025-01-17 19:47:24,392][ INFO][PID:1981807] Allocating ssh connection to builder [2025-01-17 19:47:24,395][ INFO][PID:1981807] Checking that builder machine is OK [2025-01-17 19:47:25,673][ INFO][PID:1981807] Running remote command: copr-builder-ready centos-stream-10-ppc64le [2025-01-17 19:47:26,679][ INFO][PID:1981807] Red Hat subscription not needed for centos-stream-10-ppc64le Builder is ready to be used [2025-01-17 19:47:26,680][ INFO][PID:1981807] Filling build.info file with builder info [2025-01-17 19:47:26,681][ INFO][PID:1981807] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "rhcontainerbot", "project_name": "bootc", "project_dirname": "bootc", "submitter": "https://api.github.com/users/cgwalters", "ended_on": null, "started_on": 1737143246.681496, "submitted_on": null, "status": 3, "chroot": "centos-stream-10-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "8528575-centos-stream-10-ppc64le", "build_id": 8528575, "package_name": "bootc", "package_version": "202501171941.g42108088ef-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/bootc/bootc", "git_hash": "3990f880d46e0c405f331c6d3c979cd93781c39e", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "rhcontainerbot/bootc--https://api.github.com/users/cgwalters", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/rhcontainerbot/bootc", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc", "result_dir": "08528575-bootc", "built_packages": "", "id": 8528575, "mockchain_macros": { "copr_username": "rhcontainerbot", "copr_projectname": "bootc", "vendor": "Fedora Project COPR (rhcontainerbot/bootc)" } } ] } [2025-01-17 19:47:26,779][ INFO][PID:1981807] Sending fedora-messaging bus message in build.start [2025-01-17 19:47:27,802][ INFO][PID:1981807] Sending fedora-messaging bus message in chroot.start [2025-01-17 19:47:27,861][ INFO][PID:1981807] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8528575-centos-stream-10-ppc64le --chroot centos-stream-10-ppc64le --detached [2025-01-17 19:47:29,296][ INFO][PID:1981807] Downloading the builder-live.log file, attempt 1 [2025-01-17 19:47:29,304][ INFO][PID:1981807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.104 copr-rpmbuild-log [2025-01-17 19:47:34,332][ INFO][PID:1981807] Periodic builder liveness probe: alive [2025-01-17 19:56:32,232][ ERROR][PID:1981807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-17 19:56:42,233][WARNING][PID:1981807] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-17 19:56:52,234][ INFO][PID:1981807] Downloading the builder-live.log file, attempt 2 [2025-01-17 19:56:52,236][ INFO][PID:1981807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.104 copr-rpmbuild-log [2025-01-17 19:57:34,584][ INFO][PID:1981807] Periodic builder liveness probe: alive [2025-01-17 19:57:52,305][ ERROR][PID:1981807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-17 19:58:02,312][WARNING][PID:1981807] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-17 19:58:12,314][ INFO][PID:1981807] Downloading the builder-live.log file, attempt 3 [2025-01-17 19:58:12,328][ INFO][PID:1981807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.104 copr-rpmbuild-log [2025-01-17 19:59:12,418][ ERROR][PID:1981807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-17 19:59:22,423][WARNING][PID:1981807] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-17 19:59:32,424][ INFO][PID:1981807] Downloading the builder-live.log file, attempt 4 [2025-01-17 19:59:32,426][ INFO][PID:1981807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.104 copr-rpmbuild-log [2025-01-17 20:00:32,499][ ERROR][PID:1981807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-17 20:00:42,501][ ERROR][PID:1981807] Giving up for unstable SSH, failures: 1737143802.233725, 1737143882.312382, 1737143962.423678, 1737144042.5008345 [2025-01-17 20:00:42,504][ ERROR][PID:1981807] Re-try request for task on 'ResallocHost, ticket_id=745835, hostname=140.211.168.104, name=copr_osuosl_p09_prod_00626430_20250117_191130': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-01-17 20:00:42,504][ INFO][PID:1981807] Releasing VM back to pool [2025-01-17 20:00:42,522][ INFO][PID:1981807] Retry #1 (on other host) [2025-01-17 20:00:42,524][ INFO][PID:1981807] Trying to allocate VM [2025-01-17 20:01:50,645][ INFO][PID:1981807] Allocated host ResallocHost, ticket_id=746251, hostname=140.211.168.14, name=copr_osuosl_p09_prod_00627402_20250117_195729 [2025-01-17 20:01:50,646][ INFO][PID:1981807] Allocating ssh connection to builder [2025-01-17 20:01:50,646][ INFO][PID:1981807] Checking that builder machine is OK [2025-01-17 20:01:51,874][ INFO][PID:1981807] Running remote command: copr-builder-ready centos-stream-10-ppc64le [2025-01-17 20:01:52,850][ INFO][PID:1981807] Red Hat subscription not needed for centos-stream-10-ppc64le Builder is ready to be used [2025-01-17 20:01:52,851][ INFO][PID:1981807] Filling build.info file with builder info [2025-01-17 20:01:52,860][ INFO][PID:1981807] Not re-notifying FE and msg buses for the new host. [2025-01-17 20:01:52,861][ INFO][PID:1981807] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8528575-centos-stream-10-ppc64le --chroot centos-stream-10-ppc64le --detached [2025-01-17 20:01:54,149][ INFO][PID:1981807] Downloading the builder-live.log file, attempt 1 [2025-01-17 20:01:54,150][ INFO][PID:1981807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log [2025-01-17 20:07:39,310][ INFO][PID:1981807] Periodic builder liveness probe: alive [2025-01-17 20:17:39,628][ INFO][PID:1981807] Periodic builder liveness probe: alive [2025-01-17 20:19:34,229][ ERROR][PID:1981807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-17 20:19:44,230][WARNING][PID:1981807] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-17 20:19:54,232][ INFO][PID:1981807] Downloading the builder-live.log file, attempt 2 [2025-01-17 20:19:54,234][ INFO][PID:1981807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log [2025-01-17 20:20:54,306][ ERROR][PID:1981807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-17 20:21:04,311][WARNING][PID:1981807] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-17 20:21:14,312][ INFO][PID:1981807] Downloading the builder-live.log file, attempt 3 [2025-01-17 20:21:14,314][ INFO][PID:1981807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log [2025-01-17 20:22:14,382][ ERROR][PID:1981807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-17 20:22:24,385][WARNING][PID:1981807] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-17 20:22:34,386][ INFO][PID:1981807] Downloading the builder-live.log file, attempt 4 [2025-01-17 20:22:34,387][ INFO][PID:1981807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log [2025-01-17 20:23:34,461][ ERROR][PID:1981807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-17 20:23:44,466][ ERROR][PID:1981807] Giving up for unstable SSH, failures: 1737145184.2301376, 1737145264.3110092, 1737145344.385024, 1737145424.4665298 [2025-01-17 20:23:44,468][ ERROR][PID:1981807] Re-try request for task on 'ResallocHost, ticket_id=746251, hostname=140.211.168.14, name=copr_osuosl_p09_prod_00627402_20250117_195729': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-01-17 20:23:44,469][ INFO][PID:1981807] Releasing VM back to pool [2025-01-17 20:23:44,515][ INFO][PID:1981807] Retry #2 (on other host) [2025-01-17 20:23:44,518][ INFO][PID:1981807] Trying to allocate VM [2025-01-17 20:26:02,876][ INFO][PID:1981807] Allocated host ResallocHost, ticket_id=746916, hostname=2620:52:3:1:dead:beef:cafe:c299, name=copr_p09_01_prod_00627934_20250117_202232 [2025-01-17 20:26:02,877][ INFO][PID:1981807] Allocating ssh connection to builder [2025-01-17 20:26:02,878][ INFO][PID:1981807] Checking that builder machine is OK [2025-01-17 20:26:03,688][ INFO][PID:1981807] Running remote command: copr-builder-ready centos-stream-10-ppc64le [2025-01-17 20:26:04,192][ INFO][PID:1981807] Red Hat subscription not needed for centos-stream-10-ppc64le Builder is ready to be used [2025-01-17 20:26:04,193][ INFO][PID:1981807] Filling build.info file with builder info [2025-01-17 20:26:04,202][ INFO][PID:1981807] Not re-notifying FE and msg buses for the new host. [2025-01-17 20:26:04,203][ INFO][PID:1981807] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8528575-centos-stream-10-ppc64le --chroot centos-stream-10-ppc64le --detached [2025-01-17 20:26:05,185][ INFO][PID:1981807] Downloading the builder-live.log file, attempt 1 [2025-01-17 20:26:05,187][ INFO][PID:1981807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c299 copr-rpmbuild-log [2025-01-17 20:27:40,204][ INFO][PID:1981807] Periodic builder liveness probe: alive [2025-01-17 20:37:40,307][ INFO][PID:1981807] Periodic builder liveness probe: alive [2025-01-17 20:44:20,181][ INFO][PID:1981807] Downloading results from builder [2025-01-17 20:44:20,182][ INFO][PID:1981807] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c299]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc started [2025-01-17 20:44:20,184][ INFO][PID:1981807] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c299]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/ &> /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/build-08528575.rsync.log [2025-01-17 20:44:21,110][ INFO][PID:1981807] rsyncing finished. [2025-01-17 20:44:21,111][ INFO][PID:1981807] Releasing VM back to pool [2025-01-17 20:44:21,120][ INFO][PID:1981807] Searching for 'success' file in resultdir [2025-01-17 20:44:21,121][ INFO][PID:1981807] Going to sign pkgs from source: 8528575-centos-stream-10-ppc64le in chroot: /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le [2025-01-17 20:44:21,122][ INFO][PID:1981807] Calling '/bin/sign -u rhcontainerbot#bootc@copr.fedorahosted.org -p' (attempt #1) [2025-01-17 20:44:21,432][ INFO][PID:1981807] Calling '/bin/sign -4 -h sha256 -u rhcontainerbot#bootc@copr.fedorahosted.org -r /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/bootc-debugsource-202501171941.g42108088ef-1.el10.ppc64le.rpm' (attempt #1) [2025-01-17 20:44:21,970][ INFO][PID:1981807] signed rpm: /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/bootc-debugsource-202501171941.g42108088ef-1.el10.ppc64le.rpm [2025-01-17 20:44:21,971][ INFO][PID:1981807] Calling '/bin/sign -4 -h sha256 -u rhcontainerbot#bootc@copr.fedorahosted.org -r /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/bootc-202501171941.g42108088ef-1.el10.src.rpm' (attempt #1) [2025-01-17 20:44:23,088][ INFO][PID:1981807] signed rpm: /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/bootc-202501171941.g42108088ef-1.el10.src.rpm [2025-01-17 20:44:23,089][ INFO][PID:1981807] Calling '/bin/sign -4 -h sha256 -u rhcontainerbot#bootc@copr.fedorahosted.org -r /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/bootc-debuginfo-202501171941.g42108088ef-1.el10.ppc64le.rpm' (attempt #1) [2025-01-17 20:44:24,167][ INFO][PID:1981807] signed rpm: /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/bootc-debuginfo-202501171941.g42108088ef-1.el10.ppc64le.rpm [2025-01-17 20:44:24,167][ INFO][PID:1981807] Calling '/bin/sign -4 -h sha256 -u rhcontainerbot#bootc@copr.fedorahosted.org -r /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/bootc-202501171941.g42108088ef-1.el10.ppc64le.rpm' (attempt #1) [2025-01-17 20:44:24,809][ INFO][PID:1981807] signed rpm: /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/bootc-202501171941.g42108088ef-1.el10.ppc64le.rpm [2025-01-17 20:44:24,809][ INFO][PID:1981807] Sign done [2025-01-17 20:44:24,810][ INFO][PID:1981807] Incremental createrepo run, adding 08528575-bootc into https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-10-ppc64le, (auto-create-repo=True) [2025-01-17 20:44:24,811][ INFO][PID:1981807] Running command 'copr-repo --batched /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le --add 08528575-bootc --no-appstream-metadata' as PID 2473414 [2025-01-17 20:44:25,798][ INFO][PID:1981807] Finished after 0 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le --add 08528575-bootc --no-appstream-metadata) [2025-01-17 20:44:25,799][ INFO][PID:1981807] Getting build details [2025-01-17 20:44:25,799][ INFO][PID:1981807] Listing built binary packages in /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc [2025-01-17 20:44:25,800][ INFO][PID:1981807] Built packages: ['bootc 202501171941.g42108088ef', 'bootc-debuginfo 202501171941.g42108088ef', 'bootc-debugsource 202501171941.g42108088ef'] [2025-01-17 20:44:25,801][ INFO][PID:1981807] build details: {'built_packages': 'bootc 202501171941.g42108088ef\nbootc-debuginfo 202501171941.g42108088ef\nbootc-debugsource 202501171941.g42108088ef'} [2025-01-17 20:44:25,801][ INFO][PID:1981807] Retrieving pubkey [2025-01-17 20:44:25,801][ INFO][PID:1981807] Calling '/bin/sign -u rhcontainerbot#bootc@copr.fedorahosted.org -p' (attempt #1) [2025-01-17 20:44:26,107][ INFO][PID:1981807] Added pubkey for user rhcontainerbot project bootc into: /var/lib/copr/public_html/results/rhcontainerbot/bootc/pubkey.gpg [2025-01-17 20:44:26,108][ INFO][PID:1981807] Finished build: id=8528575 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/rhcontainerbot/bootc chroot=centos-stream-10-ppc64le [2025-01-17 20:44:26,109][ INFO][PID:1981807] Worker succeeded build, took 1101.9072427749634 [2025-01-17 20:44:26,110][ INFO][PID:1981807] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "rhcontainerbot", "project_name": "bootc", "project_dirname": "bootc", "submitter": "https://api.github.com/users/cgwalters", "ended_on": 1737146666.1099303, "started_on": 1737145564.2026875, "submitted_on": null, "status": 1, "chroot": "centos-stream-10-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "8528575-centos-stream-10-ppc64le", "build_id": 8528575, "package_name": "bootc", "package_version": "202501171941.g42108088ef-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/bootc/bootc", "git_hash": "3990f880d46e0c405f331c6d3c979cd93781c39e", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "rhcontainerbot/bootc--https://api.github.com/users/cgwalters", "results": { "packages": [ { "name": "bootc", "epoch": null, "version": "202501171941.g42108088ef", "release": "1.el10", "arch": "ppc64le" }, { "name": "bootc-debuginfo", "epoch": null, "version": "202501171941.g42108088ef", "release": "1.el10", "arch": "ppc64le" }, { "name": "bootc", "epoch": null, "version": "202501171941.g42108088ef", "release": "1.el10", "arch": "src" }, { "name": "bootc-debugsource", "epoch": null, "version": "202501171941.g42108088ef", "release": "1.el10", "arch": "ppc64le" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_ppc64le", "arch_power9" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/rhcontainerbot/bootc", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc", "result_dir": "08528575-bootc", "built_packages": "bootc 202501171941.g42108088ef\nbootc-debuginfo 202501171941.g42108088ef\nbootc-debugsource 202501171941.g42108088ef", "id": 8528575, "mockchain_macros": { "copr_username": "rhcontainerbot", "copr_projectname": "bootc", "vendor": "Fedora Project COPR (rhcontainerbot/bootc)" } } ] } [2025-01-17 20:44:26,198][ INFO][PID:1981807] Sending fedora-messaging bus message in build.end [2025-01-17 20:44:26,215][ INFO][PID:1981807] Compressing /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/builder-live.log by gzip [2025-01-17 20:44:26,217][ INFO][PID:1981807] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/builder-live.log' as PID 2473585 [2025-01-17 20:44:26,237][ INFO][PID:1981807] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/builder-live.log) [2025-01-17 20:44:26,238][ INFO][PID:1981807] Compressing /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/backend.log by gzip [2025-01-17 20:44:26,239][ INFO][PID:1981807] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/bootc/centos-stream-10-ppc64le/08528575-bootc/backend.log' as PID 2473590