[2026-04-27 16:49:34,685][ INFO][PID:3784117] Marking build as starting [2026-04-27 16:49:34,856][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:49:34,867][ INFO][PID:3784117] VM allocation process starts [2026-04-27 16:49:35,010][ INFO][PID:3784117] Trying to allocate VM: ResallocHost, ticket_id=2605422, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-27 16:49:40,018][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:49:45,034][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:49:47,098][ INFO][PID:3784117] Allocated host ResallocHost, ticket_id=2605422, hostname=100.54.69.119, name=aws_x86_64_spot_prod_08300591_20260427_164516, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-27 16:49:47,099][ INFO][PID:3784117] Allocating ssh connection to builder [2026-04-27 16:49:47,102][ INFO][PID:3784117] Checking that builder machine is OK [2026-04-27 16:49:47,634][ INFO][PID:3784117] Installed copr-rpmbuild version: 1.8 [2026-04-27 16:49:47,635][ INFO][PID:3784117] Running remote command: copr-builder-ready srpm-builds [2026-04-27 16:49:48,112][ INFO][PID:3784117] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-04-27 16:49:48,114][ INFO][PID:3784117] Filling build.info file with builder info [2026-04-27 16:49:48,118][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:49:48,127][ INFO][PID:3784117] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "bengal", "project_name": "netfyr", "project_dirname": "netfyr", "submitter": "bengal", "ended_on": null, "started_on": 1777308588.127284, "submitted_on": null, "status": 3, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10396331", "build_id": 10396331, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#!/bin/bash\\ncurl https://raw.githubusercontent.com/bengal/netfyr-test-impl/refs/heads/main/scripts/build-copr.sh | bash\\n\", \"chroot\": \"fedora-44-x86_64\", \"builddeps\": \"\", \"resultdir\": \"\", \"repos\": \"\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "bengal/netfyr--bengal", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "distributions_in_build": [ "centos-stream-10", "centos-stream-9", "fedora-42", "fedora-43", "fedora-44" ], "distributions_in_project": [ "centos-stream-10", "centos-stream-9", "fedora-42", "fedora-43", "fedora-44" ], "destdir": "/var/lib/copr/public_html/results/bengal/netfyr", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/bengal/netfyr", "result_dir": "10396331", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 10396331, "mockchain_macros": { "copr_username": "bengal", "copr_projectname": "netfyr", "vendor": "Fedora Project COPR (bengal/netfyr)" } } ] } [2026-04-27 16:49:48,294][ INFO][PID:3784117] Sending fedora-messaging bus message in build.start [2026-04-27 16:49:53,400][ INFO][PID:3784117] Sending fedora-messaging bus message in chroot.start [2026-04-27 16:49:53,588][ INFO][PID:3784117] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10396331 --detached [2026-04-27 16:49:54,267][ INFO][PID:3784117] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '100.54.69.119' (ED25519) to the list of known hosts. [2026-04-27 16:49:54,279][ INFO][PID:3784117] Downloading the builder-live.log file, attempt 1 [2026-04-27 16:49:54,295][ INFO][PID:3784117] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.54.69.119 copr-rpmbuild-log [2026-04-27 16:49:59,319][ INFO][PID:3784117] Periodic builder liveness probe: alive [2026-04-27 16:49:59,322][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:50:04,333][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:50:09,341][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:50:14,349][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:50:19,357][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:50:24,376][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:50:29,383][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:50:34,396][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:50:38,581][ ERROR][PID:3784117] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-27 16:50:39,399][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:50:44,409][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:50:48,619][WARNING][PID:3784117] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-27 16:50:49,422][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:50:54,427][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:50:58,628][ INFO][PID:3784117] Downloading the builder-live.log file, attempt 2 [2026-04-27 16:50:58,630][ INFO][PID:3784117] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.54.69.119 copr-rpmbuild-log [2026-04-27 16:50:59,431][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:51:04,448][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:51:09,461][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:51:14,490][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:51:19,503][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:51:24,524][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:51:29,542][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:51:34,550][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:51:39,562][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:51:44,574][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:51:49,585][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:51:54,605][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:51:58,786][ ERROR][PID:3784117] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-27 16:51:59,640][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:52:04,659][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:52:08,857][WARNING][PID:3784117] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-27 16:52:09,668][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:52:14,678][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:52:18,860][ INFO][PID:3784117] Downloading the builder-live.log file, attempt 3 [2026-04-27 16:52:18,871][ INFO][PID:3784117] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.54.69.119 copr-rpmbuild-log [2026-04-27 16:52:19,708][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:52:24,744][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:52:29,750][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:52:34,752][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:52:39,808][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:52:44,837][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:52:49,840][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:52:54,846][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:52:59,852][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:53:04,863][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:53:09,866][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:53:14,869][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:53:18,992][ ERROR][PID:3784117] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-27 16:53:19,874][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:53:24,890][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:53:29,246][WARNING][PID:3784117] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-27 16:53:29,910][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:53:34,919][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:53:39,270][ INFO][PID:3784117] Downloading the builder-live.log file, attempt 4 [2026-04-27 16:53:39,296][ INFO][PID:3784117] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.54.69.119 copr-rpmbuild-log [2026-04-27 16:53:39,928][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:53:44,939][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:53:49,947][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:53:54,956][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:53:59,961][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:54:04,964][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:54:09,975][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:54:14,987][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:54:19,994][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:54:25,009][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:54:30,011][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:54:35,016][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:54:39,406][ ERROR][PID:3784117] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-27 16:54:40,019][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:54:45,023][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:54:49,438][ ERROR][PID:3784117] Giving up for unstable SSH, failures: 1777308648.6191442, 1777308728.8575683, 1777308809.2463672, 1777308889.438724 [2026-04-27 16:54:49,465][ ERROR][PID:3784117] Re-try request for task on 'ResallocHost, ticket_id=2605422, hostname=100.54.69.119, name=aws_x86_64_spot_prod_08300591_20260427_164516, requested_tags=['arch_x86_64', 'copr_builder']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-04-27 16:54:49,572][ INFO][PID:3784117] VM Release request [2026-04-27 16:54:49,640][ INFO][PID:3784117] Retry #1 (on other host) [2026-04-27 16:54:49,644][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:54:49,652][ INFO][PID:3784117] VM allocation process starts [2026-04-27 16:54:49,774][ INFO][PID:3784117] Trying to allocate VM: ResallocHost, ticket_id=2605697, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-27 16:54:54,788][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:54:59,792][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:55:01,894][ INFO][PID:3784117] Allocated host ResallocHost, ticket_id=2605697, hostname=34.204.189.51, name=aws_x86_64_reserved_prod_08300709_20260427_164952, requested_tags=['arch_x86_64', 'copr_builder'] [2026-04-27 16:55:01,896][ INFO][PID:3784117] Allocating ssh connection to builder [2026-04-27 16:55:01,900][ INFO][PID:3784117] Checking that builder machine is OK [2026-04-27 16:55:02,314][ INFO][PID:3784117] Installed copr-rpmbuild version: 1.8 [2026-04-27 16:55:02,315][ INFO][PID:3784117] Running remote command: copr-builder-ready srpm-builds [2026-04-27 16:55:02,583][ INFO][PID:3784117] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-04-27 16:55:02,585][ INFO][PID:3784117] Filling build.info file with builder info [2026-04-27 16:55:02,621][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:55:02,624][ INFO][PID:3784117] Not re-notifying FE and msg buses for the new host. [2026-04-27 16:55:02,625][ INFO][PID:3784117] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10396331 --detached [2026-04-27 16:55:03,286][ INFO][PID:3784117] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '34.204.189.51' (ED25519) to the list of known hosts. [2026-04-27 16:55:03,289][ INFO][PID:3784117] Downloading the builder-live.log file, attempt 1 [2026-04-27 16:55:03,298][ INFO][PID:3784117] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.204.189.51 copr-rpmbuild-log [2026-04-27 16:55:08,290][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:55:13,293][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:55:18,318][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:55:23,319][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:55:28,329][ INFO][PID:3784117] Checking for cancel request [2026-04-27 16:55:29,694][ INFO][PID:3784117] Downloading results from builder [2026-04-27 16:55:29,695][ INFO][PID:3784117] rsyncing of mockbuilder@34.204.189.51:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/bengal/netfyr/srpm-builds/10396331 started [2026-04-27 16:55:29,702][ INFO][PID:3784117] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@34.204.189.51:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/bengal/netfyr/srpm-builds/10396331/ &> /var/lib/copr/public_html/results/bengal/netfyr/srpm-builds/10396331/build-10396331.rsync.log [2026-04-27 16:55:30,147][ INFO][PID:3784117] rsyncing finished. [2026-04-27 16:55:30,151][ INFO][PID:3784117] VM Release request [2026-04-27 16:55:30,234][ INFO][PID:3784117] Searching for 'success' file in resultdir [2026-04-27 16:55:30,249][ ERROR][PID:3784117] Build failed: Backend process error: No success file => build failure [2026-04-27 16:55:30,458][ INFO][PID:3784117] Finished build: id=10396331 failed=True timeout=108000 destdir=/var/lib/copr/public_html/results/bengal/netfyr chroot=srpm-builds [2026-04-27 16:55:30,472][ ERROR][PID:3784117] Backend process error: No success file => build failure [2026-04-27 16:55:30,594][ INFO][PID:3784117] Worker failed build, took 27.9707510471344 [2026-04-27 16:55:30,598][ INFO][PID:3784117] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "bengal", "project_name": "netfyr", "project_dirname": "netfyr", "submitter": "bengal", "ended_on": 1777308930.5947666, "started_on": 1777308902.6240156, "submitted_on": null, "status": 0, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10396331", "build_id": 10396331, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#!/bin/bash\\ncurl https://raw.githubusercontent.com/bengal/netfyr-test-impl/refs/heads/main/scripts/build-copr.sh | bash\\n\", \"chroot\": \"fedora-44-x86_64\", \"builddeps\": \"\", \"resultdir\": \"\", \"repos\": \"\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "bengal/netfyr--bengal", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "distributions_in_build": [ "centos-stream-10", "centos-stream-9", "fedora-42", "fedora-43", "fedora-44" ], "distributions_in_project": [ "centos-stream-10", "centos-stream-9", "fedora-42", "fedora-43", "fedora-44" ], "destdir": "/var/lib/copr/public_html/results/bengal/netfyr", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/bengal/netfyr", "result_dir": "10396331", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 10396331, "mockchain_macros": { "copr_username": "bengal", "copr_projectname": "netfyr", "vendor": "Fedora Project COPR (bengal/netfyr)" } } ] } [2026-04-27 16:55:30,693][ INFO][PID:3784117] Sending fedora-messaging bus message in build.end [2026-04-27 16:55:30,764][ INFO][PID:3784117] Compressing /var/lib/copr/public_html/results/bengal/netfyr/srpm-builds/10396331/builder-live.log by gzip [2026-04-27 16:55:30,770][ INFO][PID:3784117] Running command 'gzip /var/lib/copr/public_html/results/bengal/netfyr/srpm-builds/10396331/builder-live.log' as PID 3850219 [2026-04-27 16:55:30,813][ INFO][PID:3784117] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/bengal/netfyr/srpm-builds/10396331/builder-live.log) [2026-04-27 16:55:30,827][ INFO][PID:3784117] Compressing /var/lib/copr/public_html/results/bengal/netfyr/srpm-builds/10396331/backend.log by gzip [2026-04-27 16:55:30,835][ INFO][PID:3784117] Running command 'gzip /var/lib/copr/public_html/results/bengal/netfyr/srpm-builds/10396331/backend.log' as PID 3850224