[2026-01-26 17:31:30,119][ INFO][PID:2155648] Marking build as starting [2026-01-26 17:31:30,420][ INFO][PID:2155648] Checking that /var/lib/copr/public_html/results/lucamagrone/shairport-sync/fedora-43-x86_64/repodata/repomd.xml exists [2026-01-26 17:31:30,429][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:31:30,430][ INFO][PID:2155648] VM allocation process starts [2026-01-26 17:31:30,442][ INFO][PID:2155648] Trying to allocate VM: ResallocHost, ticket_id=1030607, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-26 17:31:33,461][ INFO][PID:2155648] Allocated host ResallocHost, ticket_id=1030607, hostname=98.87.162.33, name=aws_x86_64_spot_prod_01605595_20260126_172623, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-26 17:31:33,462][ INFO][PID:2155648] Allocating ssh connection to builder [2026-01-26 17:31:33,462][ INFO][PID:2155648] Checking that builder machine is OK [2026-01-26 17:31:33,824][ INFO][PID:2155648] Running remote command: copr-builder-ready fedora-43-x86_64 [2026-01-26 17:31:34,066][ INFO][PID:2155648] Red Hat subscription not needed for fedora-43-x86_64 Builder is ready to be used [2026-01-26 17:31:34,068][ INFO][PID:2155648] Filling build.info file with builder info [2026-01-26 17:31:34,069][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:31:34,070][ INFO][PID:2155648] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "lucamagrone", "project_name": "shairport-sync", "project_dirname": "shairport-sync", "submitter": "https://api.github.com/users/pull%5Bbot%5D", "ended_on": null, "started_on": 1769448694.069988, "submitted_on": null, "status": 3, "chroot": "fedora-43-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10064088-fedora-43-x86_64", "build_id": 10064088, "package_name": "shairport-sync-unstable", "package_version": "5.0-161", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/lucamagrone/shairport-sync/shairport-sync-unstable", "git_hash": "9f068c3dda388a82fd3b7544837a2d0c318207f6", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "lucamagrone/shairport-sync--https://api.github.com/users/pull%5Bbot%5D", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/lucamagrone/shairport-sync/fedora-43-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/lucamagrone/shairport-sync", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/lucamagrone/shairport-sync", "result_dir": "10064088-shairport-sync-unstable", "built_packages": "", "id": 10064088, "mockchain_macros": { "copr_username": "lucamagrone", "copr_projectname": "shairport-sync", "vendor": "Fedora Project COPR (lucamagrone/shairport-sync)" } } ] } [2026-01-26 17:31:34,137][ INFO][PID:2155648] Sending fedora-messaging bus message in build.start [2026-01-26 17:31:34,812][ INFO][PID:2155648] Sending fedora-messaging bus message in chroot.start [2026-01-26 17:31:34,839][ INFO][PID:2155648] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10064088-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached [2026-01-26 17:31:35,194][ INFO][PID:2155648] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '98.87.162.33' (ED25519) to the list of known hosts. [2026-01-26 17:31:35,228][ INFO][PID:2155648] Downloading the builder-live.log file, attempt 1 [2026-01-26 17:31:35,276][ INFO][PID:2155648] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.87.162.33 copr-rpmbuild-log [2026-01-26 17:31:40,249][ INFO][PID:2155648] Periodic builder liveness probe: alive [2026-01-26 17:31:40,603][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:31:45,604][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:31:50,616][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:31:52,748][ ERROR][PID:2155648] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-26 17:31:55,618][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:32:00,619][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:32:02,749][WARNING][PID:2155648] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-26 17:32:05,624][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:32:10,626][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:32:12,750][ INFO][PID:2155648] Downloading the builder-live.log file, attempt 2 [2026-01-26 17:32:12,752][ INFO][PID:2155648] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.87.162.33 copr-rpmbuild-log [2026-01-26 17:32:15,633][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:32:20,636][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:32:25,637][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:32:30,783][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:32:35,788][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:32:40,793][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:32:45,794][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:32:50,797][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:32:55,798][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:33:00,801][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:33:05,835][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:33:10,837][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:33:12,821][ ERROR][PID:2155648] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-26 17:33:15,838][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:33:20,840][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:33:22,847][WARNING][PID:2155648] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-26 17:33:25,842][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:33:30,844][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:33:32,849][ INFO][PID:2155648] Downloading the builder-live.log file, attempt 3 [2026-01-26 17:33:32,851][ INFO][PID:2155648] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.87.162.33 copr-rpmbuild-log [2026-01-26 17:33:35,846][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:33:40,847][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:33:45,848][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:33:50,849][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:33:55,850][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:34:00,851][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:34:05,852][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:34:10,854][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:34:15,858][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:34:20,862][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:34:25,865][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:34:30,867][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:34:32,923][ ERROR][PID:2155648] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-26 17:34:35,872][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:34:40,877][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:34:42,934][WARNING][PID:2155648] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-26 17:34:45,880][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:34:50,885][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:34:52,945][ INFO][PID:2155648] Downloading the builder-live.log file, attempt 4 [2026-01-26 17:34:52,949][ INFO][PID:2155648] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.87.162.33 copr-rpmbuild-log [2026-01-26 17:34:55,889][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:35:00,890][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:35:05,944][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:35:10,945][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:35:16,016][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:35:21,022][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:35:26,024][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:35:31,025][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:35:36,026][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:35:41,027][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:35:46,028][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:35:51,029][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:35:53,027][ ERROR][PID:2155648] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-26 17:35:56,031][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:01,034][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:03,034][ ERROR][PID:2155648] Giving up for unstable SSH, failures: 1769448722.7495692, 1769448802.8477256, 1769448882.9341347, 1769448963.0339434 [2026-01-26 17:36:03,035][ ERROR][PID:2155648] Re-try request for task on 'ResallocHost, ticket_id=1030607, hostname=98.87.162.33, name=aws_x86_64_spot_prod_01605595_20260126_172623, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-26 17:36:03,035][ INFO][PID:2155648] Releasing VM back to pool [2026-01-26 17:36:03,083][ INFO][PID:2155648] Retry #1 (on other host) [2026-01-26 17:36:03,083][ INFO][PID:2155648] Checking that /var/lib/copr/public_html/results/lucamagrone/shairport-sync/fedora-43-x86_64/repodata/repomd.xml exists [2026-01-26 17:36:03,084][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:03,086][ INFO][PID:2155648] VM allocation process starts [2026-01-26 17:36:03,121][ INFO][PID:2155648] Trying to allocate VM: ResallocHost, ticket_id=1030683, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-26 17:36:06,246][ INFO][PID:2155648] Allocated host ResallocHost, ticket_id=1030683, hostname=44.210.96.204, name=aws_x86_64_normal_prod_01602978_20260126_170412, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-26 17:36:06,247][ INFO][PID:2155648] Allocating ssh connection to builder [2026-01-26 17:36:06,247][ INFO][PID:2155648] Checking that builder machine is OK [2026-01-26 17:36:06,626][ INFO][PID:2155648] Running remote command: copr-builder-ready fedora-43-x86_64 [2026-01-26 17:36:06,852][ INFO][PID:2155648] Red Hat subscription not needed for fedora-43-x86_64 Builder is ready to be used [2026-01-26 17:36:06,854][ INFO][PID:2155648] Filling build.info file with builder info [2026-01-26 17:36:06,866][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:06,868][ INFO][PID:2155648] Not re-notifying FE and msg buses for the new host. [2026-01-26 17:36:06,869][ INFO][PID:2155648] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10064088-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached [2026-01-26 17:36:07,239][ INFO][PID:2155648] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '44.210.96.204' (ED25519) to the list of known hosts. [2026-01-26 17:36:07,241][ INFO][PID:2155648] Downloading the builder-live.log file, attempt 1 [2026-01-26 17:36:07,242][ INFO][PID:2155648] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.210.96.204 copr-rpmbuild-log [2026-01-26 17:36:12,241][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:17,242][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:22,244][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:27,246][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:32,247][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:37,249][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:42,252][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:47,254][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:52,260][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:36:57,262][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:37:02,266][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:37:07,267][ INFO][PID:2155648] Checking for cancel request [2026-01-26 17:37:08,472][ INFO][PID:2155648] Downloading results from builder [2026-01-26 17:37:08,473][ INFO][PID:2155648] rsyncing of mockbuilder@44.210.96.204:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/lucamagrone/shairport-sync/fedora-43-x86_64/10064088-shairport-sync-unstable started [2026-01-26 17:37:08,474][ INFO][PID:2155648] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.210.96.204:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/lucamagrone/shairport-sync/fedora-43-x86_64/10064088-shairport-sync-unstable/ &> /var/lib/copr/public_html/results/lucamagrone/shairport-sync/fedora-43-x86_64/10064088-shairport-sync-unstable/build-10064088.rsync.log [2026-01-26 17:37:08,707][ INFO][PID:2155648] rsyncing finished. [2026-01-26 17:37:08,708][ INFO][PID:2155648] Releasing VM back to pool [2026-01-26 17:37:08,722][ INFO][PID:2155648] Searching for 'success' file in resultdir [2026-01-26 17:37:08,723][ ERROR][PID:2155648] Build failed: Backend process error: No success file => build failure [2026-01-26 17:37:08,724][ INFO][PID:2155648] Finished build: id=10064088 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/lucamagrone/shairport-sync chroot=fedora-43-x86_64 [2026-01-26 17:37:08,726][ ERROR][PID:2155648] Backend process error: No success file => build failure [2026-01-26 17:37:08,726][ INFO][PID:2155648] Worker failed build, took 61.858365535736084 [2026-01-26 17:37:08,727][ INFO][PID:2155648] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "lucamagrone", "project_name": "shairport-sync", "project_dirname": "shairport-sync", "submitter": "https://api.github.com/users/pull%5Bbot%5D", "ended_on": 1769449028.726923, "started_on": 1769448966.8685575, "submitted_on": null, "status": 0, "chroot": "fedora-43-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10064088-fedora-43-x86_64", "build_id": 10064088, "package_name": "shairport-sync-unstable", "package_version": "5.0-161", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/lucamagrone/shairport-sync/shairport-sync-unstable", "git_hash": "9f068c3dda388a82fd3b7544837a2d0c318207f6", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "lucamagrone/shairport-sync--https://api.github.com/users/pull%5Bbot%5D", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/lucamagrone/shairport-sync/fedora-43-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/lucamagrone/shairport-sync", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/lucamagrone/shairport-sync", "result_dir": "10064088-shairport-sync-unstable", "built_packages": "", "id": 10064088, "mockchain_macros": { "copr_username": "lucamagrone", "copr_projectname": "shairport-sync", "vendor": "Fedora Project COPR (lucamagrone/shairport-sync)" } } ] } [2026-01-26 17:37:08,803][ INFO][PID:2155648] Sending fedora-messaging bus message in build.end [2026-01-26 17:37:08,832][ INFO][PID:2155648] Compressing /var/lib/copr/public_html/results/lucamagrone/shairport-sync/fedora-43-x86_64/10064088-shairport-sync-unstable/builder-live.log by gzip [2026-01-26 17:37:08,834][ INFO][PID:2155648] Running command 'gzip /var/lib/copr/public_html/results/lucamagrone/shairport-sync/fedora-43-x86_64/10064088-shairport-sync-unstable/builder-live.log' as PID 2189384 [2026-01-26 17:37:08,898][ INFO][PID:2155648] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/lucamagrone/shairport-sync/fedora-43-x86_64/10064088-shairport-sync-unstable/builder-live.log) [2026-01-26 17:37:08,899][ INFO][PID:2155648] Compressing /var/lib/copr/public_html/results/lucamagrone/shairport-sync/fedora-43-x86_64/10064088-shairport-sync-unstable/backend.log by gzip [2026-01-26 17:37:08,901][ INFO][PID:2155648] Running command 'gzip /var/lib/copr/public_html/results/lucamagrone/shairport-sync/fedora-43-x86_64/10064088-shairport-sync-unstable/backend.log' as PID 2189385