[2026-01-15 12:15:02,398][ INFO][PID:805018] Marking build as starting [2026-01-15 12:15:02,455][ INFO][PID:805018] Checking that /var/lib/copr/public_html/results/r0x0d/goose/fedora-43-x86_64/repodata/repomd.xml exists [2026-01-15 12:15:02,725][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:15:02,726][ INFO][PID:805018] VM allocation process starts [2026-01-15 12:15:02,740][ INFO][PID:805018] Trying to allocate VM: ResallocHost, ticket_id=830660, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-15 12:15:05,759][ INFO][PID:805018] Allocated host ResallocHost, ticket_id=830660, hostname=52.202.46.42, name=aws_x86_64_spot_prod_01256703_20260115_120740, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-15 12:15:05,759][ INFO][PID:805018] Allocating ssh connection to builder [2026-01-15 12:15:05,760][ INFO][PID:805018] Checking that builder machine is OK [2026-01-15 12:15:06,073][ INFO][PID:805018] Running remote command: copr-builder-ready fedora-43-x86_64 [2026-01-15 12:15:06,264][ INFO][PID:805018] Red Hat subscription not needed for fedora-43-x86_64 Builder is ready to be used [2026-01-15 12:15:06,265][ INFO][PID:805018] Filling build.info file with builder info [2026-01-15 12:15:06,332][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:15:06,333][ INFO][PID:805018] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "r0x0d", "project_name": "goose", "project_dirname": "goose", "submitter": "r0x0d", "ended_on": null, "started_on": 1768479306.3334575, "submitted_on": null, "status": 3, "chroot": "fedora-43-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10014845-fedora-43-x86_64", "build_id": 10014845, "package_name": "goose", "package_version": "1.19.1-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/r0x0d/goose/goose", "git_hash": "9dcc44dbf258ffcd569c0d446e5088c6cc23f487", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "r0x0d/goose--r0x0d", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/r0x0d/goose/fedora-43-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/r0x0d/goose", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/r0x0d/goose", "result_dir": "10014845-goose", "built_packages": "", "id": 10014845, "mockchain_macros": { "copr_username": "r0x0d", "copr_projectname": "goose", "vendor": "Fedora Project COPR (r0x0d/goose)" } } ] } [2026-01-15 12:15:06,412][ INFO][PID:805018] Sending fedora-messaging bus message in build.start [2026-01-15 12:15:06,963][ INFO][PID:805018] Sending fedora-messaging bus message in chroot.start [2026-01-15 12:15:06,991][ INFO][PID:805018] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10014845-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached [2026-01-15 12:15:07,304][ INFO][PID:805018] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '52.202.46.42' (ED25519) to the list of known hosts. [2026-01-15 12:15:07,305][ INFO][PID:805018] Downloading the builder-live.log file, attempt 1 [2026-01-15 12:15:07,306][ INFO][PID:805018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.202.46.42 copr-rpmbuild-log [2026-01-15 12:15:12,314][ INFO][PID:805018] Periodic builder liveness probe: alive [2026-01-15 12:15:12,315][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:15:17,316][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:15:22,317][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:15:27,318][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:15:32,319][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:15:37,320][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:15:42,323][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:15:47,324][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:15:52,329][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:15:57,330][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:16:00,132][ ERROR][PID:805018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-15 12:16:02,331][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:16:07,332][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:16:10,133][WARNING][PID:805018] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-15 12:16:12,333][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:16:17,334][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:16:20,134][ INFO][PID:805018] Downloading the builder-live.log file, attempt 2 [2026-01-15 12:16:20,135][ INFO][PID:805018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.202.46.42 copr-rpmbuild-log [2026-01-15 12:16:22,335][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:16:27,337][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:16:32,341][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:16:37,345][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:16:42,348][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:16:47,353][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:16:52,355][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:16:57,356][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:17:02,357][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:17:07,359][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:17:12,362][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:17:17,364][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:17:20,204][ ERROR][PID:805018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-15 12:17:22,365][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:17:27,367][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:17:30,206][WARNING][PID:805018] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-15 12:17:32,373][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:17:37,374][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:17:40,208][ INFO][PID:805018] Downloading the builder-live.log file, attempt 3 [2026-01-15 12:17:40,217][ INFO][PID:805018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.202.46.42 copr-rpmbuild-log [2026-01-15 12:17:42,377][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:17:47,378][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:17:52,380][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:17:57,381][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:18:02,383][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:18:07,385][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:18:12,393][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:18:17,402][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:18:22,405][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:18:27,407][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:18:32,408][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:18:37,412][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:18:40,291][ ERROR][PID:805018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-15 12:18:42,420][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:18:47,422][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:18:50,295][WARNING][PID:805018] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-15 12:18:52,423][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:18:57,426][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:19:00,300][ INFO][PID:805018] Downloading the builder-live.log file, attempt 4 [2026-01-15 12:19:00,302][ INFO][PID:805018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.202.46.42 copr-rpmbuild-log [2026-01-15 12:19:02,427][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:19:07,437][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:19:12,439][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:19:17,440][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:19:22,441][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:19:27,443][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:19:32,446][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:19:37,449][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:19:42,461][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:19:47,464][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:19:52,484][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:19:57,491][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:00,377][ ERROR][PID:805018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-15 12:20:02,497][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:07,499][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:10,378][ ERROR][PID:805018] Giving up for unstable SSH, failures: 1768479370.133293, 1768479450.2059991, 1768479530.2952125, 1768479610.3787036 [2026-01-15 12:20:10,381][ ERROR][PID:805018] Re-try request for task on 'ResallocHost, ticket_id=830660, hostname=52.202.46.42, name=aws_x86_64_spot_prod_01256703_20260115_120740, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-15 12:20:10,381][ INFO][PID:805018] Releasing VM back to pool [2026-01-15 12:20:10,406][ INFO][PID:805018] Retry #1 (on other host) [2026-01-15 12:20:10,407][ INFO][PID:805018] Checking that /var/lib/copr/public_html/results/r0x0d/goose/fedora-43-x86_64/repodata/repomd.xml exists [2026-01-15 12:20:10,408][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:10,409][ INFO][PID:805018] VM allocation process starts [2026-01-15 12:20:10,425][ INFO][PID:805018] Trying to allocate VM: ResallocHost, ticket_id=830797, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-15 12:20:13,463][ INFO][PID:805018] Allocated host ResallocHost, ticket_id=830797, hostname=3.235.254.14, name=aws_x86_64_normalreserved_prod_01257124_20260115_121826, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-15 12:20:13,464][ INFO][PID:805018] Allocating ssh connection to builder [2026-01-15 12:20:13,465][ INFO][PID:805018] Checking that builder machine is OK [2026-01-15 12:20:13,808][ INFO][PID:805018] Running remote command: copr-builder-ready fedora-43-x86_64 [2026-01-15 12:20:14,016][ INFO][PID:805018] Red Hat subscription not needed for fedora-43-x86_64 Builder is ready to be used [2026-01-15 12:20:14,017][ INFO][PID:805018] Filling build.info file with builder info [2026-01-15 12:20:14,031][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:14,032][ INFO][PID:805018] Not re-notifying FE and msg buses for the new host. [2026-01-15 12:20:14,033][ INFO][PID:805018] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10014845-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached [2026-01-15 12:20:14,367][ INFO][PID:805018] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.235.254.14' (ED25519) to the list of known hosts. [2026-01-15 12:20:14,369][ INFO][PID:805018] Downloading the builder-live.log file, attempt 1 [2026-01-15 12:20:14,371][ INFO][PID:805018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.235.254.14 copr-rpmbuild-log [2026-01-15 12:20:19,369][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:24,372][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:29,374][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:34,377][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:39,378][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:44,380][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:49,383][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:54,390][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:20:59,400][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:21:04,401][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:21:09,403][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:21:14,405][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:21:19,408][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:21:24,410][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:21:29,411][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:21:34,412][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:21:39,413][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:21:44,414][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:21:49,417][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:21:54,418][ INFO][PID:805018] Checking for cancel request [2026-01-15 12:21:57,935][ INFO][PID:805018] Downloading results from builder [2026-01-15 12:21:57,936][ INFO][PID:805018] rsyncing of mockbuilder@3.235.254.14:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/r0x0d/goose/fedora-43-x86_64/10014845-goose started [2026-01-15 12:21:57,938][ INFO][PID:805018] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@3.235.254.14:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/r0x0d/goose/fedora-43-x86_64/10014845-goose/ &> /var/lib/copr/public_html/results/r0x0d/goose/fedora-43-x86_64/10014845-goose/build-10014845.rsync.log [2026-01-15 12:22:01,494][ INFO][PID:805018] rsyncing finished. [2026-01-15 12:22:01,495][ INFO][PID:805018] Releasing VM back to pool [2026-01-15 12:22:01,506][ INFO][PID:805018] Searching for 'success' file in resultdir [2026-01-15 12:22:01,506][ ERROR][PID:805018] Build failed: Backend process error: No success file => build failure [2026-01-15 12:22:01,507][ INFO][PID:805018] Finished build: id=10014845 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/r0x0d/goose chroot=fedora-43-x86_64 [2026-01-15 12:22:01,509][ ERROR][PID:805018] Backend process error: No success file => build failure [2026-01-15 12:22:01,509][ INFO][PID:805018] Worker failed build, took 107.47706890106201 [2026-01-15 12:22:01,510][ INFO][PID:805018] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "r0x0d", "project_name": "goose", "project_dirname": "goose", "submitter": "r0x0d", "ended_on": 1768479721.5098069, "started_on": 1768479614.032738, "submitted_on": null, "status": 0, "chroot": "fedora-43-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10014845-fedora-43-x86_64", "build_id": 10014845, "package_name": "goose", "package_version": "1.19.1-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/r0x0d/goose/goose", "git_hash": "9dcc44dbf258ffcd569c0d446e5088c6cc23f487", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "r0x0d/goose--r0x0d", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/r0x0d/goose/fedora-43-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/r0x0d/goose", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/r0x0d/goose", "result_dir": "10014845-goose", "built_packages": "", "id": 10014845, "mockchain_macros": { "copr_username": "r0x0d", "copr_projectname": "goose", "vendor": "Fedora Project COPR (r0x0d/goose)" } } ] } [2026-01-15 12:22:01,555][ INFO][PID:805018] Sending fedora-messaging bus message in build.end [2026-01-15 12:22:01,584][ INFO][PID:805018] Compressing /var/lib/copr/public_html/results/r0x0d/goose/fedora-43-x86_64/10014845-goose/builder-live.log by gzip [2026-01-15 12:22:01,585][ INFO][PID:805018] Running command 'gzip /var/lib/copr/public_html/results/r0x0d/goose/fedora-43-x86_64/10014845-goose/builder-live.log' as PID 862670 [2026-01-15 12:22:01,591][ INFO][PID:805018] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/r0x0d/goose/fedora-43-x86_64/10014845-goose/builder-live.log) [2026-01-15 12:22:01,592][ INFO][PID:805018] Compressing /var/lib/copr/public_html/results/r0x0d/goose/fedora-43-x86_64/10014845-goose/backend.log by gzip [2026-01-15 12:22:01,593][ INFO][PID:805018] Running command 'gzip /var/lib/copr/public_html/results/r0x0d/goose/fedora-43-x86_64/10014845-goose/backend.log' as PID 862677