[2026-04-01 11:42:08,284][ INFO][PID:2413595] Marking build as starting [2026-04-01 11:42:08,373][ INFO][PID:2413595] Checking that /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/repodata/repomd.xml exists [2026-04-01 11:42:08,492][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:42:08,493][ INFO][PID:2413595] VM allocation process starts [2026-04-01 11:42:08,537][ INFO][PID:2413595] Trying to allocate VM: ResallocHost, ticket_id=2132351, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-01 11:42:11,583][ INFO][PID:2413595] Allocated host ResallocHost, ticket_id=2132351, hostname=3.239.69.130, name=aws_x86_64_spot_prod_04449811_20260401_113744, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-01 11:42:11,584][ INFO][PID:2413595] Allocating ssh connection to builder [2026-04-01 11:42:11,585][ INFO][PID:2413595] Checking that builder machine is OK [2026-04-01 11:42:11,925][ INFO][PID:2413595] Running remote command: copr-builder-ready fedora-42-x86_64 [2026-04-01 11:42:12,128][ INFO][PID:2413595] Red Hat subscription not needed for fedora-42-x86_64 Builder is ready to be used [2026-04-01 11:42:12,128][ INFO][PID:2413595] Filling build.info file with builder info [2026-04-01 11:42:12,129][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:42:12,130][ INFO][PID:2413595] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "amatej", "project_name": "dnf-nightly", "project_dirname": "dnf-nightly", "submitter": "amatej", "ended_on": null, "started_on": 1775043732.1300828, "submitted_on": null, "status": 3, "chroot": "fedora-42-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10282582-fedora-42-x86_64", "build_id": 10282582, "package_name": "libdnf", "package_version": "0.76.0^20260401134112260344.master.931.ge2c749c7-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/amatej/dnf-nightly/libdnf", "git_hash": "25cc6ce4bd80474d90b423e864dd6993613c99f8", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "amatej/dnf-nightly--amatej", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/amatej/dnf-nightly/fedora-42-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/rpmsoftwaremanagement/dnf-nightly/fedora-42-x86_64/", "id": "copr_rpmsoftwaremanagement_dnf_nightly", "name": "Additional repo copr_rpmsoftwaremanagement_dnf_nightly" } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/amatej/dnf-nightly", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/amatej/dnf-nightly", "result_dir": "10282582-libdnf", "built_packages": "", "id": 10282582, "mockchain_macros": { "copr_username": "amatej", "copr_projectname": "dnf-nightly", "vendor": "Fedora Project COPR (amatej/dnf-nightly)" } } ] } [2026-04-01 11:42:12,210][ INFO][PID:2413595] Sending fedora-messaging bus message in build.start [2026-04-01 11:42:13,039][ INFO][PID:2413595] Sending fedora-messaging bus message in chroot.start [2026-04-01 11:42:13,082][ INFO][PID:2413595] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10282582-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached [2026-04-01 11:42:13,438][ INFO][PID:2413595] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.239.69.130' (ED25519) to the list of known hosts. [2026-04-01 11:42:13,439][ INFO][PID:2413595] Downloading the builder-live.log file, attempt 1 [2026-04-01 11:42:13,517][ INFO][PID:2413595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.239.69.130 copr-rpmbuild-log [2026-04-01 11:42:18,446][ INFO][PID:2413595] Periodic builder liveness probe: alive [2026-04-01 11:42:18,447][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:42:23,448][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:42:28,449][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:42:33,450][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:42:38,451][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:42:43,452][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:42:48,453][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:42:53,454][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:42:58,455][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:43:03,457][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:43:08,460][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:43:13,462][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:43:18,464][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:43:23,465][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:43:28,466][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:43:33,467][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:43:38,468][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:43:43,469][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:43:48,471][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:43:53,472][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:43:58,473][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:44:03,474][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:44:08,475][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:44:13,477][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:44:18,478][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:44:23,479][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:44:28,480][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:44:33,481][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:44:38,482][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:44:43,171][ ERROR][PID:2413595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-01 11:44:43,483][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:44:48,484][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:44:53,172][WARNING][PID:2413595] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-01 11:44:53,486][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:44:58,487][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:45:03,173][ INFO][PID:2413595] Downloading the builder-live.log file, attempt 2 [2026-04-01 11:45:03,182][ INFO][PID:2413595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.239.69.130 copr-rpmbuild-log [2026-04-01 11:45:03,488][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:45:08,489][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:45:13,491][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:45:18,492][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:45:23,493][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:45:28,494][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:45:33,495][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:45:38,496][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:45:43,530][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:45:48,531][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:45:53,532][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:45:58,532][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:46:03,250][ ERROR][PID:2413595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-01 11:46:03,533][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:46:08,535][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:46:13,250][WARNING][PID:2413595] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-01 11:46:13,536][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:46:18,537][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:46:23,251][ INFO][PID:2413595] Downloading the builder-live.log file, attempt 3 [2026-04-01 11:46:23,252][ INFO][PID:2413595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.239.69.130 copr-rpmbuild-log [2026-04-01 11:46:23,539][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:46:28,540][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:46:33,541][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:46:38,542][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:46:43,543][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:46:48,544][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:46:53,575][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:46:58,577][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:47:03,578][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:47:08,579][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:47:13,580][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:47:18,581][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:47:23,321][ ERROR][PID:2413595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-01 11:47:23,582][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:47:28,583][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:47:33,322][WARNING][PID:2413595] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-01 11:47:33,584][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:47:38,585][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:47:43,322][ INFO][PID:2413595] Downloading the builder-live.log file, attempt 4 [2026-04-01 11:47:43,324][ INFO][PID:2413595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.239.69.130 copr-rpmbuild-log [2026-04-01 11:47:43,586][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:47:48,587][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:47:53,588][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:47:58,589][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:03,605][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:08,606][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:13,607][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:18,608][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:23,609][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:28,611][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:33,613][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:38,614][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:43,393][ ERROR][PID:2413595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-01 11:48:43,620][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:48,622][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:53,500][ ERROR][PID:2413595] Giving up for unstable SSH, failures: 1775043893.1726475, 1775043973.2507877, 1775044053.322151, 1775044133.500541 [2026-04-01 11:48:53,501][ ERROR][PID:2413595] Re-try request for task on 'ResallocHost, ticket_id=2132351, hostname=3.239.69.130, name=aws_x86_64_spot_prod_04449811_20260401_113744, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-04-01 11:48:53,502][ INFO][PID:2413595] Releasing VM back to pool [2026-04-01 11:48:53,517][ INFO][PID:2413595] Retry #1 (on other host) [2026-04-01 11:48:53,517][ INFO][PID:2413595] Checking that /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/repodata/repomd.xml exists [2026-04-01 11:48:53,517][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:53,518][ INFO][PID:2413595] VM allocation process starts [2026-04-01 11:48:53,530][ INFO][PID:2413595] Trying to allocate VM: ResallocHost, ticket_id=2132475, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-01 11:48:56,546][ INFO][PID:2413595] Allocated host ResallocHost, ticket_id=2132475, hostname=44.200.129.73, name=aws_x86_64_spot_prod_04450185_20260401_114544, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-01 11:48:56,547][ INFO][PID:2413595] Allocating ssh connection to builder [2026-04-01 11:48:56,547][ INFO][PID:2413595] Checking that builder machine is OK [2026-04-01 11:48:56,862][ INFO][PID:2413595] Running remote command: copr-builder-ready fedora-42-x86_64 [2026-04-01 11:48:57,087][ INFO][PID:2413595] Red Hat subscription not needed for fedora-42-x86_64 Builder is ready to be used [2026-04-01 11:48:57,088][ INFO][PID:2413595] Filling build.info file with builder info [2026-04-01 11:48:57,088][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:48:57,089][ INFO][PID:2413595] Not re-notifying FE and msg buses for the new host. [2026-04-01 11:48:57,089][ INFO][PID:2413595] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10282582-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached [2026-04-01 11:48:57,405][ INFO][PID:2413595] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '44.200.129.73' (ED25519) to the list of known hosts. [2026-04-01 11:48:57,406][ INFO][PID:2413595] Downloading the builder-live.log file, attempt 1 [2026-04-01 11:48:57,407][ INFO][PID:2413595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.129.73 copr-rpmbuild-log [2026-04-01 11:49:02,406][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:49:07,409][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:49:12,410][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:49:17,411][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:49:22,412][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:49:27,414][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:49:32,417][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:49:37,418][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:49:42,420][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:49:47,421][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:49:52,422][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:49:57,423][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:50:02,425][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:50:07,426][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:50:12,427][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:50:13,585][ ERROR][PID:2413595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-01 11:50:17,428][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:50:22,430][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:50:23,586][WARNING][PID:2413595] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-01 11:50:27,431][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:50:32,432][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:50:33,587][ INFO][PID:2413595] Downloading the builder-live.log file, attempt 2 [2026-04-01 11:50:33,590][ INFO][PID:2413595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.129.73 copr-rpmbuild-log [2026-04-01 11:50:37,434][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:50:42,436][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:50:47,439][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:50:52,441][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:50:57,444][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:51:02,530][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:51:07,537][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:51:12,690][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:51:17,715][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:51:22,721][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:51:27,725][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:51:32,808][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:51:33,661][ ERROR][PID:2413595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-01 11:51:37,895][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:51:43,027][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:51:43,666][WARNING][PID:2413595] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-01 11:51:48,037][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:51:53,045][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:51:53,667][ INFO][PID:2413595] Downloading the builder-live.log file, attempt 3 [2026-04-01 11:51:53,674][ INFO][PID:2413595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.129.73 copr-rpmbuild-log [2026-04-01 11:51:58,048][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:52:03,052][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:52:08,057][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:52:13,060][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:52:18,070][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:52:23,102][ INFO][PID:2413595] Periodic builder liveness probe: alive [2026-04-01 11:52:23,104][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:52:28,106][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:52:33,124][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:52:38,126][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:52:43,127][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:52:48,197][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:52:53,199][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:52:53,744][ ERROR][PID:2413595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-01 11:52:58,255][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:53:03,463][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:53:03,748][WARNING][PID:2413595] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-01 11:53:08,470][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:53:13,476][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:53:13,749][ INFO][PID:2413595] Downloading the builder-live.log file, attempt 4 [2026-04-01 11:53:13,757][ INFO][PID:2413595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.129.73 copr-rpmbuild-log [2026-04-01 11:53:18,493][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:53:23,608][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:53:28,613][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:53:33,623][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:53:38,978][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:53:43,981][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:53:48,984][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:53:53,990][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:53:58,998][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:54:04,019][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:54:09,271][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:54:13,829][ ERROR][PID:2413595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-01 11:54:14,534][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:54:19,536][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:54:24,196][ ERROR][PID:2413595] Giving up for unstable SSH, failures: 1775044223.585978, 1775044303.6661906, 1775044383.7488465, 1775044464.196853 [2026-04-01 11:54:24,197][ ERROR][PID:2413595] Re-try request for task on 'ResallocHost, ticket_id=2132475, hostname=44.200.129.73, name=aws_x86_64_spot_prod_04450185_20260401_114544, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-04-01 11:54:24,198][ INFO][PID:2413595] Releasing VM back to pool [2026-04-01 11:54:24,206][ INFO][PID:2413595] Retry #2 (on other host) [2026-04-01 11:54:24,206][ INFO][PID:2413595] Checking that /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/repodata/repomd.xml exists [2026-04-01 11:54:24,207][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:54:24,207][ INFO][PID:2413595] VM allocation process starts [2026-04-01 11:54:24,259][ INFO][PID:2413595] Trying to allocate VM: ResallocHost, ticket_id=2132624, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-01 11:54:27,306][ INFO][PID:2413595] Allocated host ResallocHost, ticket_id=2132624, hostname=54.146.215.186, name=aws_x86_64_normalreserved_prod_04450533_20260401_115002, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-01 11:54:27,306][ INFO][PID:2413595] Allocating ssh connection to builder [2026-04-01 11:54:27,307][ INFO][PID:2413595] Checking that builder machine is OK [2026-04-01 11:54:27,634][ INFO][PID:2413595] Running remote command: copr-builder-ready fedora-42-x86_64 [2026-04-01 11:54:27,839][ INFO][PID:2413595] Red Hat subscription not needed for fedora-42-x86_64 Builder is ready to be used [2026-04-01 11:54:27,840][ INFO][PID:2413595] Filling build.info file with builder info [2026-04-01 11:54:27,841][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:54:27,842][ INFO][PID:2413595] Not re-notifying FE and msg buses for the new host. [2026-04-01 11:54:27,843][ INFO][PID:2413595] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10282582-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached [2026-04-01 11:54:28,166][ INFO][PID:2413595] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.146.215.186' (ED25519) to the list of known hosts. [2026-04-01 11:54:28,167][ INFO][PID:2413595] Downloading the builder-live.log file, attempt 1 [2026-04-01 11:54:28,168][ INFO][PID:2413595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.146.215.186 copr-rpmbuild-log [2026-04-01 11:54:33,167][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:54:38,168][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:54:43,169][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:54:48,170][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:54:53,173][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:54:58,174][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:55:03,175][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:55:08,177][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:55:13,188][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:55:18,190][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:55:23,191][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:55:28,194][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:55:33,201][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:55:38,202][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:55:43,203][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:55:48,204][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:55:53,205][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:55:58,206][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:56:03,207][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:56:08,299][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:56:13,300][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:56:18,302][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:56:23,303][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:56:28,304][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:56:33,305][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:56:38,306][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:56:43,308][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:56:48,312][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:56:53,313][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:56:58,314][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:57:03,315][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:57:08,316][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:57:13,317][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:57:18,318][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:57:23,319][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:57:28,320][ INFO][PID:2413595] Checking for cancel request [2026-04-01 11:57:29,070][ INFO][PID:2413595] Downloading results from builder [2026-04-01 11:57:29,070][ INFO][PID:2413595] rsyncing of mockbuilder@54.146.215.186:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf started [2026-04-01 11:57:29,071][ INFO][PID:2413595] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@54.146.215.186:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/ &> /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/build-10282582.rsync.log [2026-04-01 11:57:29,439][ INFO][PID:2413595] rsyncing finished. [2026-04-01 11:57:29,439][ INFO][PID:2413595] Releasing VM back to pool [2026-04-01 11:57:29,448][ INFO][PID:2413595] Searching for 'success' file in resultdir [2026-04-01 11:57:29,449][ INFO][PID:2413595] Going to sign pkgs from source: 10282582-fedora-42-x86_64 in chroot: /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64 [2026-04-01 11:57:29,449][ INFO][PID:2413595] Calling '/bin/sign -u amatej#dnf-nightly@copr.fedorahosted.org -p' (attempt #1) [2026-04-01 11:57:29,729][ INFO][PID:2413595] Calling '/bin/sign -4 -h sha256 -u amatej#dnf-nightly@copr.fedorahosted.org -r /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/libdnf-debuginfo-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm' (attempt #1) [2026-04-01 11:57:30,640][ INFO][PID:2413595] signed rpm: /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/libdnf-debuginfo-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm [2026-04-01 11:57:30,640][ INFO][PID:2413595] Calling '/bin/sign -4 -h sha256 -u amatej#dnf-nightly@copr.fedorahosted.org -r /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/libdnf-debugsource-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm' (attempt #1) [2026-04-01 11:57:31,251][ INFO][PID:2413595] signed rpm: /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/libdnf-debugsource-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm [2026-04-01 11:57:31,251][ INFO][PID:2413595] Calling '/bin/sign -4 -h sha256 -u amatej#dnf-nightly@copr.fedorahosted.org -r /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/python3-hawkey-debuginfo-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm' (attempt #1) [2026-04-01 11:57:31,962][ INFO][PID:2413595] signed rpm: /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/python3-hawkey-debuginfo-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm [2026-04-01 11:57:31,963][ INFO][PID:2413595] Calling '/bin/sign -4 -h sha256 -u amatej#dnf-nightly@copr.fedorahosted.org -r /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/libdnf-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm' (attempt #1) [2026-04-01 11:57:32,618][ INFO][PID:2413595] signed rpm: /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/libdnf-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm [2026-04-01 11:57:32,618][ INFO][PID:2413595] Calling '/bin/sign -4 -h sha256 -u amatej#dnf-nightly@copr.fedorahosted.org -r /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/libdnf-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.src.rpm' (attempt #1) [2026-04-01 11:57:33,221][ INFO][PID:2413595] signed rpm: /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/libdnf-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.src.rpm [2026-04-01 11:57:33,221][ INFO][PID:2413595] Calling '/bin/sign -4 -h sha256 -u amatej#dnf-nightly@copr.fedorahosted.org -r /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/python3-libdnf-debuginfo-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm' (attempt #1) [2026-04-01 11:57:34,052][ INFO][PID:2413595] signed rpm: /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/python3-libdnf-debuginfo-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm [2026-04-01 11:57:34,053][ INFO][PID:2413595] Calling '/bin/sign -4 -h sha256 -u amatej#dnf-nightly@copr.fedorahosted.org -r /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/python3-libdnf-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm' (attempt #1) [2026-04-01 11:57:34,826][ INFO][PID:2413595] signed rpm: /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/python3-libdnf-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm [2026-04-01 11:57:34,826][ INFO][PID:2413595] Calling '/bin/sign -4 -h sha256 -u amatej#dnf-nightly@copr.fedorahosted.org -r /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/python3-hawkey-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm' (attempt #1) [2026-04-01 11:57:35,538][ INFO][PID:2413595] signed rpm: /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/python3-hawkey-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm [2026-04-01 11:57:35,539][ INFO][PID:2413595] Calling '/bin/sign -4 -h sha256 -u amatej#dnf-nightly@copr.fedorahosted.org -r /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/libdnf-devel-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm' (attempt #1) [2026-04-01 11:57:36,162][ INFO][PID:2413595] signed rpm: /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/libdnf-devel-0.76.0^20260401134112260344.master.931.ge2c749c7-1.fc42.x86_64.rpm [2026-04-01 11:57:36,162][ INFO][PID:2413595] Sign done [2026-04-01 11:57:36,163][ INFO][PID:2413595] Incremental createrepo run, adding 10282582-libdnf into https://download.copr.fedorainfracloud.org/results/amatej/dnf-nightly/fedora-42-x86_64, (auto-create-repo=True) [2026-04-01 11:57:36,164][ INFO][PID:2413595] Running command 'copr-repo --batched /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64 --add 10282582-libdnf --no-appstream-metadata' as PID 2533079 [2026-04-01 11:57:37,012][ INFO][PID:2413595] Finished after 0 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64 --add 10282582-libdnf --no-appstream-metadata) [2026-04-01 11:57:37,013][ INFO][PID:2413595] Getting build details [2026-04-01 11:57:37,013][ INFO][PID:2413595] Listing built binary packages in /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf [2026-04-01 11:57:37,013][ INFO][PID:2413595] Built packages: ['python3-libdnf 0.76.0^20260401134112260344.master.931.ge2c749c7', 'libdnf-debuginfo 0.76.0^20260401134112260344.master.931.ge2c749c7', 'libdnf 0.76.0^20260401134112260344.master.931.ge2c749c7', 'python3-hawkey-debuginfo 0.76.0^20260401134112260344.master.931.ge2c749c7', 'python3-hawkey 0.76.0^20260401134112260344.master.931.ge2c749c7', 'libdnf-devel 0.76.0^20260401134112260344.master.931.ge2c749c7', 'libdnf-debugsource 0.76.0^20260401134112260344.master.931.ge2c749c7', 'python3-libdnf-debuginfo 0.76.0^20260401134112260344.master.931.ge2c749c7'] [2026-04-01 11:57:37,014][ INFO][PID:2413595] build details: {'built_packages': 'python3-libdnf 0.76.0^20260401134112260344.master.931.ge2c749c7\nlibdnf-debuginfo 0.76.0^20260401134112260344.master.931.ge2c749c7\nlibdnf 0.76.0^20260401134112260344.master.931.ge2c749c7\npython3-hawkey-debuginfo 0.76.0^20260401134112260344.master.931.ge2c749c7\npython3-hawkey 0.76.0^20260401134112260344.master.931.ge2c749c7\nlibdnf-devel 0.76.0^20260401134112260344.master.931.ge2c749c7\nlibdnf-debugsource 0.76.0^20260401134112260344.master.931.ge2c749c7\npython3-libdnf-debuginfo 0.76.0^20260401134112260344.master.931.ge2c749c7'} [2026-04-01 11:57:37,014][ INFO][PID:2413595] Retrieving pubkey [2026-04-01 11:57:37,015][ INFO][PID:2413595] Calling '/bin/sign -u amatej#dnf-nightly@copr.fedorahosted.org -p' (attempt #1) [2026-04-01 11:57:37,289][ INFO][PID:2413595] Added pubkey for user amatej project dnf-nightly into: /var/lib/copr/public_html/results/amatej/dnf-nightly/pubkey.gpg [2026-04-01 11:57:37,290][ INFO][PID:2413595] Finished build: id=10282582 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/amatej/dnf-nightly chroot=fedora-42-x86_64 [2026-04-01 11:57:37,292][ INFO][PID:2413595] Worker succeeded build, took 189.4495198726654 [2026-04-01 11:57:37,292][ INFO][PID:2413595] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "amatej", "project_name": "dnf-nightly", "project_dirname": "dnf-nightly", "submitter": "amatej", "ended_on": 1775044657.2920675, "started_on": 1775044467.8425477, "submitted_on": null, "status": 1, "chroot": "fedora-42-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10282582-fedora-42-x86_64", "build_id": 10282582, "package_name": "libdnf", "package_version": "0.76.0^20260401134112260344.master.931.ge2c749c7-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/amatej/dnf-nightly/libdnf", "git_hash": "25cc6ce4bd80474d90b423e864dd6993613c99f8", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "amatej/dnf-nightly--amatej", "results": { "packages": [ { "name": "python3-libdnf", "epoch": null, "version": "0.76.0^20260401134112260344.master.931.ge2c749c7", "release": "1.fc42", "arch": "x86_64" }, { "name": "libdnf-debuginfo", "epoch": null, "version": "0.76.0^20260401134112260344.master.931.ge2c749c7", "release": "1.fc42", "arch": "x86_64" }, { "name": "libdnf", "epoch": null, "version": "0.76.0^20260401134112260344.master.931.ge2c749c7", "release": "1.fc42", "arch": "x86_64" }, { "name": "python3-hawkey-debuginfo", "epoch": null, "version": "0.76.0^20260401134112260344.master.931.ge2c749c7", "release": "1.fc42", "arch": "x86_64" }, { "name": "python3-hawkey", "epoch": null, "version": "0.76.0^20260401134112260344.master.931.ge2c749c7", "release": "1.fc42", "arch": "x86_64" }, { "name": "libdnf-devel", "epoch": null, "version": "0.76.0^20260401134112260344.master.931.ge2c749c7", "release": "1.fc42", "arch": "x86_64" }, { "name": "libdnf", "epoch": null, "version": "0.76.0^20260401134112260344.master.931.ge2c749c7", "release": "1.fc42", "arch": "src" }, { "name": "libdnf-debugsource", "epoch": null, "version": "0.76.0^20260401134112260344.master.931.ge2c749c7", "release": "1.fc42", "arch": "x86_64" }, { "name": "python3-libdnf-debuginfo", "epoch": null, "version": "0.76.0^20260401134112260344.master.931.ge2c749c7", "release": "1.fc42", "arch": "x86_64" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/amatej/dnf-nightly/fedora-42-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/rpmsoftwaremanagement/dnf-nightly/fedora-42-x86_64/", "id": "copr_rpmsoftwaremanagement_dnf_nightly", "name": "Additional repo copr_rpmsoftwaremanagement_dnf_nightly" } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/amatej/dnf-nightly", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/amatej/dnf-nightly", "result_dir": "10282582-libdnf", "built_packages": "python3-libdnf 0.76.0^20260401134112260344.master.931.ge2c749c7\nlibdnf-debuginfo 0.76.0^20260401134112260344.master.931.ge2c749c7\nlibdnf 0.76.0^20260401134112260344.master.931.ge2c749c7\npython3-hawkey-debuginfo 0.76.0^20260401134112260344.master.931.ge2c749c7\npython3-hawkey 0.76.0^20260401134112260344.master.931.ge2c749c7\nlibdnf-devel 0.76.0^20260401134112260344.master.931.ge2c749c7\nlibdnf-debugsource 0.76.0^20260401134112260344.master.931.ge2c749c7\npython3-libdnf-debuginfo 0.76.0^20260401134112260344.master.931.ge2c749c7", "id": 10282582, "mockchain_macros": { "copr_username": "amatej", "copr_projectname": "dnf-nightly", "vendor": "Fedora Project COPR (amatej/dnf-nightly)" } } ] } [2026-04-01 11:57:37,359][ INFO][PID:2413595] Sending fedora-messaging bus message in build.end [2026-04-01 11:57:37,387][ INFO][PID:2413595] Compressing /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/builder-live.log by gzip [2026-04-01 11:57:37,389][ INFO][PID:2413595] Running command 'gzip /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/builder-live.log' as PID 2533199 [2026-04-01 11:57:37,523][ INFO][PID:2413595] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/builder-live.log) [2026-04-01 11:57:37,524][ INFO][PID:2413595] Compressing /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/backend.log by gzip [2026-04-01 11:57:37,526][ INFO][PID:2413595] Running command 'gzip /var/lib/copr/public_html/results/amatej/dnf-nightly/fedora-42-x86_64/10282582-libdnf/backend.log' as PID 2533220