[2026-02-09 20:33:44,477][ INFO][PID:2452924] Marking build as starting [2026-02-09 20:33:44,530][ INFO][PID:2452924] Checking that /var/lib/copr/public_html/results/garaevdi/pantheon-rpm/fedora-44-x86_64/repodata/repomd.xml exists [2026-02-09 20:33:44,531][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:33:44,532][ INFO][PID:2452924] VM allocation process starts [2026-02-09 20:33:44,544][ INFO][PID:2452924] Trying to allocate VM: ResallocHost, ticket_id=1244151, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-09 20:33:47,558][ INFO][PID:2452924] Allocated host ResallocHost, ticket_id=1244151, hostname=13.217.86.52, name=aws_x86_64_spot_prod_02103009_20260209_202543, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-09 20:33:47,559][ INFO][PID:2452924] Allocating ssh connection to builder [2026-02-09 20:33:47,560][ INFO][PID:2452924] Checking that builder machine is OK [2026-02-09 20:33:47,890][ INFO][PID:2452924] Running remote command: copr-builder-ready fedora-44-x86_64 [2026-02-09 20:33:48,124][ INFO][PID:2452924] Red Hat subscription not needed for fedora-44-x86_64 Builder is ready to be used [2026-02-09 20:33:48,124][ INFO][PID:2452924] Filling build.info file with builder info [2026-02-09 20:33:48,125][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:33:48,126][ INFO][PID:2452924] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "garaevdi", "project_name": "pantheon-rpm", "project_dirname": "pantheon-rpm", "submitter": "garaevdi", "ended_on": null, "started_on": 1770669228.1259186, "submitted_on": null, "status": 3, "chroot": "fedora-44-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10110064-fedora-44-x86_64", "build_id": 10110064, "package_name": "elementary-monitor", "package_version": "8.0.1-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/garaevdi/pantheon-rpm/elementary-monitor", "git_hash": "8ca0e53cb067779158ed382e73af174f17b1c9ef", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "garaevdi/pantheon-rpm--garaevdi", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/garaevdi/pantheon-rpm/fedora-44-x86_64/", "id": "copr_base", "module_hotfixes": true, "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/garaevdi/pantheon-rpm", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/garaevdi/pantheon-rpm", "result_dir": "10110064-elementary-monitor", "built_packages": "", "id": 10110064, "mockchain_macros": { "copr_username": "garaevdi", "copr_projectname": "pantheon-rpm", "vendor": "Fedora Project COPR (garaevdi/pantheon-rpm)" } } ] } [2026-02-09 20:33:48,166][ INFO][PID:2452924] Sending fedora-messaging bus message in build.start [2026-02-09 20:33:48,739][ INFO][PID:2452924] Sending fedora-messaging bus message in chroot.start [2026-02-09 20:33:48,773][ INFO][PID:2452924] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10110064-fedora-44-x86_64 --chroot fedora-44-x86_64 --detached [2026-02-09 20:33:49,104][ INFO][PID:2452924] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '13.217.86.52' (ED25519) to the list of known hosts. [2026-02-09 20:33:49,105][ INFO][PID:2452924] Downloading the builder-live.log file, attempt 1 [2026-02-09 20:33:49,106][ INFO][PID:2452924] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.217.86.52 copr-rpmbuild-log [2026-02-09 20:33:54,119][ INFO][PID:2452924] Periodic builder liveness probe: alive [2026-02-09 20:33:54,120][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:33:59,121][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:34:04,122][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:34:09,123][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:34:14,124][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:34:19,124][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:34:22,312][ ERROR][PID:2452924] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-09 20:34:24,125][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:34:29,126][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:34:32,313][WARNING][PID:2452924] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-09 20:34:34,128][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:34:39,129][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:34:42,313][ INFO][PID:2452924] Downloading the builder-live.log file, attempt 2 [2026-02-09 20:34:42,315][ INFO][PID:2452924] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.217.86.52 copr-rpmbuild-log [2026-02-09 20:34:44,130][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:34:49,134][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:34:54,135][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:34:59,137][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:35:04,138][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:35:09,139][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:35:14,140][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:35:19,163][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:35:24,164][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:35:29,165][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:35:34,166][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:35:39,167][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:35:42,383][ ERROR][PID:2452924] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-09 20:35:44,214][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:35:49,215][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:35:52,384][WARNING][PID:2452924] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-09 20:35:54,217][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:35:59,218][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:36:02,385][ INFO][PID:2452924] Downloading the builder-live.log file, attempt 3 [2026-02-09 20:36:02,388][ INFO][PID:2452924] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.217.86.52 copr-rpmbuild-log [2026-02-09 20:36:04,219][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:36:09,221][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:36:14,224][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:36:19,226][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:36:24,226][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:36:29,228][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:36:34,233][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:36:39,234][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:36:44,235][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:36:49,236][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:36:54,238][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:36:59,239][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:37:02,457][ ERROR][PID:2452924] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-09 20:37:04,240][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:37:09,241][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:37:12,465][WARNING][PID:2452924] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-09 20:37:14,242][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:37:19,243][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:37:22,466][ INFO][PID:2452924] Downloading the builder-live.log file, attempt 4 [2026-02-09 20:37:22,525][ INFO][PID:2452924] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.217.86.52 copr-rpmbuild-log [2026-02-09 20:37:24,244][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:37:29,245][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:37:34,246][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:37:39,247][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:37:44,248][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:37:49,249][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:37:54,250][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:37:59,251][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:38:04,252][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:38:09,253][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:38:14,254][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:38:19,255][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:38:22,590][ ERROR][PID:2452924] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-09 20:38:24,256][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:38:29,257][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:38:32,603][ ERROR][PID:2452924] Giving up for unstable SSH, failures: 1770669272.3129733, 1770669352.3842123, 1770669432.4656596, 1770669512.603724 [2026-02-09 20:38:32,604][ ERROR][PID:2452924] Re-try request for task on 'ResallocHost, ticket_id=1244151, hostname=13.217.86.52, name=aws_x86_64_spot_prod_02103009_20260209_202543, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-02-09 20:38:32,605][ INFO][PID:2452924] Releasing VM back to pool [2026-02-09 20:38:32,613][ INFO][PID:2452924] Retry #1 (on other host) [2026-02-09 20:38:32,614][ INFO][PID:2452924] Checking that /var/lib/copr/public_html/results/garaevdi/pantheon-rpm/fedora-44-x86_64/repodata/repomd.xml exists [2026-02-09 20:38:32,614][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:38:32,615][ INFO][PID:2452924] VM allocation process starts [2026-02-09 20:38:32,623][ INFO][PID:2452924] Trying to allocate VM: ResallocHost, ticket_id=1244246, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-09 20:38:35,636][ INFO][PID:2452924] Allocated host ResallocHost, ticket_id=1244246, hostname=13.221.77.73, name=aws_x86_64_normalreserved_prod_02103073_20260209_203105, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-09 20:38:35,637][ INFO][PID:2452924] Allocating ssh connection to builder [2026-02-09 20:38:35,637][ INFO][PID:2452924] Checking that builder machine is OK [2026-02-09 20:38:35,987][ INFO][PID:2452924] Running remote command: copr-builder-ready fedora-44-x86_64 [2026-02-09 20:38:36,199][ INFO][PID:2452924] Red Hat subscription not needed for fedora-44-x86_64 Builder is ready to be used [2026-02-09 20:38:36,200][ INFO][PID:2452924] Filling build.info file with builder info [2026-02-09 20:38:36,230][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:38:36,231][ INFO][PID:2452924] Not re-notifying FE and msg buses for the new host. [2026-02-09 20:38:36,231][ INFO][PID:2452924] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10110064-fedora-44-x86_64 --chroot fedora-44-x86_64 --detached [2026-02-09 20:38:36,568][ INFO][PID:2452924] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '13.221.77.73' (ED25519) to the list of known hosts. [2026-02-09 20:38:36,568][ INFO][PID:2452924] Downloading the builder-live.log file, attempt 1 [2026-02-09 20:38:36,570][ INFO][PID:2452924] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.221.77.73 copr-rpmbuild-log [2026-02-09 20:38:41,569][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:38:46,570][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:38:51,571][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:38:56,572][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:39:01,573][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:39:06,574][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:39:11,575][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:39:16,576][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:39:21,578][ INFO][PID:2452924] Checking for cancel request [2026-02-09 20:39:21,767][ INFO][PID:2452924] Downloading results from builder [2026-02-09 20:39:21,768][ INFO][PID:2452924] rsyncing of mockbuilder@13.221.77.73:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/garaevdi/pantheon-rpm/fedora-44-x86_64/10110064-elementary-monitor started [2026-02-09 20:39:21,769][ INFO][PID:2452924] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@13.221.77.73:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/garaevdi/pantheon-rpm/fedora-44-x86_64/10110064-elementary-monitor/ &> /var/lib/copr/public_html/results/garaevdi/pantheon-rpm/fedora-44-x86_64/10110064-elementary-monitor/build-10110064.rsync.log [2026-02-09 20:39:22,305][ INFO][PID:2452924] rsyncing finished. [2026-02-09 20:39:22,305][ INFO][PID:2452924] Releasing VM back to pool [2026-02-09 20:39:22,315][ INFO][PID:2452924] Searching for 'success' file in resultdir [2026-02-09 20:39:22,316][ ERROR][PID:2452924] Build failed: Backend process error: No success file => build failure [2026-02-09 20:39:22,317][ INFO][PID:2452924] Finished build: id=10110064 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/garaevdi/pantheon-rpm chroot=fedora-44-x86_64 [2026-02-09 20:39:22,319][ ERROR][PID:2452924] Backend process error: No success file => build failure [2026-02-09 20:39:22,319][ INFO][PID:2452924] Worker failed build, took 46.08846855163574 [2026-02-09 20:39:22,320][ INFO][PID:2452924] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "garaevdi", "project_name": "pantheon-rpm", "project_dirname": "pantheon-rpm", "submitter": "garaevdi", "ended_on": 1770669562.3195887, "started_on": 1770669516.23112, "submitted_on": null, "status": 0, "chroot": "fedora-44-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "10110064-fedora-44-x86_64", "build_id": 10110064, "package_name": "elementary-monitor", "package_version": "8.0.1-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/garaevdi/pantheon-rpm/elementary-monitor", "git_hash": "8ca0e53cb067779158ed382e73af174f17b1c9ef", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "garaevdi/pantheon-rpm--garaevdi", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/garaevdi/pantheon-rpm/fedora-44-x86_64/", "id": "copr_base", "module_hotfixes": true, "name": "Copr repository", "priority": null } ], "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/garaevdi/pantheon-rpm", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/garaevdi/pantheon-rpm", "result_dir": "10110064-elementary-monitor", "built_packages": "", "id": 10110064, "mockchain_macros": { "copr_username": "garaevdi", "copr_projectname": "pantheon-rpm", "vendor": "Fedora Project COPR (garaevdi/pantheon-rpm)" } } ] } [2026-02-09 20:39:22,364][ INFO][PID:2452924] Sending fedora-messaging bus message in build.end [2026-02-09 20:39:22,396][ INFO][PID:2452924] Compressing /var/lib/copr/public_html/results/garaevdi/pantheon-rpm/fedora-44-x86_64/10110064-elementary-monitor/builder-live.log by gzip [2026-02-09 20:39:22,397][ INFO][PID:2452924] Running command 'gzip /var/lib/copr/public_html/results/garaevdi/pantheon-rpm/fedora-44-x86_64/10110064-elementary-monitor/builder-live.log' as PID 2483230 [2026-02-09 20:39:22,647][ INFO][PID:2452924] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/garaevdi/pantheon-rpm/fedora-44-x86_64/10110064-elementary-monitor/builder-live.log) [2026-02-09 20:39:22,648][ INFO][PID:2452924] Compressing /var/lib/copr/public_html/results/garaevdi/pantheon-rpm/fedora-44-x86_64/10110064-elementary-monitor/backend.log by gzip [2026-02-09 20:39:22,649][ INFO][PID:2452924] Running command 'gzip /var/lib/copr/public_html/results/garaevdi/pantheon-rpm/fedora-44-x86_64/10110064-elementary-monitor/backend.log' as PID 2483274