[2026-04-07 20:30:39,263][ INFO][PID:3113688] Marking build as starting [2026-04-07 20:30:39,310][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:30:39,313][ INFO][PID:3113688] VM allocation process starts [2026-04-07 20:30:39,339][ INFO][PID:3113688] Trying to allocate VM: ResallocHost, ticket_id=2211584, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-07 20:30:42,434][ INFO][PID:3113688] Allocated host ResallocHost, ticket_id=2211584, hostname=3.238.133.215, name=aws_x86_64_spot_prod_05912587_20260407_201631, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-07 20:30:42,435][ INFO][PID:3113688] Allocating ssh connection to builder [2026-04-07 20:30:42,436][ INFO][PID:3113688] Checking that builder machine is OK [2026-04-07 20:30:42,746][ INFO][PID:3113688] Running remote command: copr-builder-ready srpm-builds [2026-04-07 20:30:42,959][ INFO][PID:3113688] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-04-07 20:30:42,960][ INFO][PID:3113688] Filling build.info file with builder info [2026-04-07 20:30:42,960][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:30:42,962][ INFO][PID:3113688] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "averbyts", "project_name": "HEPrpms", "project_dirname": "HEPrpms", "submitter": "averbyts", "ended_on": null, "started_on": 1775593842.9622626, "submitted_on": null, "status": 3, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10298768", "build_id": 10298768, "package_name": "recola2", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#!/bin/bash\\ngit clone --depth 1 https://github.com/andriish/HEPrpms.git\\ncd HEPrpms\\nsh srpmsbuild.sh recola2 2.3.0\\n\", \"chroot\": \"epel-8-x86_64\", \"builddeps\": \"git spectool wget\", \"resultdir\": \"HEPrpms/recola2/2.3.0/rpmbuild/SOURCES/\", \"repos\": \"\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "averbyts/HEPrpms--averbyts", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "destdir": "/var/lib/copr/public_html/results/averbyts/HEPrpms", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms", "result_dir": "10298768", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 10298768, "mockchain_macros": { "copr_username": "averbyts", "copr_projectname": "HEPrpms", "vendor": "Fedora Project COPR (averbyts/HEPrpms)" } } ] } [2026-04-07 20:30:43,005][ INFO][PID:3113688] Sending fedora-messaging bus message in build.start [2026-04-07 20:30:43,653][ INFO][PID:3113688] Sending fedora-messaging bus message in chroot.start [2026-04-07 20:30:43,682][ INFO][PID:3113688] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10298768 --detached [2026-04-07 20:30:43,999][ INFO][PID:3113688] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.238.133.215' (ED25519) to the list of known hosts. [2026-04-07 20:30:44,001][ INFO][PID:3113688] Downloading the builder-live.log file, attempt 1 [2026-04-07 20:30:44,003][ INFO][PID:3113688] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.133.215 copr-rpmbuild-log [2026-04-07 20:30:49,010][ INFO][PID:3113688] Periodic builder liveness probe: alive [2026-04-07 20:30:49,013][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:30:54,015][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:30:59,016][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:31:04,021][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:31:09,022][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:31:14,023][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:31:19,024][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:31:24,025][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:31:29,026][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:31:34,027][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:31:39,028][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:31:44,029][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:31:49,030][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:31:54,031][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:31:59,042][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:32:04,043][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:32:09,044][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:32:11,878][ ERROR][PID:3113688] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-07 20:32:14,045][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:32:19,046][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:32:21,879][WARNING][PID:3113688] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-07 20:32:24,047][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:32:29,048][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:32:31,879][ INFO][PID:3113688] Downloading the builder-live.log file, attempt 2 [2026-04-07 20:32:31,881][ INFO][PID:3113688] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.133.215 copr-rpmbuild-log [2026-04-07 20:32:34,049][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:32:39,050][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:32:44,051][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:32:49,052][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:32:54,053][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:32:59,054][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:33:04,055][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:33:09,055][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:33:14,056][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:33:19,057][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:33:24,058][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:33:29,059][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:33:31,943][ ERROR][PID:3113688] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-07 20:33:34,060][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:33:39,060][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:33:41,958][WARNING][PID:3113688] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-07 20:33:44,061][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:33:49,062][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:33:51,959][ INFO][PID:3113688] Downloading the builder-live.log file, attempt 3 [2026-04-07 20:33:51,960][ INFO][PID:3113688] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.133.215 copr-rpmbuild-log [2026-04-07 20:33:54,063][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:33:59,064][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:34:04,065][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:34:09,066][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:34:14,067][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:34:19,068][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:34:24,069][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:34:29,071][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:34:34,072][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:34:39,072][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:34:44,073][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:34:49,074][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:34:52,027][ ERROR][PID:3113688] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-07 20:34:54,075][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:34:59,077][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:35:02,058][WARNING][PID:3113688] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-07 20:35:04,083][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:35:09,084][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:35:12,058][ INFO][PID:3113688] Downloading the builder-live.log file, attempt 4 [2026-04-07 20:35:12,059][ INFO][PID:3113688] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.133.215 copr-rpmbuild-log [2026-04-07 20:35:14,085][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:35:19,086][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:35:24,087][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:35:29,088][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:35:34,089][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:35:39,090][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:35:44,092][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:35:49,093][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:35:54,093][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:35:59,094][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:36:04,095][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:36:09,096][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:36:12,126][ ERROR][PID:3113688] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-07 20:36:14,097][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:36:19,098][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:36:22,176][ ERROR][PID:3113688] Giving up for unstable SSH, failures: 1775593941.8789911, 1775594021.9584024, 1775594102.0579789, 1775594182.1767247 [2026-04-07 20:36:22,177][ ERROR][PID:3113688] Re-try request for task on 'ResallocHost, ticket_id=2211584, hostname=3.238.133.215, name=aws_x86_64_spot_prod_05912587_20260407_201631, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-04-07 20:36:22,178][ INFO][PID:3113688] Releasing VM back to pool [2026-04-07 20:36:22,186][ INFO][PID:3113688] Retry #1 (on other host) [2026-04-07 20:36:22,187][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:36:22,187][ INFO][PID:3113688] VM allocation process starts [2026-04-07 20:36:22,249][ INFO][PID:3113688] Trying to allocate VM: ResallocHost, ticket_id=2211646, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-07 20:36:25,293][ INFO][PID:3113688] Allocated host ResallocHost, ticket_id=2211646, hostname=100.48.61.201, name=aws_x86_64_normalreserved_prod_05912618_20260407_201733, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-07 20:36:25,294][ INFO][PID:3113688] Allocating ssh connection to builder [2026-04-07 20:36:25,294][ INFO][PID:3113688] Checking that builder machine is OK [2026-04-07 20:36:25,661][ INFO][PID:3113688] Running remote command: copr-builder-ready srpm-builds [2026-04-07 20:36:25,887][ INFO][PID:3113688] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-04-07 20:36:25,887][ INFO][PID:3113688] Filling build.info file with builder info [2026-04-07 20:36:25,896][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:36:25,897][ INFO][PID:3113688] Not re-notifying FE and msg buses for the new host. [2026-04-07 20:36:25,897][ INFO][PID:3113688] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10298768 --detached [2026-04-07 20:36:26,242][ INFO][PID:3113688] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '100.48.61.201' (ED25519) to the list of known hosts. [2026-04-07 20:36:26,243][ INFO][PID:3113688] Downloading the builder-live.log file, attempt 1 [2026-04-07 20:36:26,245][ INFO][PID:3113688] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@100.48.61.201 copr-rpmbuild-log [2026-04-07 20:36:31,244][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:36:36,245][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:36:41,246][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:36:46,247][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:36:51,248][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:36:56,249][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:37:01,250][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:37:06,251][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:37:11,252][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:37:16,253][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:37:21,254][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:37:26,255][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:37:31,256][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:37:36,257][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:37:41,258][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:37:46,259][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:37:51,260][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:37:56,265][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:38:01,266][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:38:06,267][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:38:11,268][ INFO][PID:3113688] Checking for cancel request [2026-04-07 20:38:11,645][ INFO][PID:3113688] Downloading results from builder [2026-04-07 20:38:11,646][ INFO][PID:3113688] rsyncing of mockbuilder@100.48.61.201:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298768 started [2026-04-07 20:38:11,646][ INFO][PID:3113688] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@100.48.61.201:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298768/ &> /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298768/build-10298768.rsync.log [2026-04-07 20:38:11,983][ INFO][PID:3113688] rsyncing finished. [2026-04-07 20:38:11,984][ INFO][PID:3113688] Releasing VM back to pool [2026-04-07 20:38:11,994][ INFO][PID:3113688] Searching for 'success' file in resultdir [2026-04-07 20:38:11,995][ INFO][PID:3113688] Getting build details [2026-04-07 20:38:11,995][ INFO][PID:3113688] Retrieving SRPM info from /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298768 [2026-04-07 20:38:11,996][ INFO][PID:3113688] SRPM URL: https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms/srpm-builds/10298768/recola2-2.3.0-2.src.rpm [2026-04-07 20:38:11,996][ INFO][PID:3113688] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms/srpm-builds/10298768/recola2-2.3.0-2.src.rpm', 'pkg_name': 'recola2', 'pkg_version': '2.3.0-2'} [2026-04-07 20:38:11,996][ INFO][PID:3113688] Finished build: id=10298768 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/averbyts/HEPrpms chroot=srpm-builds [2026-04-07 20:38:11,998][ INFO][PID:3113688] Worker succeeded build, took 106.10132241249084 [2026-04-07 20:38:11,998][ INFO][PID:3113688] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "averbyts", "project_name": "HEPrpms", "project_dirname": "HEPrpms", "submitter": "averbyts", "ended_on": 1775594291.9984136, "started_on": 1775594185.8970912, "submitted_on": null, "status": 1, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10298768", "build_id": 10298768, "package_name": "recola2", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#!/bin/bash\\ngit clone --depth 1 https://github.com/andriish/HEPrpms.git\\ncd HEPrpms\\nsh srpmsbuild.sh recola2 2.3.0\\n\", \"chroot\": \"epel-8-x86_64\", \"builddeps\": \"git spectool wget\", \"resultdir\": \"HEPrpms/recola2/2.3.0/rpmbuild/SOURCES/\", \"repos\": \"\"}", "pkg_name": "recola2", "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": "https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms/srpm-builds/10298768/recola2-2.3.0-2.src.rpm", "uses_devel_repo": null, "sandbox": "averbyts/HEPrpms--averbyts", "results": { "name": "recola2", "epoch": null, "version": "2.3.0", "release": "2", "exclusivearch": [], "excludearch": [] }, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "destdir": "/var/lib/copr/public_html/results/averbyts/HEPrpms", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms", "result_dir": "10298768", "built_packages": "", "tags": [ "arch_x86_64" ], "pkg_version": "2.3.0-2", "id": 10298768, "mockchain_macros": { "copr_username": "averbyts", "copr_projectname": "HEPrpms", "vendor": "Fedora Project COPR (averbyts/HEPrpms)" } } ] } [2026-04-07 20:38:12,044][ INFO][PID:3113688] Sending fedora-messaging bus message in build.end [2026-04-07 20:38:12,071][ INFO][PID:3113688] Compressing /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298768/builder-live.log by gzip [2026-04-07 20:38:12,072][ INFO][PID:3113688] Running command 'gzip /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298768/builder-live.log' as PID 3141488 [2026-04-07 20:38:12,130][ INFO][PID:3113688] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298768/builder-live.log) [2026-04-07 20:38:12,131][ INFO][PID:3113688] Compressing /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298768/backend.log by gzip [2026-04-07 20:38:12,132][ INFO][PID:3113688] Running command 'gzip /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298768/backend.log' as PID 3141494