[2026-04-07 20:30:39,297][ INFO][PID:3113694] Marking build as starting [2026-04-07 20:30:39,339][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:30:39,340][ INFO][PID:3113694] VM allocation process starts [2026-04-07 20:30:39,461][ INFO][PID:3113694] Trying to allocate VM: ResallocHost, ticket_id=2211586, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-07 20:30:42,481][ INFO][PID:3113694] Allocated host ResallocHost, ticket_id=2211586, hostname=35.175.133.47, name=aws_x86_64_spot_prod_05912630_20260407_201748, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-07 20:30:42,482][ INFO][PID:3113694] Allocating ssh connection to builder [2026-04-07 20:30:42,483][ INFO][PID:3113694] Checking that builder machine is OK [2026-04-07 20:30:42,681][ INFO][PID:3113694] Running remote command: copr-builder-ready srpm-builds [2026-04-07 20:30:42,889][ INFO][PID:3113694] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-04-07 20:30:42,890][ INFO][PID:3113694] Filling build.info file with builder info [2026-04-07 20:30:42,891][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:30:42,893][ INFO][PID:3113694] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "averbyts", "project_name": "HEPrpms", "project_dirname": "HEPrpms", "submitter": "averbyts", "ended_on": null, "started_on": 1775593842.8932223, "submitted_on": null, "status": 3, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10298770", "build_id": 10298770, "package_name": "golem95", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#!/bin/bash\\ngit clone --depth 1 https://github.com/andriish/HEPrpms.git\\ncd HEPrpms\\nsh srpmsbuild.sh golem95 1.3.4\\n\", \"chroot\": \"epel-8-x86_64\", \"builddeps\": \"git spectool wget\", \"resultdir\": \"HEPrpms/golem95/1.3.4/rpmbuild/SOURCES/\", \"repos\": \"\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "averbyts/HEPrpms--averbyts", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "destdir": "/var/lib/copr/public_html/results/averbyts/HEPrpms", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms", "result_dir": "10298770", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 10298770, "mockchain_macros": { "copr_username": "averbyts", "copr_projectname": "HEPrpms", "vendor": "Fedora Project COPR (averbyts/HEPrpms)" } } ] } [2026-04-07 20:30:42,943][ INFO][PID:3113694] Sending fedora-messaging bus message in build.start [2026-04-07 20:30:43,631][ INFO][PID:3113694] Sending fedora-messaging bus message in chroot.start [2026-04-07 20:30:43,662][ INFO][PID:3113694] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10298770 --detached [2026-04-07 20:30:43,989][ INFO][PID:3113694] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '35.175.133.47' (ED25519) to the list of known hosts. [2026-04-07 20:30:43,992][ INFO][PID:3113694] Downloading the builder-live.log file, attempt 1 [2026-04-07 20:30:44,000][ INFO][PID:3113694] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.175.133.47 copr-rpmbuild-log [2026-04-07 20:30:49,004][ INFO][PID:3113694] Periodic builder liveness probe: alive [2026-04-07 20:30:49,004][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:30:54,006][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:30:59,007][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:31:04,008][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:31:09,009][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:31:14,010][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:31:19,011][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:31:24,012][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:31:29,014][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:31:34,015][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:31:39,016][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:31:44,017][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:31:49,018][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:31:54,019][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:31:59,042][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:32:04,043][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:32:09,044][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:32:11,931][ ERROR][PID:3113694] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-07 20:32:14,045][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:32:19,046][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:32:21,931][WARNING][PID:3113694] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-07 20:32:24,047][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:32:29,048][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:32:31,932][ INFO][PID:3113694] Downloading the builder-live.log file, attempt 2 [2026-04-07 20:32:31,934][ INFO][PID:3113694] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.175.133.47 copr-rpmbuild-log [2026-04-07 20:32:34,049][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:32:39,050][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:32:44,052][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:32:49,053][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:32:54,054][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:32:59,055][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:33:04,056][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:33:09,056][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:33:14,074][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:33:19,075][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:33:24,076][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:33:29,077][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:33:32,001][ ERROR][PID:3113694] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-07 20:33:34,078][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:33:39,079][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:33:42,028][WARNING][PID:3113694] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-07 20:33:44,080][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:33:49,081][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:33:52,028][ INFO][PID:3113694] Downloading the builder-live.log file, attempt 3 [2026-04-07 20:33:52,030][ INFO][PID:3113694] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.175.133.47 copr-rpmbuild-log [2026-04-07 20:33:54,082][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:33:59,083][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:34:04,084][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:34:09,085][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:34:14,085][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:34:19,086][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:34:24,087][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:34:29,089][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:34:34,090][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:34:39,091][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:34:44,092][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:34:49,093][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:34:52,096][ ERROR][PID:3113694] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-07 20:34:54,094][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:34:59,095][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:35:02,115][WARNING][PID:3113694] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-04-07 20:35:04,096][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:35:09,096][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:35:12,116][ INFO][PID:3113694] Downloading the builder-live.log file, attempt 4 [2026-04-07 20:35:12,117][ INFO][PID:3113694] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.175.133.47 copr-rpmbuild-log [2026-04-07 20:35:14,097][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:35:19,098][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:35:24,099][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:35:29,100][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:35:34,101][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:35:39,102][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:35:44,103][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:35:49,104][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:35:54,105][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:35:59,106][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:36:04,107][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:36:09,108][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:36:12,184][ ERROR][PID:3113694] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-04-07 20:36:14,109][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:36:19,110][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:36:22,193][ ERROR][PID:3113694] Giving up for unstable SSH, failures: 1775593941.9319336, 1775594022.0281615, 1775594102.1152844, 1775594182.1929395 [2026-04-07 20:36:22,194][ ERROR][PID:3113694] Re-try request for task on 'ResallocHost, ticket_id=2211586, hostname=35.175.133.47, name=aws_x86_64_spot_prod_05912630_20260407_201748, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-04-07 20:36:22,194][ INFO][PID:3113694] Releasing VM back to pool [2026-04-07 20:36:22,243][ INFO][PID:3113694] Retry #1 (on other host) [2026-04-07 20:36:22,244][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:36:22,244][ INFO][PID:3113694] VM allocation process starts [2026-04-07 20:36:22,293][ INFO][PID:3113694] Trying to allocate VM: ResallocHost, ticket_id=2211647, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-07 20:36:25,305][ INFO][PID:3113694] Allocated host ResallocHost, ticket_id=2211647, hostname=3.238.151.72, name=aws_x86_64_normalreserved_prod_05912734_20260407_202200, requested_tags=['copr_builder', 'arch_x86_64'] [2026-04-07 20:36:25,305][ INFO][PID:3113694] Allocating ssh connection to builder [2026-04-07 20:36:25,306][ INFO][PID:3113694] Checking that builder machine is OK [2026-04-07 20:36:25,520][ INFO][PID:3113694] Running remote command: copr-builder-ready srpm-builds [2026-04-07 20:36:25,729][ INFO][PID:3113694] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-04-07 20:36:25,730][ INFO][PID:3113694] Filling build.info file with builder info [2026-04-07 20:36:25,734][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:36:25,734][ INFO][PID:3113694] Not re-notifying FE and msg buses for the new host. [2026-04-07 20:36:25,735][ INFO][PID:3113694] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10298770 --detached [2026-04-07 20:36:26,060][ INFO][PID:3113694] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.238.151.72' (ED25519) to the list of known hosts. [2026-04-07 20:36:26,061][ INFO][PID:3113694] Downloading the builder-live.log file, attempt 1 [2026-04-07 20:36:26,139][ INFO][PID:3113694] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.238.151.72 copr-rpmbuild-log [2026-04-07 20:36:31,061][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:36:36,062][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:36:41,063][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:36:46,064][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:36:51,083][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:36:56,202][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:37:01,203][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:37:06,204][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:37:11,206][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:37:16,207][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:37:21,208][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:37:26,209][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:37:31,223][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:37:36,224][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:37:41,225][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:37:46,226][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:37:51,227][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:37:56,228][ INFO][PID:3113694] Checking for cancel request [2026-04-07 20:38:00,050][ INFO][PID:3113694] Downloading results from builder [2026-04-07 20:38:00,051][ INFO][PID:3113694] rsyncing of mockbuilder@3.238.151.72:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298770 started [2026-04-07 20:38:00,052][ INFO][PID:3113694] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@3.238.151.72:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298770/ &> /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298770/build-10298770.rsync.log [2026-04-07 20:38:00,308][ INFO][PID:3113694] rsyncing finished. [2026-04-07 20:38:00,309][ INFO][PID:3113694] Releasing VM back to pool [2026-04-07 20:38:00,318][ INFO][PID:3113694] Searching for 'success' file in resultdir [2026-04-07 20:38:00,319][ INFO][PID:3113694] Getting build details [2026-04-07 20:38:00,320][ INFO][PID:3113694] Retrieving SRPM info from /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298770 [2026-04-07 20:38:00,321][ INFO][PID:3113694] SRPM URL: https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms/srpm-builds/10298770/golem95-1.3.4-1.src.rpm [2026-04-07 20:38:00,321][ INFO][PID:3113694] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms/srpm-builds/10298770/golem95-1.3.4-1.src.rpm', 'pkg_name': 'golem95', 'pkg_version': '1.3.4-1'} [2026-04-07 20:38:00,321][ INFO][PID:3113694] Finished build: id=10298770 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/averbyts/HEPrpms chroot=srpm-builds [2026-04-07 20:38:00,323][ INFO][PID:3113694] Worker succeeded build, took 94.58842396736145 [2026-04-07 20:38:00,323][ INFO][PID:3113694] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "averbyts", "project_name": "HEPrpms", "project_dirname": "HEPrpms", "submitter": "averbyts", "ended_on": 1775594280.3233202, "started_on": 1775594185.7348962, "submitted_on": null, "status": 1, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10298770", "build_id": 10298770, "package_name": "golem95", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#!/bin/bash\\ngit clone --depth 1 https://github.com/andriish/HEPrpms.git\\ncd HEPrpms\\nsh srpmsbuild.sh golem95 1.3.4\\n\", \"chroot\": \"epel-8-x86_64\", \"builddeps\": \"git spectool wget\", \"resultdir\": \"HEPrpms/golem95/1.3.4/rpmbuild/SOURCES/\", \"repos\": \"\"}", "pkg_name": "golem95", "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": "https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms/srpm-builds/10298770/golem95-1.3.4-1.src.rpm", "uses_devel_repo": null, "sandbox": "averbyts/HEPrpms--averbyts", "results": { "name": "golem95", "epoch": null, "version": "1.3.4", "release": "1", "exclusivearch": [], "excludearch": [] }, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "destdir": "/var/lib/copr/public_html/results/averbyts/HEPrpms", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/averbyts/HEPrpms", "result_dir": "10298770", "built_packages": "", "tags": [ "arch_x86_64" ], "pkg_version": "1.3.4-1", "id": 10298770, "mockchain_macros": { "copr_username": "averbyts", "copr_projectname": "HEPrpms", "vendor": "Fedora Project COPR (averbyts/HEPrpms)" } } ] } [2026-04-07 20:38:00,366][ INFO][PID:3113694] Sending fedora-messaging bus message in build.end [2026-04-07 20:38:00,393][ INFO][PID:3113694] Compressing /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298770/builder-live.log by gzip [2026-04-07 20:38:00,394][ INFO][PID:3113694] Running command 'gzip /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298770/builder-live.log' as PID 3141004 [2026-04-07 20:38:00,411][ INFO][PID:3113694] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298770/builder-live.log) [2026-04-07 20:38:00,412][ INFO][PID:3113694] Compressing /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298770/backend.log by gzip [2026-04-07 20:38:00,413][ INFO][PID:3113694] Running command 'gzip /var/lib/copr/public_html/results/averbyts/HEPrpms/srpm-builds/10298770/backend.log' as PID 3141005