[2026-03-26 14:03:46,074][ INFO][PID:3050757] Marking build as starting [2026-03-26 14:03:46,111][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:03:46,113][ INFO][PID:3050757] VM allocation process starts [2026-03-26 14:03:46,124][ INFO][PID:3050757] Trying to allocate VM: ResallocHost, ticket_id=2041127, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-26 14:03:51,125][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:03:52,209][ INFO][PID:3050757] Allocated host ResallocHost, ticket_id=2041127, hostname=52.206.246.66, name=aws_x86_64_spot_prod_04291728_20260326_140106, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-26 14:03:52,210][ INFO][PID:3050757] Allocating ssh connection to builder [2026-03-26 14:03:52,211][ INFO][PID:3050757] Checking that builder machine is OK [2026-03-26 14:03:52,563][ INFO][PID:3050757] Running remote command: copr-builder-ready srpm-builds [2026-03-26 14:03:52,806][ INFO][PID:3050757] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-03-26 14:03:52,806][ INFO][PID:3050757] Filling build.info file with builder info [2026-03-26 14:03:52,807][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:03:52,808][ INFO][PID:3050757] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "@cockpit", "project_name": "main-builds", "project_dirname": "main-builds", "submitter": "packit", "ended_on": null, "started_on": 1774533832.8080282, "submitted_on": null, "status": 3, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10264011", "build_id": 10264011, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"\\n#!/bin/sh\\n\\ngit config --global user.email \\\"hello@packit.dev\\\"\\ngit config --global user.name \\\"Packit\\\"\\nresultdir=$PWD\\npackit -d prepare-sources --result-dir \\\"$resultdir\\\" --ref main --job-config-index 3 --merged-ref '6216b992c80268d171a1c575384f180921ad977e' --no-create-symlinks https://github.com/cockpit-project/cockpit-machines\\n\\n\", \"chroot\": \"fedora-43-x86_64\", \"builddeps\": \"packit jq make nodejs\", \"resultdir\": null, \"repos\": \"copr://packit/packit-stable\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "@cockpit/main-builds--packit", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/packit-stable/fedora-43-x86_64/", "id": "copr_packit_packit_stable", "name": "Additional repo copr_packit_packit_stable" } ], "background": false, "destdir": "/var/lib/copr/public_html/results/@cockpit/main-builds", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@cockpit/main-builds", "result_dir": "10264011", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 10264011, "mockchain_macros": { "copr_username": "@cockpit", "copr_projectname": "main-builds", "vendor": "Fedora Project COPR (@cockpit/main-builds)" } } ] } [2026-03-26 14:03:52,869][ INFO][PID:3050757] Sending fedora-messaging bus message in build.start [2026-03-26 14:03:53,566][ INFO][PID:3050757] Sending fedora-messaging bus message in chroot.start [2026-03-26 14:03:53,594][ INFO][PID:3050757] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10264011 --detached [2026-03-26 14:03:53,933][ INFO][PID:3050757] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '52.206.246.66' (ED25519) to the list of known hosts. [2026-03-26 14:03:53,934][ INFO][PID:3050757] Downloading the builder-live.log file, attempt 1 [2026-03-26 14:03:53,935][ INFO][PID:3050757] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.206.246.66 copr-rpmbuild-log [2026-03-26 14:03:58,947][ INFO][PID:3050757] Periodic builder liveness probe: alive [2026-03-26 14:03:58,948][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:04:03,949][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:04:08,950][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:04:13,951][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:04:18,958][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:04:23,962][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:04:28,966][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:04:33,968][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:04:38,971][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:04:43,974][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:04:48,980][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:04:53,988][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:04:54,795][ ERROR][PID:3050757] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 14:04:58,994][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:05:03,995][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:05:04,796][WARNING][PID:3050757] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-26 14:05:08,996][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:05:13,997][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:05:14,797][ INFO][PID:3050757] Downloading the builder-live.log file, attempt 2 [2026-03-26 14:05:14,801][ INFO][PID:3050757] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.206.246.66 copr-rpmbuild-log [2026-03-26 14:05:19,010][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:05:24,012][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:05:29,019][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:05:34,025][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:05:39,026][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:05:44,030][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:05:49,035][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:05:54,036][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:05:59,038][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:06:04,039][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:06:09,046][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:06:14,050][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:06:14,877][ ERROR][PID:3050757] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 14:06:19,051][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:06:24,056][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:06:24,890][WARNING][PID:3050757] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-26 14:06:29,057][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:06:34,058][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:06:34,891][ INFO][PID:3050757] Downloading the builder-live.log file, attempt 3 [2026-03-26 14:06:34,893][ INFO][PID:3050757] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.206.246.66 copr-rpmbuild-log [2026-03-26 14:06:39,060][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:06:44,061][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:06:49,062][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:06:54,063][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:06:59,064][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:07:04,066][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:07:09,079][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:07:14,080][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:07:19,089][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:07:24,090][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:07:29,099][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:07:34,099][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:07:34,960][ ERROR][PID:3050757] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 14:07:39,107][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:07:44,112][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:07:44,975][WARNING][PID:3050757] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-26 14:07:49,113][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:07:54,114][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:07:54,976][ INFO][PID:3050757] Downloading the builder-live.log file, attempt 4 [2026-03-26 14:07:54,978][ INFO][PID:3050757] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.206.246.66 copr-rpmbuild-log [2026-03-26 14:07:59,116][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:08:04,117][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:08:09,118][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:08:14,135][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:08:19,137][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:08:24,142][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:08:29,144][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:08:34,145][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:08:39,146][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:08:44,147][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:08:49,158][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:08:54,160][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:08:55,046][ ERROR][PID:3050757] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-26 14:08:59,161][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:09:04,162][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:09:05,059][ ERROR][PID:3050757] Giving up for unstable SSH, failures: 1774533904.7962065, 1774533984.8905268, 1774534064.9752445, 1774534145.05868 [2026-03-26 14:09:05,060][ ERROR][PID:3050757] Re-try request for task on 'ResallocHost, ticket_id=2041127, hostname=52.206.246.66, name=aws_x86_64_spot_prod_04291728_20260326_140106, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-03-26 14:09:05,061][ INFO][PID:3050757] Releasing VM back to pool [2026-03-26 14:09:05,074][ INFO][PID:3050757] Retry #1 (on other host) [2026-03-26 14:09:05,076][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:09:05,077][ INFO][PID:3050757] VM allocation process starts [2026-03-26 14:09:05,090][ INFO][PID:3050757] Trying to allocate VM: ResallocHost, ticket_id=2041276, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-26 14:09:10,091][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:09:15,092][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:09:20,094][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:09:25,102][ INFO][PID:3050757] Checking for cancel request [2026-03-26 14:09:25,105][ INFO][PID:3050757] calling cancel callback, and waiting [2026-03-26 14:09:25,111][ INFO][PID:3050757] Releasing VM back to pool [2026-03-26 14:09:33,129][ ERROR][PID:3050757] Build was canceled [2026-03-26 14:09:33,153][WARNING][PID:3050757] Switching not-finished job state to 'failed' [2026-03-26 14:09:33,153][ INFO][PID:3050757] Worker failed build, took 340.3450062274933 [2026-03-26 14:09:33,154][ INFO][PID:3050757] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "@cockpit", "project_name": "main-builds", "project_dirname": "main-builds", "submitter": "packit", "ended_on": 1774534173.1530344, "started_on": 1774533832.8080282, "submitted_on": null, "status": 0, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10264011", "build_id": 10264011, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"\\n#!/bin/sh\\n\\ngit config --global user.email \\\"hello@packit.dev\\\"\\ngit config --global user.name \\\"Packit\\\"\\nresultdir=$PWD\\npackit -d prepare-sources --result-dir \\\"$resultdir\\\" --ref main --job-config-index 3 --merged-ref '6216b992c80268d171a1c575384f180921ad977e' --no-create-symlinks https://github.com/cockpit-project/cockpit-machines\\n\\n\", \"chroot\": \"fedora-43-x86_64\", \"builddeps\": \"packit jq make nodejs\", \"resultdir\": null, \"repos\": \"copr://packit/packit-stable\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "@cockpit/main-builds--packit", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/packit-stable/fedora-43-x86_64/", "id": "copr_packit_packit_stable", "name": "Additional repo copr_packit_packit_stable" } ], "background": false, "destdir": "/var/lib/copr/public_html/results/@cockpit/main-builds", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@cockpit/main-builds", "result_dir": "10264011", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 10264011, "mockchain_macros": { "copr_username": "@cockpit", "copr_projectname": "main-builds", "vendor": "Fedora Project COPR (@cockpit/main-builds)" } } ] } [2026-03-26 14:09:33,203][ INFO][PID:3050757] Sending fedora-messaging bus message in build.end [2026-03-26 14:09:33,238][ INFO][PID:3050757] Compressing /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10264011/builder-live.log by gzip [2026-03-26 14:09:33,240][ INFO][PID:3050757] Running command 'gzip /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10264011/builder-live.log' as PID 3099816 [2026-03-26 14:09:33,425][ INFO][PID:3050757] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10264011/builder-live.log) [2026-03-26 14:09:33,427][ INFO][PID:3050757] Compressing /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10264011/backend.log by gzip [2026-03-26 14:09:33,431][ INFO][PID:3050757] Running command 'gzip /var/lib/copr/public_html/results/@cockpit/main-builds/srpm-builds/10264011/backend.log' as PID 3099843