[2026-01-12 11:31:35,189][  INFO][PID:3295149] Marking build as starting
[2026-01-12 11:31:35,256][  INFO][PID:3295149] Checking that /var/lib/copr/public_html/results/packit/ansasaki-rust-keylime-master-fedora/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-01-12 11:31:35,332][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:31:35,333][  INFO][PID:3295149] VM allocation process starts
[2026-01-12 11:31:35,346][  INFO][PID:3295149] Trying to allocate VM: ResallocHost, ticket_id=775267, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 11:31:38,382][  INFO][PID:3295149] Allocated host ResallocHost, ticket_id=775267, hostname=18.212.193.125, name=aws_x86_64_spot_prod_01138548_20260112_112842, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 11:31:38,383][  INFO][PID:3295149] Allocating ssh connection to builder
[2026-01-12 11:31:38,383][  INFO][PID:3295149] Checking that builder machine is OK
[2026-01-12 11:31:38,698][  INFO][PID:3295149] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2026-01-12 11:31:38,901][  INFO][PID:3295149] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2026-01-12 11:31:38,902][  INFO][PID:3295149] Filling build.info file with builder info
[2026-01-12 11:31:38,903][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:31:38,905][  INFO][PID:3295149] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "packit",
            "project_name": "ansasaki-rust-keylime-master-fedora",
            "project_dirname": "ansasaki-rust-keylime-master-fedora",
            "submitter": "packit",
            "ended_on": null,
            "started_on": 1768217498.9051332,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10000652-fedora-rawhide-x86_64",
            "build_id": 10000652,
            "package_name": "keylime-agent-rust",
            "package_version": "0.2.3-1.20260112113054935981.master.404.ga00a324",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/packit/ansasaki-rust-keylime-master-fedora/keylime-agent-rust",
            "git_hash": "83f39fd78643fe2e6bffeb2bd791cffd5f791011",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "packit/ansasaki-rust-keylime-master-fedora--packit",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/ansasaki-rust-keylime-master-fedora/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/packit/ansasaki-rust-keylime-master-fedora",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/packit/ansasaki-rust-keylime-master-fedora",
            "result_dir": "10000652-keylime-agent-rust",
            "built_packages": "",
            "id": 10000652,
            "mockchain_macros": {
                "copr_username": "packit",
                "copr_projectname": "ansasaki-rust-keylime-master-fedora",
                "vendor": "Fedora Project COPR (packit/ansasaki-rust-keylime-master-fedora)"
            }
        }
    ]
}
[2026-01-12 11:31:38,960][  INFO][PID:3295149] Sending fedora-messaging bus message in build.start
[2026-01-12 11:31:39,969][  INFO][PID:3295149] Sending fedora-messaging bus message in chroot.start
[2026-01-12 11:31:40,011][  INFO][PID:3295149] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10000652-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2026-01-12 11:31:40,348][  INFO][PID:3295149] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '18.212.193.125' (ED25519) to the list of known hosts.


[2026-01-12 11:31:40,352][  INFO][PID:3295149] Downloading the builder-live.log file, attempt 1
[2026-01-12 11:31:40,358][  INFO][PID:3295149] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.193.125 copr-rpmbuild-log
[2026-01-12 11:31:45,360][  INFO][PID:3295149] Periodic builder liveness probe: alive
[2026-01-12 11:31:45,361][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:31:50,362][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:31:55,363][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:32:00,534][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:32:05,535][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:32:10,536][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:32:15,614][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:32:20,618][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:32:25,620][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:32:30,622][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:32:35,855][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:32:40,931][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:32:45,933][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:32:50,936][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:32:56,056][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:33:01,074][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:33:06,077][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:33:11,079][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:33:16,112][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:33:21,129][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:33:26,330][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:33:31,331][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:33:36,332][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:33:41,352][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:33:46,354][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:33:51,355][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:33:56,357][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:01,358][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:06,359][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:11,361][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:16,362][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:21,363][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:26,364][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:31,365][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:36,367][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:41,368][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:46,370][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:51,371][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:56,372][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:34:59,818][ ERROR][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:35:01,374][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:35:06,375][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:35:09,819][WARNING][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:35:11,410][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:35:16,411][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:35:19,820][  INFO][PID:3295149] Downloading the builder-live.log file, attempt 2
[2026-01-12 11:35:19,822][  INFO][PID:3295149] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.193.125 copr-rpmbuild-log
[2026-01-12 11:35:21,412][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:35:26,413][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:35:31,797][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:35:36,799][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:35:41,800][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:35:46,803][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:35:51,804][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:35:56,805][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:36:01,806][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:36:06,807][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:36:11,808][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:36:16,809][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:36:19,889][ ERROR][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:36:21,811][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:36:26,812][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:36:29,894][WARNING][PID:3295149] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:36:31,813][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:36:36,814][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:36:39,895][  INFO][PID:3295149] Downloading the builder-live.log file, attempt 3
[2026-01-12 11:36:39,896][  INFO][PID:3295149] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.193.125 copr-rpmbuild-log
[2026-01-12 11:36:41,956][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:36:46,959][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:36:51,961][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:36:56,963][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:02,134][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:07,139][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:12,141][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:17,142][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:22,143][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:27,144][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:32,340][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:37,342][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:39,966][ ERROR][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:37:42,343][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:47,345][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:49,972][WARNING][PID:3295149] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:37:52,346][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:57,347][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:37:59,973][  INFO][PID:3295149] Downloading the builder-live.log file, attempt 4
[2026-01-12 11:37:59,975][  INFO][PID:3295149] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.212.193.125 copr-rpmbuild-log
[2026-01-12 11:38:02,348][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:38:07,349][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:38:12,350][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:38:17,351][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:38:22,352][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:38:27,353][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:38:32,354][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:38:37,355][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:38:42,356][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:38:47,357][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:38:52,358][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:38:57,359][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:00,042][ ERROR][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:39:02,360][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:07,361][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:10,053][ ERROR][PID:3295149] Giving up for unstable SSH, failures: 1768217709.8195019, 1768217789.8945386, 1768217869.9723823, 1768217950.0535176
[2026-01-12 11:39:10,054][ ERROR][PID:3295149] Re-try request for task on 'ResallocHost, ticket_id=775267, hostname=18.212.193.125, name=aws_x86_64_spot_prod_01138548_20260112_112842, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-01-12 11:39:10,055][  INFO][PID:3295149] Releasing VM back to pool
[2026-01-12 11:39:10,065][  INFO][PID:3295149] Retry #1 (on other host)
[2026-01-12 11:39:10,066][  INFO][PID:3295149] Checking that /var/lib/copr/public_html/results/packit/ansasaki-rust-keylime-master-fedora/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-01-12 11:39:10,066][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:10,067][  INFO][PID:3295149] VM allocation process starts
[2026-01-12 11:39:10,076][  INFO][PID:3295149] Trying to allocate VM: ResallocHost, ticket_id=775367, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 11:39:13,097][  INFO][PID:3295149] Allocated host ResallocHost, ticket_id=775367, hostname=3.236.20.29, name=aws_x86_64_spot_prod_01138773_20260112_113706, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 11:39:13,097][  INFO][PID:3295149] Allocating ssh connection to builder
[2026-01-12 11:39:13,098][  INFO][PID:3295149] Checking that builder machine is OK
[2026-01-12 11:39:13,428][  INFO][PID:3295149] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2026-01-12 11:39:13,634][  INFO][PID:3295149] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2026-01-12 11:39:13,635][  INFO][PID:3295149] Filling build.info file with builder info
[2026-01-12 11:39:13,662][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:13,663][  INFO][PID:3295149] Not re-notifying FE and msg buses for the new host.
[2026-01-12 11:39:13,664][  INFO][PID:3295149] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10000652-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2026-01-12 11:39:13,985][  INFO][PID:3295149] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '3.236.20.29' (ED25519) to the list of known hosts.


[2026-01-12 11:39:13,986][  INFO][PID:3295149] Downloading the builder-live.log file, attempt 1
[2026-01-12 11:39:13,987][  INFO][PID:3295149] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.20.29 copr-rpmbuild-log
[2026-01-12 11:39:18,986][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:23,988][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:28,989][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:33,990][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:38,993][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:43,994][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:48,995][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:53,996][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:39:58,997][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:40:03,998][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:40:08,999][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:40:14,000][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:40:19,001][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:40:24,002][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:40:29,003][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:40:34,004][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:40:39,005][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:40:44,006][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:40:49,007][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:40:54,013][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:40:59,014][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:41:04,015][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:41:09,016][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:41:14,017][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:41:19,018][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:41:24,019][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:41:29,020][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:41:34,021][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:41:39,022][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:41:44,023][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:41:49,031][  INFO][PID:3295149] Periodic builder liveness probe: alive
[2026-01-12 11:41:49,032][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:41:54,032][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:41:59,033][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:42:04,034][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:42:09,036][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:42:14,036][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:42:19,037][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:42:24,039][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:42:29,040][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:42:34,040][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:42:39,041][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:42:44,042][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:42:49,044][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:42:54,046][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:42:59,049][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:43:04,050][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:43:09,051][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:43:10,648][ ERROR][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:43:14,052][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:43:19,053][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:43:20,649][WARNING][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:43:24,055][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:43:29,057][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:43:30,650][  INFO][PID:3295149] Downloading the builder-live.log file, attempt 2
[2026-01-12 11:43:30,653][  INFO][PID:3295149] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.20.29 copr-rpmbuild-log
[2026-01-12 11:43:34,058][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:43:39,059][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:43:44,060][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:43:49,061][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:43:54,063][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:43:59,064][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:44:04,065][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:44:09,066][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:44:14,067][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:44:19,068][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:44:24,069][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:44:29,070][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:44:30,721][ ERROR][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:44:34,071][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:44:39,072][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:44:40,721][WARNING][PID:3295149] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:44:44,073][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:44:49,074][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:44:50,722][  INFO][PID:3295149] Downloading the builder-live.log file, attempt 3
[2026-01-12 11:44:50,723][  INFO][PID:3295149] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.20.29 copr-rpmbuild-log
[2026-01-12 11:44:54,075][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:44:59,076][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:45:04,077][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:45:09,078][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:45:14,079][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:45:19,080][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:45:24,081][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:45:29,082][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:45:34,084][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:45:39,085][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:45:44,086][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:45:49,087][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:45:50,791][ ERROR][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:45:54,088][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:45:59,089][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:46:00,802][WARNING][PID:3295149] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:46:04,090][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:46:09,093][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:46:10,803][  INFO][PID:3295149] Downloading the builder-live.log file, attempt 4
[2026-01-12 11:46:10,804][  INFO][PID:3295149] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.236.20.29 copr-rpmbuild-log
[2026-01-12 11:46:14,094][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:46:19,095][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:46:24,097][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:46:29,098][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:46:34,099][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:46:39,100][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:46:44,104][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:46:49,105][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:46:54,106][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:46:59,107][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:04,109][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:09,110][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:10,870][ ERROR][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:47:14,111][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:19,112][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:20,901][ ERROR][PID:3295149] Giving up for unstable SSH, failures: 1768218200.649642, 1768218280.7218437, 1768218360.8025541, 1768218440.9011126
[2026-01-12 11:47:20,902][ ERROR][PID:3295149] Re-try request for task on 'ResallocHost, ticket_id=775367, hostname=3.236.20.29, name=aws_x86_64_spot_prod_01138773_20260112_113706, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-01-12 11:47:20,902][  INFO][PID:3295149] Releasing VM back to pool
[2026-01-12 11:47:20,982][  INFO][PID:3295149] Retry #2 (on other host)
[2026-01-12 11:47:20,983][  INFO][PID:3295149] Checking that /var/lib/copr/public_html/results/packit/ansasaki-rust-keylime-master-fedora/fedora-rawhide-x86_64/repodata/repomd.xml exists
[2026-01-12 11:47:20,984][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:20,985][  INFO][PID:3295149] VM allocation process starts
[2026-01-12 11:47:21,025][  INFO][PID:3295149] Trying to allocate VM: ResallocHost, ticket_id=775452, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 11:47:24,048][  INFO][PID:3295149] Allocated host ResallocHost, ticket_id=775452, hostname=52.91.98.123, name=aws_x86_64_spot_prod_01138958_20260112_114508, requested_tags=['copr_builder', 'arch_x86_64']
[2026-01-12 11:47:24,048][  INFO][PID:3295149] Allocating ssh connection to builder
[2026-01-12 11:47:24,049][  INFO][PID:3295149] Checking that builder machine is OK
[2026-01-12 11:47:24,367][  INFO][PID:3295149] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2026-01-12 11:47:24,570][  INFO][PID:3295149] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2026-01-12 11:47:24,570][  INFO][PID:3295149] Filling build.info file with builder info
[2026-01-12 11:47:24,577][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:24,578][  INFO][PID:3295149] Not re-notifying FE and msg buses for the new host.
[2026-01-12 11:47:24,578][  INFO][PID:3295149] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10000652-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2026-01-12 11:47:24,911][  INFO][PID:3295149] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '52.91.98.123' (ED25519) to the list of known hosts.


[2026-01-12 11:47:24,912][  INFO][PID:3295149] Downloading the builder-live.log file, attempt 1
[2026-01-12 11:47:24,913][  INFO][PID:3295149] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.91.98.123 copr-rpmbuild-log
[2026-01-12 11:47:29,912][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:34,914][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:39,916][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:44,917][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:49,918][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:54,920][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:47:59,921][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:48:04,922][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:48:09,923][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:48:14,924][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:48:19,925][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:48:24,926][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:48:29,927][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:48:34,928][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:48:39,929][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:48:44,931][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:48:49,931][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:48:54,933][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:48:59,934][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:49:04,202][ ERROR][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:49:04,936][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:49:09,937][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:49:14,203][WARNING][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:49:14,938][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:49:19,939][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:49:24,204][  INFO][PID:3295149] Downloading the builder-live.log file, attempt 2
[2026-01-12 11:49:24,205][  INFO][PID:3295149] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.91.98.123 copr-rpmbuild-log
[2026-01-12 11:49:24,940][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:49:29,941][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:49:34,942][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:49:39,943][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:49:44,943][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:49:49,944][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:49:54,945][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:49:59,946][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:50:04,947][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:50:09,948][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:50:14,949][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:50:19,953][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:50:24,277][ ERROR][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:50:24,954][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:50:29,956][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:50:34,285][WARNING][PID:3295149] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:50:34,957][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:50:39,958][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:50:44,286][  INFO][PID:3295149] Downloading the builder-live.log file, attempt 3
[2026-01-12 11:50:44,288][  INFO][PID:3295149] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.91.98.123 copr-rpmbuild-log
[2026-01-12 11:50:44,959][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:50:49,961][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:50:54,962][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:50:59,963][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:51:04,964][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:51:09,965][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:51:14,966][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:51:19,967][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:51:24,970][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:51:29,983][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:51:34,992][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:51:39,997][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:51:44,365][ ERROR][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:51:44,998][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:51:50,010][  INFO][PID:3295149] Periodic builder liveness probe: alive
[2026-01-12 11:51:50,011][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:51:54,366][WARNING][PID:3295149] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-12 11:51:55,012][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:52:00,023][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:52:04,367][  INFO][PID:3295149] Downloading the builder-live.log file, attempt 4
[2026-01-12 11:52:04,368][  INFO][PID:3295149] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@52.91.98.123 copr-rpmbuild-log
[2026-01-12 11:52:05,024][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:52:10,025][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:52:15,026][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:52:20,027][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:52:25,028][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:52:30,030][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:52:35,031][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:52:40,032][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:52:45,033][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:52:50,034][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:52:55,035][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:53:00,037][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:53:04,434][ ERROR][PID:3295149] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-12 11:53:05,038][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:53:10,039][  INFO][PID:3295149] Checking for cancel request
[2026-01-12 11:53:14,442][ ERROR][PID:3295149] Giving up for unstable SSH, failures: 1768218554.2033963, 1768218634.2855308, 1768218714.366354, 1768218794.4425862
[2026-01-12 11:53:14,443][ ERROR][PID:3295149] Re-try request for task on 'ResallocHost, ticket_id=775452, hostname=52.91.98.123, name=aws_x86_64_spot_prod_01138958_20260112_114508, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-01-12 11:53:14,444][  INFO][PID:3295149] Releasing VM back to pool
[2026-01-12 11:53:14,484][ ERROR][PID:3295149] Backend process error: Three host tried without success: {'18.212.193.125', '52.91.98.123', '3.236.20.29'}
[2026-01-12 11:53:14,485][WARNING][PID:3295149] Switching not-finished job state to 'failed'
[2026-01-12 11:53:14,485][  INFO][PID:3295149] Worker failed build, took 349.90674901008606
[2026-01-12 11:53:14,485][  INFO][PID:3295149] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "packit",
            "project_name": "ansasaki-rust-keylime-master-fedora",
            "project_dirname": "ansasaki-rust-keylime-master-fedora",
            "submitter": "packit",
            "ended_on": 1768218794.4851313,
            "started_on": 1768218444.5783823,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10000652-fedora-rawhide-x86_64",
            "build_id": 10000652,
            "package_name": "keylime-agent-rust",
            "package_version": "0.2.3-1.20260112113054935981.master.404.ga00a324",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/packit/ansasaki-rust-keylime-master-fedora/keylime-agent-rust",
            "git_hash": "83f39fd78643fe2e6bffeb2bd791cffd5f791011",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "packit/ansasaki-rust-keylime-master-fedora--packit",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/ansasaki-rust-keylime-master-fedora/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/packit/ansasaki-rust-keylime-master-fedora",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/packit/ansasaki-rust-keylime-master-fedora",
            "result_dir": "10000652-keylime-agent-rust",
            "built_packages": "",
            "id": 10000652,
            "mockchain_macros": {
                "copr_username": "packit",
                "copr_projectname": "ansasaki-rust-keylime-master-fedora",
                "vendor": "Fedora Project COPR (packit/ansasaki-rust-keylime-master-fedora)"
            }
        }
    ]
}
[2026-01-12 11:53:14,532][  INFO][PID:3295149] Sending fedora-messaging bus message in build.end
[2026-01-12 11:53:14,598][  INFO][PID:3295149] Compressing /var/lib/copr/public_html/results/packit/ansasaki-rust-keylime-master-fedora/fedora-rawhide-x86_64/10000652-keylime-agent-rust/builder-live.log by gzip
[2026-01-12 11:53:14,599][  INFO][PID:3295149] Running command 'gzip /var/lib/copr/public_html/results/packit/ansasaki-rust-keylime-master-fedora/fedora-rawhide-x86_64/10000652-keylime-agent-rust/builder-live.log' as PID 3418719
[2026-01-12 11:53:14,600][  INFO][PID:3295149] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/packit/ansasaki-rust-keylime-master-fedora/fedora-rawhide-x86_64/10000652-keylime-agent-rust/builder-live.log)
[2026-01-12 11:53:14,601][  INFO][PID:3295149] Compressing /var/lib/copr/public_html/results/packit/ansasaki-rust-keylime-master-fedora/fedora-rawhide-x86_64/10000652-keylime-agent-rust/backend.log by gzip
[2026-01-12 11:53:14,602][  INFO][PID:3295149] Running command 'gzip /var/lib/copr/public_html/results/packit/ansasaki-rust-keylime-master-fedora/fedora-rawhide-x86_64/10000652-keylime-agent-rust/backend.log' as PID 3418720