[2026-02-06 17:19:52,182][  INFO][PID:4063099] Marking build as starting
[2026-02-06 17:19:52,250][  INFO][PID:4063099] Checking that /var/lib/copr/public_html/results/scottames/vicinae/fedora-43-x86_64/repodata/repomd.xml exists
[2026-02-06 17:19:52,252][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:19:52,253][  INFO][PID:4063099] VM allocation process starts
[2026-02-06 17:19:52,360][  INFO][PID:4063099] Trying to allocate VM: ResallocHost, ticket_id=1203968, requested_tags=['copr_builder', 'arch_x86_64']
[2026-02-06 17:19:55,387][  INFO][PID:4063099] Allocated host ResallocHost, ticket_id=1203968, hostname=44.211.117.170, name=aws_x86_64_spot_prod_01984963_20260206_171356, requested_tags=['copr_builder', 'arch_x86_64']
[2026-02-06 17:19:55,387][  INFO][PID:4063099] Allocating ssh connection to builder
[2026-02-06 17:19:55,388][  INFO][PID:4063099] Checking that builder machine is OK
[2026-02-06 17:19:55,720][  INFO][PID:4063099] Running remote command: copr-builder-ready fedora-43-x86_64
[2026-02-06 17:19:55,959][  INFO][PID:4063099] Red Hat subscription not needed for fedora-43-x86_64
Builder is ready to be used

[2026-02-06 17:19:55,959][  INFO][PID:4063099] Filling build.info file with builder info
[2026-02-06 17:19:55,960][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:19:55,961][  INFO][PID:4063099] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "scottames",
            "project_name": "vicinae",
            "project_dirname": "vicinae",
            "submitter": "scottames",
            "ended_on": null,
            "started_on": 1770398395.9613757,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-43-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10100589-fedora-43-x86_64",
            "build_id": 10100589,
            "package_name": "vicinae",
            "package_version": "1:0.19.6-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/scottames/vicinae/vicinae",
            "git_hash": "3f1e0f11c97ecdfe35cb51f3eebb88a2d3d35313",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "scottames/vicinae--scottames",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/scottames/vicinae/fedora-43-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/scottames/vicinae/fedora-43-x86_64/",
                    "id": "copr_scottames_vicinae",
                    "name": "Additional repo copr_scottames_vicinae"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/scottames/vicinae",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/scottames/vicinae",
            "result_dir": "10100589-vicinae",
            "built_packages": "",
            "id": 10100589,
            "mockchain_macros": {
                "copr_username": "scottames",
                "copr_projectname": "vicinae",
                "vendor": "Fedora Project COPR (scottames/vicinae)"
            }
        }
    ]
}
[2026-02-06 17:19:56,021][  INFO][PID:4063099] Sending fedora-messaging bus message in build.start
[2026-02-06 17:19:56,647][  INFO][PID:4063099] Sending fedora-messaging bus message in chroot.start
[2026-02-06 17:19:56,675][  INFO][PID:4063099] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10100589-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached
[2026-02-06 17:19:57,012][  INFO][PID:4063099] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '44.211.117.170' (ED25519) to the list of known hosts.


[2026-02-06 17:19:57,013][  INFO][PID:4063099] Downloading the builder-live.log file, attempt 1
[2026-02-06 17:19:57,014][  INFO][PID:4063099] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.211.117.170 copr-rpmbuild-log
[2026-02-06 17:20:02,023][  INFO][PID:4063099] Periodic builder liveness probe: alive
[2026-02-06 17:20:02,024][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:20:07,025][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:20:12,026][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:20:17,036][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:20:22,037][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:20:27,038][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:20:32,039][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:20:33,606][ ERROR][PID:4063099] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-06 17:20:37,040][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:20:42,041][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:20:43,607][WARNING][PID:4063099] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-06 17:20:47,042][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:20:52,044][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:20:53,607][  INFO][PID:4063099] Downloading the builder-live.log file, attempt 2
[2026-02-06 17:20:53,609][  INFO][PID:4063099] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.211.117.170 copr-rpmbuild-log
[2026-02-06 17:20:57,045][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:21:02,046][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:21:07,047][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:21:12,048][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:21:17,052][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:21:22,054][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:21:27,055][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:21:32,057][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:21:37,058][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:21:42,059][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:21:47,060][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:21:52,061][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:21:53,675][ ERROR][PID:4063099] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-06 17:21:57,062][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:22:02,063][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:22:03,685][WARNING][PID:4063099] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-06 17:22:07,064][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:22:12,065][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:22:13,686][  INFO][PID:4063099] Downloading the builder-live.log file, attempt 3
[2026-02-06 17:22:13,687][  INFO][PID:4063099] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.211.117.170 copr-rpmbuild-log
[2026-02-06 17:22:17,067][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:22:22,069][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:22:27,073][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:22:32,074][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:22:37,075][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:22:42,076][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:22:47,077][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:22:52,078][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:22:57,079][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:23:02,081][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:23:07,094][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:23:12,095][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:23:13,760][ ERROR][PID:4063099] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-06 17:23:17,096][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:23:22,097][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:23:23,769][WARNING][PID:4063099] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-06 17:23:27,098][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:23:32,102][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:23:33,770][  INFO][PID:4063099] Downloading the builder-live.log file, attempt 4
[2026-02-06 17:23:33,771][  INFO][PID:4063099] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.211.117.170 copr-rpmbuild-log
[2026-02-06 17:23:37,103][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:23:42,104][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:23:47,105][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:23:52,106][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:23:57,107][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:02,108][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:07,109][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:12,110][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:17,111][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:22,112][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:27,113][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:32,114][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:33,838][ ERROR][PID:4063099] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-06 17:24:37,115][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:42,116][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:43,907][ ERROR][PID:4063099] Giving up for unstable SSH, failures: 1770398443.6069968, 1770398523.6852612, 1770398603.7692964, 1770398683.907414
[2026-02-06 17:24:43,908][ ERROR][PID:4063099] Re-try request for task on 'ResallocHost, ticket_id=1203968, hostname=44.211.117.170, name=aws_x86_64_spot_prod_01984963_20260206_171356, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-02-06 17:24:43,909][  INFO][PID:4063099] Releasing VM back to pool
[2026-02-06 17:24:43,972][  INFO][PID:4063099] Retry #1 (on other host)
[2026-02-06 17:24:43,973][  INFO][PID:4063099] Checking that /var/lib/copr/public_html/results/scottames/vicinae/fedora-43-x86_64/repodata/repomd.xml exists
[2026-02-06 17:24:43,973][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:43,974][  INFO][PID:4063099] VM allocation process starts
[2026-02-06 17:24:43,996][  INFO][PID:4063099] Trying to allocate VM: ResallocHost, ticket_id=1204018, requested_tags=['copr_builder', 'arch_x86_64']
[2026-02-06 17:24:47,009][  INFO][PID:4063099] Allocated host ResallocHost, ticket_id=1204018, hostname=54.224.31.169, name=aws_x86_64_spot_prod_01984971_20260206_171402, requested_tags=['copr_builder', 'arch_x86_64']
[2026-02-06 17:24:47,009][  INFO][PID:4063099] Allocating ssh connection to builder
[2026-02-06 17:24:47,010][  INFO][PID:4063099] Checking that builder machine is OK
[2026-02-06 17:24:47,333][  INFO][PID:4063099] Running remote command: copr-builder-ready fedora-43-x86_64
[2026-02-06 17:24:47,572][  INFO][PID:4063099] Red Hat subscription not needed for fedora-43-x86_64
Builder is ready to be used

[2026-02-06 17:24:47,573][  INFO][PID:4063099] Filling build.info file with builder info
[2026-02-06 17:24:47,577][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:47,578][  INFO][PID:4063099] Not re-notifying FE and msg buses for the new host.
[2026-02-06 17:24:47,578][  INFO][PID:4063099] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10100589-fedora-43-x86_64 --chroot fedora-43-x86_64 --detached
[2026-02-06 17:24:47,915][  INFO][PID:4063099] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '54.224.31.169' (ED25519) to the list of known hosts.


[2026-02-06 17:24:47,915][  INFO][PID:4063099] Downloading the builder-live.log file, attempt 1
[2026-02-06 17:24:47,917][  INFO][PID:4063099] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.224.31.169 copr-rpmbuild-log
[2026-02-06 17:24:52,916][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:24:57,918][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:25:02,919][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:25:07,920][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:25:12,921][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:25:17,922][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:25:22,923][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:25:27,924][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:25:32,925][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:25:37,926][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:25:42,927][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:25:47,928][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:25:52,929][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:25:57,930][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:26:02,932][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:26:07,933][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:26:12,934][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:26:17,935][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:26:22,936][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:26:27,937][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:26:32,940][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:26:37,941][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:26:42,942][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:26:47,943][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:26:52,944][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:26:57,945][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:27:02,946][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:27:07,947][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:27:12,948][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:27:17,949][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:27:22,950][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:27:27,951][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:27:32,952][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:27:37,953][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:27:42,953][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:27:47,954][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:27:52,955][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:27:57,956][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:28:02,957][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:28:07,958][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:28:12,959][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:28:17,960][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:28:22,961][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:28:27,962][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:28:32,963][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:28:37,964][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:28:42,965][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:28:47,966][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:28:52,967][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:28:57,968][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:29:02,969][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:29:07,970][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:29:12,971][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:29:17,972][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:29:22,973][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:29:27,977][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:29:32,978][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:29:37,979][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:29:42,980][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:29:47,981][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:29:52,982][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:29:57,983][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:30:02,991][  INFO][PID:4063099] Periodic builder liveness probe: alive
[2026-02-06 17:30:02,992][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:30:07,992][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:30:12,994][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:30:18,000][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:30:23,001][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:30:28,002][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:30:33,003][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:30:38,004][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:30:43,005][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:30:48,006][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:30:53,006][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:30:58,007][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:31:03,009][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:31:08,010][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:31:13,011][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:31:18,012][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:31:23,013][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:31:28,014][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:31:33,015][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:31:38,016][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:31:43,017][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:31:48,018][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:31:53,019][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:31:53,458][ ERROR][PID:4063099] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-06 17:31:58,020][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:32:03,035][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:32:03,459][WARNING][PID:4063099] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-06 17:32:08,036][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:32:13,037][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:32:13,459][  INFO][PID:4063099] Downloading the builder-live.log file, attempt 2
[2026-02-06 17:32:13,461][  INFO][PID:4063099] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.224.31.169 copr-rpmbuild-log
[2026-02-06 17:32:18,038][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:32:23,039][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:32:28,041][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:32:33,042][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:32:38,044][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:32:43,045][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:32:48,046][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:32:53,054][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:32:58,063][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:33:03,080][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:33:08,085][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:33:13,086][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:33:13,527][ ERROR][PID:4063099] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-06 17:33:18,087][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:33:23,088][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:33:23,546][WARNING][PID:4063099] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-06 17:33:28,089][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:33:33,091][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:33:33,547][  INFO][PID:4063099] Downloading the builder-live.log file, attempt 3
[2026-02-06 17:33:33,548][  INFO][PID:4063099] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.224.31.169 copr-rpmbuild-log
[2026-02-06 17:33:38,092][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:33:43,126][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:33:48,127][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:33:53,128][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:33:58,183][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:34:03,184][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:34:08,185][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:34:13,186][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:34:18,187][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:34:23,188][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:34:28,189][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:34:33,196][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:34:33,601][ ERROR][PID:4063099] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-06 17:34:38,197][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:34:43,198][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:34:43,607][WARNING][PID:4063099] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-02-06 17:34:48,199][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:34:53,200][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:34:53,608][  INFO][PID:4063099] Downloading the builder-live.log file, attempt 4
[2026-02-06 17:34:53,609][  INFO][PID:4063099] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.224.31.169 copr-rpmbuild-log
[2026-02-06 17:34:58,202][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:35:03,203][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:35:08,207][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:35:13,208][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:35:18,209][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:35:23,210][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:35:28,211][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:35:33,212][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:35:38,234][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:35:43,235][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:35:48,236][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:35:53,237][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:35:53,652][ ERROR][PID:4063099] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-02-06 17:35:58,238][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:36:03,239][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:36:03,661][ ERROR][PID:4063099] Giving up for unstable SSH, failures: 1770399123.4590874, 1770399203.5463307, 1770399283.6073186, 1770399363.661798
[2026-02-06 17:36:03,662][ ERROR][PID:4063099] Re-try request for task on 'ResallocHost, ticket_id=1204018, hostname=54.224.31.169, name=aws_x86_64_spot_prod_01984971_20260206_171402, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-02-06 17:36:03,663][  INFO][PID:4063099] Releasing VM back to pool
[2026-02-06 17:36:03,672][  INFO][PID:4063099] Retry #2 (on other host)
[2026-02-06 17:36:03,673][  INFO][PID:4063099] Checking that /var/lib/copr/public_html/results/scottames/vicinae/fedora-43-x86_64/repodata/repomd.xml exists
[2026-02-06 17:36:03,673][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:36:03,674][  INFO][PID:4063099] VM allocation process starts
[2026-02-06 17:36:03,686][  INFO][PID:4063099] Trying to allocate VM: ResallocHost, ticket_id=1204096, requested_tags=['copr_builder', 'arch_x86_64']
[2026-02-06 17:36:08,687][  INFO][PID:4063099] Checking for cancel request
[2026-02-06 17:36:09,713][  INFO][PID:4063099] Allocated host ResallocHost, ticket_id=1204096, hostname=13.217.35.71, name=aws_x86_64_spot_prod_01985026_20260206_171649, requested_tags=['copr_builder', 'arch_x86_64']
[2026-02-06 17:36:09,713][  INFO][PID:4063099] Allocating ssh connection to builder
[2026-02-06 17:36:09,714][  INFO][PID:4063099] Checking that builder machine is OK
[2026-02-06 17:37:09,783][ ERROR][PID:4063099] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 13.217.35.71 port 22: Connection timed out

[2026-02-06 17:37:19,783][ ERROR][PID:4063099] Re-try request for task on 'ResallocHost, ticket_id=1204096, hostname=13.217.35.71, name=aws_x86_64_spot_prod_01985026_20260206_171649, requested_tags=['copr_builder', 'arch_x86_64']': Unable to finish after 1 SSH attempts
[2026-02-06 17:37:19,784][  INFO][PID:4063099] Releasing VM back to pool
[2026-02-06 17:37:19,794][ ERROR][PID:4063099] Backend process error: Three host tried without success: {'13.217.35.71', '54.224.31.169', '44.211.117.170'}
[2026-02-06 17:37:19,794][WARNING][PID:4063099] Switching not-finished job state to 'failed'
[2026-02-06 17:37:19,795][  INFO][PID:4063099] Worker failed build, took 752.2161595821381
[2026-02-06 17:37:19,795][  INFO][PID:4063099] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "scottames",
            "project_name": "vicinae",
            "project_dirname": "vicinae",
            "submitter": "scottames",
            "ended_on": 1770399439.7945817,
            "started_on": 1770398687.578422,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-43-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10100589-fedora-43-x86_64",
            "build_id": 10100589,
            "package_name": "vicinae",
            "package_version": "1:0.19.6-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/scottames/vicinae/vicinae",
            "git_hash": "3f1e0f11c97ecdfe35cb51f3eebb88a2d3d35313",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "scottames/vicinae--scottames",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/scottames/vicinae/fedora-43-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/scottames/vicinae/fedora-43-x86_64/",
                    "id": "copr_scottames_vicinae",
                    "name": "Additional repo copr_scottames_vicinae"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/scottames/vicinae",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/scottames/vicinae",
            "result_dir": "10100589-vicinae",
            "built_packages": "",
            "id": 10100589,
            "mockchain_macros": {
                "copr_username": "scottames",
                "copr_projectname": "vicinae",
                "vendor": "Fedora Project COPR (scottames/vicinae)"
            }
        }
    ]
}
[2026-02-06 17:37:19,847][  INFO][PID:4063099] Sending fedora-messaging bus message in build.end
[2026-02-06 17:37:19,891][  INFO][PID:4063099] Compressing /var/lib/copr/public_html/results/scottames/vicinae/fedora-43-x86_64/10100589-vicinae/builder-live.log by gzip
[2026-02-06 17:37:19,893][  INFO][PID:4063099] Running command 'gzip /var/lib/copr/public_html/results/scottames/vicinae/fedora-43-x86_64/10100589-vicinae/builder-live.log' as PID 4132718
[2026-02-06 17:37:20,026][  INFO][PID:4063099] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/scottames/vicinae/fedora-43-x86_64/10100589-vicinae/builder-live.log)
[2026-02-06 17:37:20,027][  INFO][PID:4063099] Compressing /var/lib/copr/public_html/results/scottames/vicinae/fedora-43-x86_64/10100589-vicinae/backend.log by gzip
[2026-02-06 17:37:20,028][  INFO][PID:4063099] Running command 'gzip /var/lib/copr/public_html/results/scottames/vicinae/fedora-43-x86_64/10100589-vicinae/backend.log' as PID 4132736