[2025-09-11 17:23:27,233][  INFO][PID:269433] Marking build as starting
[2025-09-11 17:23:27,274][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:23:27,275][  INFO][PID:269433] VM allocation process starts
[2025-09-11 17:23:27,361][  INFO][PID:269433] Trying to allocate VM: ResallocHost, ticket_id=5011897, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 17:23:30,466][  INFO][PID:269433] Allocated host ResallocHost, ticket_id=5011897, hostname=54.198.112.99, name=aws_x86_64_spot_prod_06719870_20250911_171559, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 17:23:30,466][  INFO][PID:269433] Allocating ssh connection to builder
[2025-09-11 17:23:30,467][  INFO][PID:269433] Checking that builder machine is OK
[2025-09-11 17:23:30,886][  INFO][PID:269433] Running remote command: copr-builder-ready epel-9-x86_64
[2025-09-11 17:23:31,088][  INFO][PID:269433] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2025-09-11 17:23:31,089][  INFO][PID:269433] Filling build.info file with builder info
[2025-09-11 17:23:31,089][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:23:31,090][  INFO][PID:269433] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "bieszczaders",
            "project_name": "kernel-cachyos",
            "project_dirname": "kernel-cachyos",
            "submitter": "trixieua",
            "ended_on": null,
            "started_on": 1757611411.0904546,
            "submitted_on": null,
            "status": 3,
            "chroot": "epel-9-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9546924-epel-9-x86_64",
            "build_id": 9546924,
            "package_name": "kernel-cachyos-server",
            "package_version": "6.16.6-cachyserver1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/bieszczaders/kernel-cachyos/kernel-cachyos-server",
            "git_hash": "2ed9b600ea08f77d8e726f301f668d51c2b53f2c",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "bieszczaders/kernel-cachyos--trixieua",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/bieszczaders/kernel-cachyos/epel-9-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/bieszczaders/kernel-cachyos",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/bieszczaders/kernel-cachyos",
            "result_dir": "09546924-kernel-cachyos-server",
            "built_packages": "",
            "id": 9546924,
            "mockchain_macros": {
                "copr_username": "bieszczaders",
                "copr_projectname": "kernel-cachyos",
                "vendor": "Fedora Project COPR (bieszczaders/kernel-cachyos)"
            }
        }
    ]
}
[2025-09-11 17:23:31,120][  INFO][PID:269433] Sending fedora-messaging bus message in build.start
[2025-09-11 17:23:32,008][  INFO][PID:269433] Sending fedora-messaging bus message in chroot.start
[2025-09-11 17:23:32,038][  INFO][PID:269433] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9546924-epel-9-x86_64 --chroot epel-9-x86_64 --detached
[2025-09-11 17:23:32,371][  INFO][PID:269433] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '54.198.112.99' (ED25519) to the list of known hosts.


[2025-09-11 17:23:32,373][  INFO][PID:269433] Downloading the builder-live.log file, attempt 1
[2025-09-11 17:23:32,374][  INFO][PID:269433] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.198.112.99 copr-rpmbuild-log
[2025-09-11 17:23:37,382][  INFO][PID:269433] Periodic builder liveness probe: alive
[2025-09-11 17:23:37,382][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:23:42,383][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:23:47,384][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:23:52,385][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:23:57,387][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:24:02,388][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:24:07,389][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:24:12,390][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:24:17,391][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:24:22,392][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:24:27,393][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:24:32,395][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:24:37,396][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:24:42,397][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:24:47,398][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:24:52,400][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:24:57,403][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:25:02,404][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:25:07,406][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:25:12,407][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:25:17,409][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:25:22,410][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:25:27,411][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:25:32,412][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:25:37,414][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:25:42,415][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:25:47,416][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:25:52,417][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:25:57,418][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:26:02,419][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:26:07,421][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:26:12,422][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:26:17,423][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:26:22,425][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:26:27,426][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:26:32,427][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:26:37,428][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:26:42,430][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:26:47,431][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:26:52,432][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:26:57,433][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:27:02,434][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:27:07,435][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:27:12,436][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:27:17,437][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:27:22,439][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:27:27,440][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:27:32,441][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:27:37,442][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:27:42,443][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:27:47,444][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:27:52,445][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:27:57,446][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:28:02,446][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:28:07,448][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:28:12,449][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:28:17,450][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:28:22,451][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:28:27,452][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:28:32,453][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:28:37,454][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:28:42,455][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:28:47,456][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:28:52,457][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:28:57,458][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:29:02,459][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:29:07,460][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:29:12,461][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:29:17,463][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:29:22,464][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:29:27,465][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:29:32,466][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:29:37,467][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:29:42,468][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:29:47,470][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:29:52,472][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:29:57,473][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:30:02,474][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:30:07,475][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:30:12,476][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:30:17,477][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:30:22,478][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:30:27,480][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:30:32,481][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:30:37,483][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:30:42,484][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:30:47,485][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:30:52,487][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:30:57,488][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:31:02,489][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:31:07,494][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:31:12,496][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:31:17,497][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:31:22,498][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:31:27,499][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:31:32,501][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:31:37,504][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:31:42,505][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:31:47,507][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:31:52,517][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:31:57,522][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:32:02,524][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:32:07,525][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:32:12,527][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:32:17,529][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:32:22,530][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:32:27,532][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:32:32,535][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:32:37,539][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:32:42,542][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:32:47,543][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:32:52,546][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:32:57,551][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:33:02,552][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:33:07,561][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:33:12,562][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:33:17,564][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:33:22,565][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:33:27,568][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:33:32,574][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:33:37,589][  INFO][PID:269433] Periodic builder liveness probe: alive
[2025-09-11 17:33:37,591][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:33:42,598][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:33:47,600][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:33:52,601][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:33:57,616][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:34:02,619][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:34:07,620][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:34:12,623][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:34:17,625][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:34:19,921][ ERROR][PID:269433] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 17:34:22,626][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:34:27,628][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:34:29,922][WARNING][PID:269433] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 17:34:32,631][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:34:37,637][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:34:39,923][  INFO][PID:269433] Downloading the builder-live.log file, attempt 2
[2025-09-11 17:34:39,925][  INFO][PID:269433] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.198.112.99 copr-rpmbuild-log
[2025-09-11 17:34:42,638][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:34:47,640][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:34:52,642][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:34:57,643][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:35:02,649][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:35:07,662][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:35:12,664][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:35:17,668][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:35:22,669][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:35:27,670][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:35:32,671][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:35:37,672][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:35:40,003][ ERROR][PID:269433] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 17:35:42,674][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:35:47,676][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:35:50,005][WARNING][PID:269433] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 17:35:52,677][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:35:57,678][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:36:00,006][  INFO][PID:269433] Downloading the builder-live.log file, attempt 3
[2025-09-11 17:36:00,008][  INFO][PID:269433] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.198.112.99 copr-rpmbuild-log
[2025-09-11 17:36:02,683][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:36:07,685][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:36:12,686][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:36:17,687][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:36:22,688][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:36:27,689][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:36:32,690][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:36:37,691][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:36:42,692][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:36:47,693][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:36:52,694][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:36:57,696][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:37:00,077][ ERROR][PID:269433] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 17:37:02,701][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:37:07,702][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:37:10,078][WARNING][PID:269433] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 17:37:12,704][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:37:17,707][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:37:20,079][  INFO][PID:269433] Downloading the builder-live.log file, attempt 4
[2025-09-11 17:37:20,081][  INFO][PID:269433] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.198.112.99 copr-rpmbuild-log
[2025-09-11 17:37:22,708][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:37:27,710][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:37:32,718][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:37:37,721][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:37:42,726][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:37:47,727][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:37:52,729][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:37:57,730][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:02,732][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:07,733][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:12,735][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:17,736][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:20,150][ ERROR][PID:269433] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 17:38:22,738][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:27,741][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:30,151][ ERROR][PID:269433] Giving up for unstable SSH, failures: 1757612069.9227157, 1757612150.0052297, 1757612230.078557, 1757612310.1512933
[2025-09-11 17:38:30,152][ ERROR][PID:269433] Re-try request for task on 'ResallocHost, ticket_id=5011897, hostname=54.198.112.99, name=aws_x86_64_spot_prod_06719870_20250911_171559, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-09-11 17:38:30,153][  INFO][PID:269433] Releasing VM back to pool
[2025-09-11 17:38:30,163][  INFO][PID:269433] Retry #1 (on other host)
[2025-09-11 17:38:30,163][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:30,164][  INFO][PID:269433] VM allocation process starts
[2025-09-11 17:38:30,180][  INFO][PID:269433] Trying to allocate VM: ResallocHost, ticket_id=5012275, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 17:38:33,202][  INFO][PID:269433] Allocated host ResallocHost, ticket_id=5012275, hostname=44.220.252.172, name=aws_x86_64_spot_prod_06720393_20250911_173534, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 17:38:33,203][  INFO][PID:269433] Allocating ssh connection to builder
[2025-09-11 17:38:33,204][  INFO][PID:269433] Checking that builder machine is OK
[2025-09-11 17:38:33,670][  INFO][PID:269433] Running remote command: copr-builder-ready epel-9-x86_64
[2025-09-11 17:38:33,881][  INFO][PID:269433] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2025-09-11 17:38:33,882][  INFO][PID:269433] Filling build.info file with builder info
[2025-09-11 17:38:33,886][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:33,888][  INFO][PID:269433] Not re-notifying FE and msg buses for the new host.
[2025-09-11 17:38:33,889][  INFO][PID:269433] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9546924-epel-9-x86_64 --chroot epel-9-x86_64 --detached
[2025-09-11 17:38:34,221][  INFO][PID:269433] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '44.220.252.172' (ED25519) to the list of known hosts.


[2025-09-11 17:38:34,222][  INFO][PID:269433] Downloading the builder-live.log file, attempt 1
[2025-09-11 17:38:34,224][  INFO][PID:269433] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.220.252.172 copr-rpmbuild-log
[2025-09-11 17:38:39,223][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:44,224][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:49,225][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:54,226][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:38:59,228][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:39:04,229][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:39:09,232][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:39:14,238][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:39:19,246][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:39:24,256][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:39:29,264][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:39:34,266][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:39:39,269][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:39:44,272][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:39:49,277][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:39:54,279][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:39:59,280][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:40:04,285][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:40:09,288][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:40:14,289][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:40:19,296][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:40:24,297][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:40:29,298][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:40:34,299][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:40:39,301][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:40:44,302][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:40:49,303][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:40:54,305][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:40:59,306][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:41:04,310][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:41:09,313][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:41:14,314][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:41:19,315][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:41:24,316][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:41:29,320][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:41:34,321][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:41:39,327][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:41:44,330][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:41:49,333][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:41:54,338][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:41:59,340][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:42:04,343][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:42:09,345][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:42:14,346][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:42:19,347][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:42:24,352][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:42:29,353][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:42:34,357][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:42:39,358][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:42:44,359][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:42:49,360][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:42:54,361][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:42:59,362][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:43:04,363][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:43:09,366][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:43:14,367][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:43:19,368][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:43:24,369][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:43:29,371][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:43:34,373][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:43:39,386][  INFO][PID:269433] Periodic builder liveness probe: alive
[2025-09-11 17:43:39,387][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:43:44,388][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:43:49,391][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:43:54,392][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:43:59,393][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:44:04,394][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:44:09,396][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:44:14,397][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:44:16,970][ ERROR][PID:269433] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 17:44:19,398][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:44:24,399][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:44:26,971][WARNING][PID:269433] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 17:44:29,401][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:44:34,402][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:44:36,972][  INFO][PID:269433] Downloading the builder-live.log file, attempt 2
[2025-09-11 17:44:36,974][  INFO][PID:269433] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.220.252.172 copr-rpmbuild-log
[2025-09-11 17:44:39,404][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:44:44,408][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:44:49,409][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:44:54,410][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:44:59,411][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:45:04,412][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:45:09,413][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:45:14,414][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:45:19,416][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:45:24,417][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:45:29,418][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:45:34,419][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:45:37,043][ ERROR][PID:269433] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 17:45:39,420][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:45:44,421][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:45:47,044][WARNING][PID:269433] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 17:45:49,422][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:45:54,423][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:45:57,045][  INFO][PID:269433] Downloading the builder-live.log file, attempt 3
[2025-09-11 17:45:57,046][  INFO][PID:269433] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.220.252.172 copr-rpmbuild-log
[2025-09-11 17:45:59,424][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:46:04,426][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:46:09,427][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:46:14,431][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:46:19,432][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:46:24,433][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:46:29,434][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:46:34,435][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:46:39,437][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:46:44,439][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:46:49,441][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:46:54,442][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:46:57,113][ ERROR][PID:269433] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 17:46:59,444][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:47:04,445][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:47:07,117][WARNING][PID:269433] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-09-11 17:47:09,446][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:47:14,447][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:47:17,118][  INFO][PID:269433] Downloading the builder-live.log file, attempt 4
[2025-09-11 17:47:17,119][  INFO][PID:269433] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.220.252.172 copr-rpmbuild-log
[2025-09-11 17:47:19,449][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:47:24,451][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:47:29,453][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:47:34,454][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:47:39,455][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:47:44,456][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:47:49,457][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:47:54,458][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:47:59,459][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:48:04,460][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:48:09,462][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:48:14,463][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:48:17,188][ ERROR][PID:269433] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-09-11 17:48:19,464][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:48:24,465][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:48:27,189][ ERROR][PID:269433] Giving up for unstable SSH, failures: 1757612666.9716558, 1757612747.044359, 1757612827.1175025, 1757612907.1896136
[2025-09-11 17:48:27,193][ ERROR][PID:269433] Re-try request for task on 'ResallocHost, ticket_id=5012275, hostname=44.220.252.172, name=aws_x86_64_spot_prod_06720393_20250911_173534, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-09-11 17:48:27,194][  INFO][PID:269433] Releasing VM back to pool
[2025-09-11 17:48:27,241][  INFO][PID:269433] Retry #2 (on other host)
[2025-09-11 17:48:27,242][  INFO][PID:269433] Checking for cancel request
[2025-09-11 17:48:27,245][  INFO][PID:269433] VM allocation process starts
[2025-09-11 17:48:27,269][  INFO][PID:269433] Trying to allocate VM: ResallocHost, ticket_id=5012549, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 17:48:30,308][  INFO][PID:269433] Allocated host ResallocHost, ticket_id=5012549, hostname=107.20.29.129, name=aws_x86_64_spot_prod_06720560_20250911_174039, requested_tags=['copr_builder', 'arch_x86_64']
[2025-09-11 17:48:30,309][  INFO][PID:269433] Allocating ssh connection to builder
[2025-09-11 17:48:30,310][  INFO][PID:269433] Checking that builder machine is OK
[2025-09-11 17:49:30,379][ ERROR][PID:269433] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 107.20.29.129 port 22: Connection timed out

[2025-09-11 17:49:40,381][ ERROR][PID:269433] Re-try request for task on 'ResallocHost, ticket_id=5012549, hostname=107.20.29.129, name=aws_x86_64_spot_prod_06720560_20250911_174039, requested_tags=['copr_builder', 'arch_x86_64']': Unable to finish after 1 SSH attempts
[2025-09-11 17:49:40,381][  INFO][PID:269433] Releasing VM back to pool
[2025-09-11 17:49:40,400][ ERROR][PID:269433] Backend process error: Three host tried without success: {'44.220.252.172', '107.20.29.129', '54.198.112.99'}
[2025-09-11 17:49:40,402][WARNING][PID:269433] Switching not-finished job state to 'failed'
[2025-09-11 17:49:40,403][  INFO][PID:269433] Worker failed build, took 666.5144081115723
[2025-09-11 17:49:40,404][  INFO][PID:269433] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "bieszczaders",
            "project_name": "kernel-cachyos",
            "project_dirname": "kernel-cachyos",
            "submitter": "trixieua",
            "ended_on": 1757612980.402613,
            "started_on": 1757612313.8882048,
            "submitted_on": null,
            "status": 0,
            "chroot": "epel-9-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "9546924-epel-9-x86_64",
            "build_id": 9546924,
            "package_name": "kernel-cachyos-server",
            "package_version": "6.16.6-cachyserver1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/bieszczaders/kernel-cachyos/kernel-cachyos-server",
            "git_hash": "2ed9b600ea08f77d8e726f301f668d51c2b53f2c",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "bieszczaders/kernel-cachyos--trixieua",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/bieszczaders/kernel-cachyos/epel-9-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/bieszczaders/kernel-cachyos",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/bieszczaders/kernel-cachyos",
            "result_dir": "09546924-kernel-cachyos-server",
            "built_packages": "",
            "id": 9546924,
            "mockchain_macros": {
                "copr_username": "bieszczaders",
                "copr_projectname": "kernel-cachyos",
                "vendor": "Fedora Project COPR (bieszczaders/kernel-cachyos)"
            }
        }
    ]
}
[2025-09-11 17:49:40,440][  INFO][PID:269433] Sending fedora-messaging bus message in build.end
[2025-09-11 17:49:40,479][  INFO][PID:269433] Compressing /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos/epel-9-x86_64/09546924-kernel-cachyos-server/builder-live.log by gzip
[2025-09-11 17:49:40,481][  INFO][PID:269433] Running command 'gzip /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos/epel-9-x86_64/09546924-kernel-cachyos-server/builder-live.log' as PID 452961
[2025-09-11 17:49:40,488][  INFO][PID:269433] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos/epel-9-x86_64/09546924-kernel-cachyos-server/builder-live.log)
[2025-09-11 17:49:40,489][  INFO][PID:269433] Compressing /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos/epel-9-x86_64/09546924-kernel-cachyos-server/backend.log by gzip
[2025-09-11 17:49:40,491][  INFO][PID:269433] Running command 'gzip /var/lib/copr/public_html/results/bieszczaders/kernel-cachyos/epel-9-x86_64/09546924-kernel-cachyos-server/backend.log' as PID 452962