[2026-04-20 14:01:57,463][  INFO][PID:2568064] Marking build as starting
[2026-04-20 14:01:57,648][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:01:57,671][  INFO][PID:2568064] VM allocation process starts
[2026-04-20 14:01:57,835][  INFO][PID:2568064] Trying to allocate VM: ResallocHost, ticket_id=2423472, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:02:02,853][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:02:07,866][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:02:12,883][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:02:17,908][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:02:22,961][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:02:27,992][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:02:33,013][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:02:38,022][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:02:43,066][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:02:48,091][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:02:53,111][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:02:58,129][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:03:03,134][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:03:08,145][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:03:13,164][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:03:18,175][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:03:23,215][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:03:28,220][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:03:33,236][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:03:38,243][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:03:43,248][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:03:48,250][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:03:53,252][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:03:58,260][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:04:03,270][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:04:08,277][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:04:13,301][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:04:16,683][  INFO][PID:2568064] Allocated host ResallocHost, ticket_id=2423472, hostname=2620:52:6:1161:dead:beef:cafe:c11b, name=vmhost_p09_02_prod_07133923_20260420_135859, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:04:16,685][  INFO][PID:2568064] Allocating ssh connection to builder
[2026-04-20 14:04:16,686][  INFO][PID:2568064] Checking that builder machine is OK
[2026-04-20 14:04:17,426][ ERROR][PID:2568064] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c11b' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c11b: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:04:27,591][ ERROR][PID:2568064] Re-try request for task on 'ResallocHost, ticket_id=2423472, hostname=2620:52:6:1161:dead:beef:cafe:c11b, name=vmhost_p09_02_prod_07133923_20260420_135859, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:04:27,680][  INFO][PID:2568064] VM Release request
[2026-04-20 14:04:27,739][  INFO][PID:2568064] Retry #1 (on other host)
[2026-04-20 14:04:27,741][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:04:27,745][  INFO][PID:2568064] VM allocation process starts
[2026-04-20 14:04:27,804][  INFO][PID:2568064] Trying to allocate VM: ResallocHost, ticket_id=2423523, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:04:32,822][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:04:37,826][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:04:42,846][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:04:47,851][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:04:52,860][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:04:57,865][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:05:02,870][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:05:07,903][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:05:12,917][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:05:17,924][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:05:22,947][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:05:27,979][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:05:32,989][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:05:37,995][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:05:43,000][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:05:48,009][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:05:53,016][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:05:56,849][  INFO][PID:2568064] Allocated host ResallocHost, ticket_id=2423523, hostname=2620:52:6:1161:dead:beef:cafe:c116, name=vmhost_p09_02_prod_07134023_20260420_140033, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:05:56,855][  INFO][PID:2568064] Allocating ssh connection to builder
[2026-04-20 14:05:56,859][  INFO][PID:2568064] Checking that builder machine is OK
[2026-04-20 14:05:57,317][ ERROR][PID:2568064] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c116' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c116: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:06:07,403][ ERROR][PID:2568064] Re-try request for task on 'ResallocHost, ticket_id=2423523, hostname=2620:52:6:1161:dead:beef:cafe:c116, name=vmhost_p09_02_prod_07134023_20260420_140033, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:06:07,449][  INFO][PID:2568064] VM Release request
[2026-04-20 14:06:07,466][  INFO][PID:2568064] Retry #2 (on other host)
[2026-04-20 14:06:07,470][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:06:07,475][  INFO][PID:2568064] VM allocation process starts
[2026-04-20 14:06:07,513][  INFO][PID:2568064] Trying to allocate VM: ResallocHost, ticket_id=2423593, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:06:12,528][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:06:17,536][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:06:22,544][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:06:27,554][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:06:32,577][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:06:35,905][  INFO][PID:2568064] Allocated host ResallocHost, ticket_id=2423593, hostname=2620:52:6:1161:dead:beef:cafe:c116, name=vmhost_p09_02_prod_07134023_20260420_140033, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:06:35,908][  INFO][PID:2568064] Allocating ssh connection to builder
[2026-04-20 14:06:35,910][  INFO][PID:2568064] Checking that builder machine is OK
[2026-04-20 14:06:36,422][ ERROR][PID:2568064] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c116' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c116: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:06:46,660][ ERROR][PID:2568064] Re-try request for task on 'ResallocHost, ticket_id=2423593, hostname=2620:52:6:1161:dead:beef:cafe:c116, name=vmhost_p09_02_prod_07134023_20260420_140033, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:06:46,783][  INFO][PID:2568064] VM Release request
[2026-04-20 14:06:46,898][  INFO][PID:2568064] Retry #3 (on other host)
[2026-04-20 14:06:46,900][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:06:46,904][  INFO][PID:2568064] VM allocation process starts
[2026-04-20 14:06:46,951][  INFO][PID:2568064] Trying to allocate VM: ResallocHost, ticket_id=2423603, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:06:51,965][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:06:56,973][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:07:01,979][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:07:05,185][  INFO][PID:2568064] Allocated host ResallocHost, ticket_id=2423603, hostname=2620:52:6:1161:dead:beef:cafe:c116, name=vmhost_p09_02_prod_07134023_20260420_140033, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:07:05,188][  INFO][PID:2568064] Allocating ssh connection to builder
[2026-04-20 14:07:05,190][  INFO][PID:2568064] Checking that builder machine is OK
[2026-04-20 14:07:05,562][ ERROR][PID:2568064] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c116' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c116: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:07:15,617][ ERROR][PID:2568064] Re-try request for task on 'ResallocHost, ticket_id=2423603, hostname=2620:52:6:1161:dead:beef:cafe:c116, name=vmhost_p09_02_prod_07134023_20260420_140033, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:07:15,631][  INFO][PID:2568064] VM Release request
[2026-04-20 14:07:18,652][  INFO][PID:2568064] Retry #4 (on other host)
[2026-04-20 14:07:18,654][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:07:18,659][  INFO][PID:2568064] VM allocation process starts
[2026-04-20 14:07:18,691][  INFO][PID:2568064] Trying to allocate VM: ResallocHost, ticket_id=2423619, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:07:23,716][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:07:28,734][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:07:33,751][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:07:38,775][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:07:43,797][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:07:48,803][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:07:53,807][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:07:58,819][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:08:03,825][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:08:08,829][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:08:13,835][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:08:18,844][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:08:23,849][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:08:28,861][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:08:33,866][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:08:38,879][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:08:43,884][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:08:48,897][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:08:53,902][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:08:58,915][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:09:03,933][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:09:08,945][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:09:13,973][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:09:19,004][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:09:24,024][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:09:29,050][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:09:34,063][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:09:39,068][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:09:44,073][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:09:49,083][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:09:54,101][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:09:59,117][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:10:04,130][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:10:09,133][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:10:14,141][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:10:19,153][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:10:24,182][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:10:29,189][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:10:34,192][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:10:39,196][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:10:44,202][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:10:49,219][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:10:54,233][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:10:59,252][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:11:04,262][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:11:09,264][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:11:14,272][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:11:19,276][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:11:24,294][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:11:29,304][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:11:34,315][  INFO][PID:2568064] Checking for cancel request
[2026-04-20 14:11:37,583][  INFO][PID:2568064] Allocated host ResallocHost, ticket_id=2423619, hostname=3.92.128.117, name=aws_x86_64_reserved_prod_07134474_20260420_140738, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:11:37,585][  INFO][PID:2568064] Allocating ssh connection to builder
[2026-04-20 14:11:37,591][  INFO][PID:2568064] Checking that builder machine is OK
[2026-04-20 14:11:37,888][ ERROR][PID:2568064] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '3.92.128.117' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@3.92.128.117: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:11:47,986][ ERROR][PID:2568064] Re-try request for task on 'ResallocHost, ticket_id=2423619, hostname=3.92.128.117, name=aws_x86_64_reserved_prod_07134474_20260420_140738, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:11:48,019][  INFO][PID:2568064] VM Release request
[2026-04-20 14:11:48,125][ ERROR][PID:2568064] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c116', '3.92.128.117', '2620:52:6:1161:dead:beef:cafe:c11b'}
[2026-04-20 14:11:48,179][WARNING][PID:2568064] Switching not-finished job state to 'failed'
[2026-04-20 14:11:48,183][  INFO][PID:2568064] Worker failed build, took 591.035078048706
[2026-04-20 14:11:48,186][  INFO][PID:2568064] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "jstephen",
            "project_name": "sssd-testing",
            "project_dirname": "sssd-testing",
            "submitter": "jstephen",
            "ended_on": 1776694308.178949,
            "started_on": 1776693717.143871,
            "submitted_on": null,
            "status": 0,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "10348694",
            "build_id": 10348694,
            "package_name": null,
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 2,
            "source_json": "{\"url\": \"https://copr.fedorainfracloud.org/tmp/tmpmnacwu6q/sssd-2.13.0-0.20260420.1000.git1bdc46268.fc43.src.rpm\", \"pkg\": \"sssd-2.13.0-0.20260420.1000.git1bdc46268.fc43.src.rpm\", \"tmp\": \"tmpmnacwu6q\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "jstephen/sssd-testing--jstephen",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": false,
            "distributions_in_build": [],
            "distributions_in_project": [
                "fedora-42",
                "fedora-43",
                "fedora-44",
                "fedora-rawhide"
            ],
            "destdir": "/var/lib/copr/public_html/results/jstephen/sssd-testing",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/jstephen/sssd-testing",
            "result_dir": "10348694",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "id": 10348694,
            "mockchain_macros": {
                "copr_username": "jstephen",
                "copr_projectname": "sssd-testing",
                "vendor": "Fedora Project COPR (jstephen/sssd-testing)"
            }
        }
    ]
}
[2026-04-20 14:11:48,563][  INFO][PID:2568064] Sending fedora-messaging bus message in build.end
[2026-04-20 14:11:50,633][  INFO][PID:2568064] Compressing /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348694/builder-live.log by gzip
[2026-04-20 14:11:50,647][  INFO][PID:2568064] Running command 'gzip /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348694/builder-live.log' as PID 2723914
[2026-04-20 14:11:50,660][  INFO][PID:2568064] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348694/builder-live.log)
stdout:

stderr:
gzip: /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348694/builder-live.log: No such file or directory


[2026-04-20 14:11:50,665][ ERROR][PID:2568064] Unable to compress file /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348694/builder-live.log
[2026-04-20 14:11:50,721][  INFO][PID:2568064] Compressing /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348694/backend.log by gzip
[2026-04-20 14:11:50,735][  INFO][PID:2568064] Running command 'gzip /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348694/backend.log' as PID 2723932