[2026-04-20 14:14:41,942][  INFO][PID:2763690] Marking build as starting
[2026-04-20 14:14:42,014][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:14:42,019][  INFO][PID:2763690] VM allocation process starts
[2026-04-20 14:14:42,094][  INFO][PID:2763690] Trying to allocate VM: ResallocHost, ticket_id=2423779, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:14:47,103][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:14:52,105][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:14:57,111][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:15:02,116][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:15:07,119][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:15:12,121][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:15:17,126][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:15:22,170][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:15:27,208][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:15:32,224][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:15:37,243][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:15:42,274][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:15:47,280][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:15:52,287][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:15:57,293][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:16:02,301][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:16:07,308][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:16:12,312][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:16:17,317][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:16:22,335][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:16:27,358][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:16:32,383][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:16:37,393][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:16:42,403][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:16:47,412][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:16:52,422][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:16:57,433][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:17:02,444][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:17:07,479][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:17:12,495][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:17:17,504][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:17:22,527][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:17:27,540][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:17:32,555][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:17:37,561][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:17:42,568][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:17:47,572][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:17:52,576][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:17:57,578][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:18:01,007][  INFO][PID:2763690] Allocated host ResallocHost, ticket_id=2423779, hostname=140.211.168.9, name=copr_osuosl_p09_prod_07134726_20260420_141218, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:18:01,008][  INFO][PID:2763690] Allocating ssh connection to builder
[2026-04-20 14:18:01,011][  INFO][PID:2763690] Checking that builder machine is OK
[2026-04-20 14:18:01,919][ ERROR][PID:2763690] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '140.211.168.9' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@140.211.168.9: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:18:11,991][ ERROR][PID:2763690] Re-try request for task on 'ResallocHost, ticket_id=2423779, hostname=140.211.168.9, name=copr_osuosl_p09_prod_07134726_20260420_141218, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:18:12,288][  INFO][PID:2763690] VM Release request
[2026-04-20 14:18:12,421][  INFO][PID:2763690] Retry #1 (on other host)
[2026-04-20 14:18:12,426][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:18:12,431][  INFO][PID:2763690] VM allocation process starts
[2026-04-20 14:18:12,540][  INFO][PID:2763690] Trying to allocate VM: ResallocHost, ticket_id=2423859, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:18:17,562][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:18:22,581][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:18:27,595][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:18:32,607][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:18:37,610][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:18:42,612][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:18:47,619][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:18:52,626][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:18:57,630][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:19:02,640][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:19:07,646][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:19:12,663][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:19:17,684][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:19:21,203][  INFO][PID:2763690] Allocated host ResallocHost, ticket_id=2423859, hostname=2620:52:6:1161:dead:beef:cafe:c10b, name=vmhost_p09_01_prod_07134914_20260420_141504, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:19:21,208][  INFO][PID:2763690] Allocating ssh connection to builder
[2026-04-20 14:19:21,210][  INFO][PID:2763690] Checking that builder machine is OK
[2026-04-20 14:19:21,579][ ERROR][PID:2763690] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c10b' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c10b: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:19:31,635][ ERROR][PID:2763690] Re-try request for task on 'ResallocHost, ticket_id=2423859, hostname=2620:52:6:1161:dead:beef:cafe:c10b, name=vmhost_p09_01_prod_07134914_20260420_141504, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:19:31,800][  INFO][PID:2763690] VM Release request
[2026-04-20 14:19:31,882][  INFO][PID:2763690] Retry #2 (on other host)
[2026-04-20 14:19:31,888][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:19:31,895][  INFO][PID:2763690] VM allocation process starts
[2026-04-20 14:19:31,929][  INFO][PID:2763690] Trying to allocate VM: ResallocHost, ticket_id=2423901, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:19:36,939][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:19:41,953][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:19:46,959][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:19:50,177][  INFO][PID:2763690] Allocated host ResallocHost, ticket_id=2423901, hostname=2620:52:6:1161:dead:beef:cafe:c10b, name=vmhost_p09_01_prod_07134914_20260420_141504, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:19:50,183][  INFO][PID:2763690] Allocating ssh connection to builder
[2026-04-20 14:19:50,184][  INFO][PID:2763690] Checking that builder machine is OK
[2026-04-20 14:19:50,660][ ERROR][PID:2763690] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c10b' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c10b: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:20:00,880][ ERROR][PID:2763690] Re-try request for task on 'ResallocHost, ticket_id=2423901, hostname=2620:52:6:1161:dead:beef:cafe:c10b, name=vmhost_p09_01_prod_07134914_20260420_141504, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:20:00,919][  INFO][PID:2763690] VM Release request
[2026-04-20 14:20:00,973][  INFO][PID:2763690] Retry #3 (on other host)
[2026-04-20 14:20:00,976][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:20:00,981][  INFO][PID:2763690] VM allocation process starts
[2026-04-20 14:20:00,998][  INFO][PID:2763690] Trying to allocate VM: ResallocHost, ticket_id=2423914, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:20:06,005][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:20:11,020][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:20:16,028][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:20:21,032][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:20:26,034][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:20:31,037][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:20:36,044][  INFO][PID:2763690] Checking for cancel request
[2026-04-20 14:20:39,424][  INFO][PID:2763690] Allocated host ResallocHost, ticket_id=2423914, hostname=3.80.145.227, name=aws_x86_64_reserved_prod_07135112_20260420_141745, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 14:20:39,425][  INFO][PID:2763690] Allocating ssh connection to builder
[2026-04-20 14:20:39,430][  INFO][PID:2763690] Checking that builder machine is OK
[2026-04-20 14:20:39,792][ ERROR][PID:2763690] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '3.80.145.227' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@3.80.145.227: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 14:20:49,872][ ERROR][PID:2763690] Re-try request for task on 'ResallocHost, ticket_id=2423914, hostname=3.80.145.227, name=aws_x86_64_reserved_prod_07135112_20260420_141745, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 14:20:50,031][  INFO][PID:2763690] VM Release request
[2026-04-20 14:20:50,139][ ERROR][PID:2763690] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c10b', '140.211.168.9', '3.80.145.227'}
[2026-04-20 14:20:50,269][WARNING][PID:2763690] Switching not-finished job state to 'failed'
[2026-04-20 14:20:50,272][  INFO][PID:2763690] Worker failed build, took 368.4069015979767
[2026-04-20 14:20:50,276][  INFO][PID:2763690] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "jstephen",
            "project_name": "sssd-testing",
            "project_dirname": "sssd-testing",
            "submitter": "jstephen",
            "ended_on": 1776694850.2695537,
            "started_on": 1776694481.862652,
            "submitted_on": null,
            "status": 0,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "10348748",
            "build_id": 10348748,
            "package_name": null,
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 2,
            "source_json": "{\"url\": \"https://copr.fedorainfracloud.org/tmp/tmp2es4u_sj/sssd-2.13.0-0.20260420.1013.git1bdc46268.fc43.src.rpm\", \"pkg\": \"sssd-2.13.0-0.20260420.1013.git1bdc46268.fc43.src.rpm\", \"tmp\": \"tmp2es4u_sj\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "jstephen/sssd-testing--jstephen",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": false,
            "distributions_in_build": [],
            "distributions_in_project": [
                "fedora-42",
                "fedora-43",
                "fedora-44",
                "fedora-rawhide"
            ],
            "destdir": "/var/lib/copr/public_html/results/jstephen/sssd-testing",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/jstephen/sssd-testing",
            "result_dir": "10348748",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "id": 10348748,
            "mockchain_macros": {
                "copr_username": "jstephen",
                "copr_projectname": "sssd-testing",
                "vendor": "Fedora Project COPR (jstephen/sssd-testing)"
            }
        }
    ]
}
[2026-04-20 14:20:50,517][  INFO][PID:2763690] Sending fedora-messaging bus message in build.end
[2026-04-20 14:20:53,062][  INFO][PID:2763690] Compressing /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348748/builder-live.log by gzip
[2026-04-20 14:20:53,082][  INFO][PID:2763690] Running command 'gzip /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348748/builder-live.log' as PID 2850580
[2026-04-20 14:20:53,127][  INFO][PID:2763690] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348748/builder-live.log)
stdout:

stderr:
gzip: /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348748/builder-live.log: No such file or directory


[2026-04-20 14:20:53,147][ ERROR][PID:2763690] Unable to compress file /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348748/builder-live.log
[2026-04-20 14:20:53,404][  INFO][PID:2763690] Compressing /var/lib/copr/public_html/results/jstephen/sssd-testing/srpm-builds/10348748/backend.log by gzip