[2026-04-20 13:33:11,715][  INFO][PID:2167308] Marking build as starting
[2026-04-20 13:33:11,886][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:33:11,897][  INFO][PID:2167308] VM allocation process starts
[2026-04-20 13:33:11,945][  INFO][PID:2167308] Trying to allocate VM: ResallocHost, ticket_id=2422499, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:33:16,953][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:33:21,960][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:33:26,962][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:33:31,964][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:33:36,967][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:33:40,125][  INFO][PID:2167308] Allocated host ResallocHost, ticket_id=2422499, hostname=2620:52:6:1161:dead:beef:cafe:c144, name=vmhost_p09_03_prod_07116035_20260420_121301, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:33:40,129][  INFO][PID:2167308] Allocating ssh connection to builder
[2026-04-20 13:33:40,133][  INFO][PID:2167308] Checking that builder machine is OK
[2026-04-20 13:33:40,467][ ERROR][PID:2167308] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c144' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c144: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:33:50,530][ ERROR][PID:2167308] Re-try request for task on 'ResallocHost, ticket_id=2422499, hostname=2620:52:6:1161:dead:beef:cafe:c144, name=vmhost_p09_03_prod_07116035_20260420_121301, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:33:50,577][  INFO][PID:2167308] VM Release request
[2026-04-20 13:33:50,648][  INFO][PID:2167308] Retry #1 (on other host)
[2026-04-20 13:33:50,649][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:33:50,650][  INFO][PID:2167308] VM allocation process starts
[2026-04-20 13:33:50,683][  INFO][PID:2167308] Trying to allocate VM: ResallocHost, ticket_id=2422520, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:33:55,689][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:34:00,698][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:34:05,702][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:34:08,832][  INFO][PID:2167308] Allocated host ResallocHost, ticket_id=2422520, hostname=2620:52:6:1161:dead:beef:cafe:c163, name=vmhost_p09_03_prod_07123297_20260420_124101, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:34:08,834][  INFO][PID:2167308] Allocating ssh connection to builder
[2026-04-20 13:34:08,840][  INFO][PID:2167308] Checking that builder machine is OK
[2026-04-20 13:34:09,222][ ERROR][PID:2167308] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c163' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c163: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:34:19,271][ ERROR][PID:2167308] Re-try request for task on 'ResallocHost, ticket_id=2422520, hostname=2620:52:6:1161:dead:beef:cafe:c163, name=vmhost_p09_03_prod_07123297_20260420_124101, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:34:19,311][  INFO][PID:2167308] VM Release request
[2026-04-20 13:34:19,358][  INFO][PID:2167308] Retry #2 (on other host)
[2026-04-20 13:34:19,365][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:34:19,384][  INFO][PID:2167308] VM allocation process starts
[2026-04-20 13:34:19,472][  INFO][PID:2167308] Trying to allocate VM: ResallocHost, ticket_id=2422551, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:34:24,488][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:34:29,497][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:34:34,501][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:34:39,512][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:34:44,531][  INFO][PID:2167308] Checking for cancel request
[2026-04-20 13:34:47,851][  INFO][PID:2167308] Allocated host ResallocHost, ticket_id=2422551, hostname=2620:52:6:1161:dead:beef:cafe:c16c, name=vmhost_p09_04_prod_07130246_20260420_131140, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:34:47,852][  INFO][PID:2167308] Allocating ssh connection to builder
[2026-04-20 13:34:47,857][  INFO][PID:2167308] Checking that builder machine is OK
[2026-04-20 13:34:48,297][ ERROR][PID:2167308] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c16c' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c16c: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:34:58,338][ ERROR][PID:2167308] Re-try request for task on 'ResallocHost, ticket_id=2422551, hostname=2620:52:6:1161:dead:beef:cafe:c16c, name=vmhost_p09_04_prod_07130246_20260420_131140, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:34:58,364][  INFO][PID:2167308] VM Release request
[2026-04-20 13:34:58,453][ ERROR][PID:2167308] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c144', '2620:52:6:1161:dead:beef:cafe:c16c', '2620:52:6:1161:dead:beef:cafe:c163'}
[2026-04-20 13:34:58,580][WARNING][PID:2167308] Switching not-finished job state to 'failed'
[2026-04-20 13:34:58,605][  INFO][PID:2167308] Worker failed build, took 107.10341548919678
[2026-04-20 13:34:58,617][  INFO][PID:2167308] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "@rocm-packagers-sig",
            "project_name": "rocm-test",
            "project_dirname": "rocm-test",
            "submitter": "https://src.fedoraproject.org/user/trix",
            "ended_on": 1776692098.5775785,
            "started_on": 1776691991.474163,
            "submitted_on": null,
            "status": 0,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "10348611",
            "build_id": 10348611,
            "package_name": "rocblas",
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 10,
            "source_json": "{\"clone_url\": \"https://src.fedoraproject.org/rpms/rocblas\", \"distgit\": \"fedora\", \"committish\": \"9d2ab3f360e248fd9a88ff35b1d2a6cba15b7956\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "@rocm-packagers-sig/rocm-test--https://src.fedoraproject.org/user/trix",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": true,
            "distributions_in_build": [
                "fedora-rawhide"
            ],
            "distributions_in_project": [
                "fedora-rawhide"
            ],
            "destdir": "/var/lib/copr/public_html/results/@rocm-packagers-sig/rocm-test",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@rocm-packagers-sig/rocm-test",
            "result_dir": "10348611",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "id": 10348611,
            "mockchain_macros": {
                "copr_username": "@rocm-packagers-sig",
                "copr_projectname": "rocm-test",
                "vendor": "Fedora Project COPR (@rocm-packagers-sig/rocm-test)"
            }
        }
    ]
}
[2026-04-20 13:34:58,869][  INFO][PID:2167308] Sending fedora-messaging bus message in build.end
[2026-04-20 13:35:01,834][  INFO][PID:2167308] Compressing /var/lib/copr/public_html/results/@rocm-packagers-sig/rocm-test/srpm-builds/10348611/builder-live.log by gzip
[2026-04-20 13:35:01,863][  INFO][PID:2167308] Running command 'gzip /var/lib/copr/public_html/results/@rocm-packagers-sig/rocm-test/srpm-builds/10348611/builder-live.log' as PID 2192334
[2026-04-20 13:35:01,865][  INFO][PID:2167308] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/@rocm-packagers-sig/rocm-test/srpm-builds/10348611/builder-live.log)
stdout:

stderr:
gzip: /var/lib/copr/public_html/results/@rocm-packagers-sig/rocm-test/srpm-builds/10348611/builder-live.log: No such file or directory


[2026-04-20 13:35:01,869][ ERROR][PID:2167308] Unable to compress file /var/lib/copr/public_html/results/@rocm-packagers-sig/rocm-test/srpm-builds/10348611/builder-live.log
[2026-04-20 13:35:01,928][  INFO][PID:2167308] Compressing /var/lib/copr/public_html/results/@rocm-packagers-sig/rocm-test/srpm-builds/10348611/backend.log by gzip
[2026-04-20 13:35:01,938][  INFO][PID:2167308] Running command 'gzip /var/lib/copr/public_html/results/@rocm-packagers-sig/rocm-test/srpm-builds/10348611/backend.log' as PID 2192357