[2026-04-20 13:46:03,964][  INFO][PID:2346609] Marking build as starting
[2026-04-20 13:46:04,065][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:46:04,071][  INFO][PID:2346609] VM allocation process starts
[2026-04-20 13:46:04,107][  INFO][PID:2346609] Trying to allocate VM: ResallocHost, ticket_id=2422904, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:46:09,113][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:46:14,116][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:46:19,143][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:46:24,150][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:46:29,154][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:46:34,157][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:46:39,159][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:46:42,362][  INFO][PID:2346609] Allocated host ResallocHost, ticket_id=2422904, hostname=98.92.88.32, name=aws_x86_64_reserved_prod_07133205_20260420_134346, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:46:42,364][  INFO][PID:2346609] Allocating ssh connection to builder
[2026-04-20 13:46:42,367][  INFO][PID:2346609] Checking that builder machine is OK
[2026-04-20 13:46:42,552][ ERROR][PID:2346609] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '98.92.88.32' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@98.92.88.32: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:46:52,579][ ERROR][PID:2346609] Re-try request for task on 'ResallocHost, ticket_id=2422904, hostname=98.92.88.32, name=aws_x86_64_reserved_prod_07133205_20260420_134346, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:46:52,625][  INFO][PID:2346609] VM Release request
[2026-04-20 13:46:52,697][  INFO][PID:2346609] Retry #1 (on other host)
[2026-04-20 13:46:52,704][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:46:52,714][  INFO][PID:2346609] VM allocation process starts
[2026-04-20 13:46:52,741][  INFO][PID:2346609] Trying to allocate VM: ResallocHost, ticket_id=2422936, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:46:57,754][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:47:02,757][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:47:07,777][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:47:12,792][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:47:17,798][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:47:22,800][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:47:27,806][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:47:30,986][  INFO][PID:2346609] Allocated host ResallocHost, ticket_id=2422936, hostname=2620:52:6:1161:dead:beef:cafe:c162, name=vmhost_x86_03_prod_07133168_20260420_134328, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:47:30,991][  INFO][PID:2346609] Allocating ssh connection to builder
[2026-04-20 13:47:30,998][  INFO][PID:2346609] Checking that builder machine is OK
[2026-04-20 13:47:31,371][ ERROR][PID:2346609] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c162' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c162: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:47:41,467][ ERROR][PID:2346609] Re-try request for task on 'ResallocHost, ticket_id=2422936, hostname=2620:52:6:1161:dead:beef:cafe:c162, name=vmhost_x86_03_prod_07133168_20260420_134328, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:47:41,522][  INFO][PID:2346609] VM Release request
[2026-04-20 13:47:41,589][  INFO][PID:2346609] Retry #2 (on other host)
[2026-04-20 13:47:41,592][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:47:41,620][  INFO][PID:2346609] VM allocation process starts
[2026-04-20 13:47:41,689][  INFO][PID:2346609] Trying to allocate VM: ResallocHost, ticket_id=2422975, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:47:46,696][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:47:51,704][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:47:56,716][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:48:01,725][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:48:06,755][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:48:11,763][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:48:16,770][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:48:21,780][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:48:26,783][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:48:30,083][  INFO][PID:2346609] Allocated host ResallocHost, ticket_id=2422975, hostname=2620:52:6:1161:dead:beef:cafe:c162, name=vmhost_x86_03_prod_07133168_20260420_134328, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:48:30,085][  INFO][PID:2346609] Allocating ssh connection to builder
[2026-04-20 13:48:30,090][  INFO][PID:2346609] Checking that builder machine is OK
[2026-04-20 13:48:30,407][ ERROR][PID:2346609] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c162' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c162: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:48:40,444][ ERROR][PID:2346609] Re-try request for task on 'ResallocHost, ticket_id=2422975, hostname=2620:52:6:1161:dead:beef:cafe:c162, name=vmhost_x86_03_prod_07133168_20260420_134328, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:48:40,470][  INFO][PID:2346609] VM Release request
[2026-04-20 13:48:40,493][  INFO][PID:2346609] Retry #3 (on other host)
[2026-04-20 13:48:40,499][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:48:40,502][  INFO][PID:2346609] VM allocation process starts
[2026-04-20 13:48:40,529][  INFO][PID:2346609] Trying to allocate VM: ResallocHost, ticket_id=2423013, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:48:45,536][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:48:50,540][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:48:55,550][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:49:00,557][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:49:05,565][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:49:10,575][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:49:15,590][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:49:18,749][  INFO][PID:2346609] Allocated host ResallocHost, ticket_id=2423013, hostname=2620:52:6:1161:dead:beef:cafe:c162, name=vmhost_x86_03_prod_07133168_20260420_134328, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:49:18,752][  INFO][PID:2346609] Allocating ssh connection to builder
[2026-04-20 13:49:18,765][  INFO][PID:2346609] Checking that builder machine is OK
[2026-04-20 13:49:19,078][ ERROR][PID:2346609] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c162' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c162: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:49:29,227][ ERROR][PID:2346609] Re-try request for task on 'ResallocHost, ticket_id=2423013, hostname=2620:52:6:1161:dead:beef:cafe:c162, name=vmhost_x86_03_prod_07133168_20260420_134328, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:49:29,341][  INFO][PID:2346609] VM Release request
[2026-04-20 13:49:29,441][  INFO][PID:2346609] Retry #4 (on other host)
[2026-04-20 13:49:29,458][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:49:29,496][  INFO][PID:2346609] VM allocation process starts
[2026-04-20 13:49:29,580][  INFO][PID:2346609] Trying to allocate VM: ResallocHost, ticket_id=2423046, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:49:34,588][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:49:39,592][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:49:44,597][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:49:49,608][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:49:54,614][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:49:59,623][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:50:04,627][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:50:09,630][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:50:14,634][  INFO][PID:2346609] Checking for cancel request
[2026-04-20 13:50:17,850][  INFO][PID:2346609] Allocated host ResallocHost, ticket_id=2423046, hostname=2620:52:6:1161:dead:beef:cafe:c132, name=vmhost_p09_04_prod_07132682_20260420_133556, requested_tags=['arch_noarch', 'copr_builder']
[2026-04-20 13:50:17,851][  INFO][PID:2346609] Allocating ssh connection to builder
[2026-04-20 13:50:17,852][  INFO][PID:2346609] Checking that builder machine is OK
[2026-04-20 13:50:18,262][ ERROR][PID:2346609] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c132' (ED25519) to the list of known hosts.
Load key "/home/copr/.ssh/id_rsa": Permission denied
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@2620:52:6:1161:dead:beef:cafe:c132: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2026-04-20 13:50:28,334][ ERROR][PID:2346609] Re-try request for task on 'ResallocHost, ticket_id=2423046, hostname=2620:52:6:1161:dead:beef:cafe:c132, name=vmhost_p09_04_prod_07132682_20260420_133556, requested_tags=['arch_noarch', 'copr_builder']': Unable to finish after 1 SSH attempts
[2026-04-20 13:50:28,366][  INFO][PID:2346609] VM Release request
[2026-04-20 13:50:28,398][ ERROR][PID:2346609] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c132', '2620:52:6:1161:dead:beef:cafe:c162', '98.92.88.32'}
[2026-04-20 13:50:28,453][WARNING][PID:2346609] Switching not-finished job state to 'failed'
[2026-04-20 13:50:28,462][  INFO][PID:2346609] Worker failed build, took 264.58800196647644
[2026-04-20 13:50:28,464][  INFO][PID:2346609] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "@rocm-packagers-sig",
            "project_name": "preview",
            "project_dirname": "preview",
            "submitter": "trix",
            "ended_on": 1776693028.4535096,
            "started_on": 1776692763.8655076,
            "submitted_on": null,
            "status": 0,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "10348648",
            "build_id": 10348648,
            "package_name": null,
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 2,
            "source_json": "{\"url\": \"https://copr.fedorainfracloud.org/tmp/tmpc0767ppz/rocprim-7.12.0-0.fc45.src.rpm\", \"pkg\": \"rocprim-7.12.0-0.fc45.src.rpm\", \"tmp\": \"tmpc0767ppz\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "@rocm-packagers-sig/preview--trix",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [],
            "background": false,
            "distributions_in_build": [
                "fedora-rawhide"
            ],
            "distributions_in_project": [
                "fedora-rawhide"
            ],
            "destdir": "/var/lib/copr/public_html/results/@rocm-packagers-sig/preview",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@rocm-packagers-sig/preview",
            "result_dir": "10348648",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "id": 10348648,
            "mockchain_macros": {
                "copr_username": "@rocm-packagers-sig",
                "copr_projectname": "preview",
                "vendor": "Fedora Project COPR (@rocm-packagers-sig/preview)"
            }
        }
    ]
}
[2026-04-20 13:50:28,521][  INFO][PID:2346609] Sending fedora-messaging bus message in build.end
[2026-04-20 13:50:29,407][  INFO][PID:2346609] Compressing /var/lib/copr/public_html/results/@rocm-packagers-sig/preview/srpm-builds/10348648/builder-live.log by gzip
[2026-04-20 13:50:29,434][  INFO][PID:2346609] Running command 'gzip /var/lib/copr/public_html/results/@rocm-packagers-sig/preview/srpm-builds/10348648/builder-live.log' as PID 2406044
[2026-04-20 13:50:29,455][  INFO][PID:2346609] Finished after 0 seconds with exit code 1 (gzip /var/lib/copr/public_html/results/@rocm-packagers-sig/preview/srpm-builds/10348648/builder-live.log)
stdout:

stderr:
gzip: /var/lib/copr/public_html/results/@rocm-packagers-sig/preview/srpm-builds/10348648/builder-live.log: No such file or directory


[2026-04-20 13:50:29,472][ ERROR][PID:2346609] Unable to compress file /var/lib/copr/public_html/results/@rocm-packagers-sig/preview/srpm-builds/10348648/builder-live.log
[2026-04-20 13:50:29,697][  INFO][PID:2346609] Compressing /var/lib/copr/public_html/results/@rocm-packagers-sig/preview/srpm-builds/10348648/backend.log by gzip
[2026-04-20 13:50:29,710][  INFO][PID:2346609] Running command 'gzip /var/lib/copr/public_html/results/@rocm-packagers-sig/preview/srpm-builds/10348648/backend.log' as PID 2406101