[2026-01-28 17:04:50,880][  INFO][PID:622344] Marking build as starting
[2026-01-28 17:04:51,025][  INFO][PID:622344] Checking that /var/lib/copr/public_html/results/@meta/drgn/centos-stream+epel-next-9-ppc64le/repodata/repomd.xml exists
[2026-01-28 17:04:51,106][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:04:51,107][  INFO][PID:622344] VM allocation process starts
[2026-01-28 17:04:51,119][  INFO][PID:622344] Trying to allocate VM: ResallocHost, ticket_id=1074235, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2026-01-28 17:04:56,120][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:05:01,121][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:05:03,152][  INFO][PID:622344] Allocated host ResallocHost, ticket_id=1074235, hostname=2620:52:6:1161:dead:beef:cafe:c10a, name=vmhost_p09_02_prod_01709122_20260128_170210, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2026-01-28 17:05:03,153][  INFO][PID:622344] Allocating ssh connection to builder
[2026-01-28 17:05:03,153][  INFO][PID:622344] Checking that builder machine is OK
[2026-01-28 17:05:04,104][  INFO][PID:622344] Running remote command: copr-builder-ready centos-stream+epel-next-9-ppc64le
[2026-01-28 17:05:04,680][  INFO][PID:622344] Red Hat subscription not needed for centos-stream+epel-next-9-ppc64le
Builder is ready to be used

[2026-01-28 17:05:04,680][  INFO][PID:622344] Filling build.info file with builder info
[2026-01-28 17:05:04,681][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:05:04,682][  INFO][PID:622344] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@meta",
            "project_name": "drgn",
            "project_dirname": "drgn",
            "submitter": "packit",
            "ended_on": null,
            "started_on": 1769619904.682222,
            "submitted_on": null,
            "status": 3,
            "chroot": "centos-stream+epel-next-9-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "10072774-centos-stream+epel-next-9-ppc64le",
            "build_id": 10072774,
            "package_name": "python-drgn",
            "package_version": "0.0.33+158.g8adc2264-1.20260128165741536433.main.158.g8adc2264",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@meta/drgn/python-drgn",
            "git_hash": "392742313a0e3913c7e6ac46ee4b2441749392db",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@meta/drgn--packit",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@meta/drgn/centos-stream+epel-next-9-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://kojihub.stream.centos.org/kojifiles/repos/c9s-build/latest/ppc64le/",
                    "id": "https_kojihub_stream_centos_org_kojifiles_repos_c9s_build_latest_ppc64le",
                    "name": "Additional repo https_kojihub_stream_centos_org_kojifiles_repos_c9s_build_latest_ppc64le"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le",
                "arch_power9"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@meta/drgn",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@meta/drgn",
            "result_dir": "10072774-python-drgn",
            "built_packages": "",
            "id": 10072774,
            "mockchain_macros": {
                "copr_username": "@meta",
                "copr_projectname": "drgn",
                "vendor": "Fedora Project COPR (@meta/drgn)"
            }
        }
    ]
}
[2026-01-28 17:05:04,793][  INFO][PID:622344] Sending fedora-messaging bus message in build.start
[2026-01-28 17:05:05,380][  INFO][PID:622344] Sending fedora-messaging bus message in chroot.start
[2026-01-28 17:05:05,408][  INFO][PID:622344] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10072774-centos-stream+epel-next-9-ppc64le --chroot centos-stream+epel-next-9-ppc64le --detached
[2026-01-28 17:05:06,411][  INFO][PID:622344] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c10a' (ED25519) to the list of known hosts.


[2026-01-28 17:05:06,412][  INFO][PID:622344] Downloading the builder-live.log file, attempt 1
[2026-01-28 17:05:06,414][  INFO][PID:622344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c10a copr-rpmbuild-log
[2026-01-28 17:05:11,423][  INFO][PID:622344] Periodic builder liveness probe: alive
[2026-01-28 17:05:11,423][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:05:16,424][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:05:21,425][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:05:26,426][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:05:31,427][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:05:36,430][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:05:41,431][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:05:46,433][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:05:51,439][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:05:56,442][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:06:01,445][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:06:06,450][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:06:11,452][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:06:16,454][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:06:21,455][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:06:26,464][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:06:31,467][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:06:36,470][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:06:41,474][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:06:46,479][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:06:51,482][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:06:56,495][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:07:01,501][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:07:06,504][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:07:11,507][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:07:16,510][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:07:21,511][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:07:26,512][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:07:31,513][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:07:36,514][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:07:41,515][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:07:46,518][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:07:51,526][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:07:56,527][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:08:01,532][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:08:06,534][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:08:11,537][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:08:16,539][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:08:21,541][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:08:26,542][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:08:31,550][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:08:36,553][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:08:41,556][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:08:46,558][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:08:51,564][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:08:56,566][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:09:01,568][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:09:06,570][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:09:11,571][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:09:16,572][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:09:21,573][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:09:26,574][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:09:31,576][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:09:36,577][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:09:41,579][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:09:46,581][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:09:51,583][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:09:56,584][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:10:01,587][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:10:06,589][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:10:11,590][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:10:16,591][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:10:21,594][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:10:26,596][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:10:31,597][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:10:36,598][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:10:41,599][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:10:46,606][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:10:51,372][ ERROR][PID:622344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-28 17:10:51,609][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:10:56,610][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:11:01,378][WARNING][PID:622344] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-28 17:11:01,613][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:11:06,615][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:11:11,381][  INFO][PID:622344] Downloading the builder-live.log file, attempt 2
[2026-01-28 17:11:11,382][  INFO][PID:622344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c10a copr-rpmbuild-log
[2026-01-28 17:11:11,619][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:11:16,620][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:11:21,621][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:11:26,622][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:11:31,623][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:11:36,624][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:11:41,625][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:11:46,626][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:11:51,629][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:11:56,630][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:12:01,631][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:12:06,636][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:12:11,452][ ERROR][PID:622344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-28 17:12:11,642][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:12:16,643][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:12:21,458][WARNING][PID:622344] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-28 17:12:21,644][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:12:26,645][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:12:31,459][  INFO][PID:622344] Downloading the builder-live.log file, attempt 3
[2026-01-28 17:12:31,463][  INFO][PID:622344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c10a copr-rpmbuild-log
[2026-01-28 17:12:31,647][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:12:36,650][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:12:41,652][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:12:46,653][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:12:51,655][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:12:56,656][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:13:01,658][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:13:06,659][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:13:11,660][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:13:16,677][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:13:21,681][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:13:26,683][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:13:31,535][ ERROR][PID:622344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-28 17:13:31,685][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:13:36,688][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:13:41,547][WARNING][PID:622344] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-28 17:13:41,689][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:13:46,690][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:13:51,547][  INFO][PID:622344] Downloading the builder-live.log file, attempt 4
[2026-01-28 17:13:51,551][  INFO][PID:622344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c10a copr-rpmbuild-log
[2026-01-28 17:13:51,692][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:13:56,693][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:14:01,694][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:14:06,695][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:14:11,696][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:14:16,709][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:14:21,711][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:14:26,713][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:14:31,715][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:14:36,716][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:14:41,717][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:14:46,718][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:14:51,621][ ERROR][PID:622344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-28 17:14:51,719][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:14:56,720][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:15:01,637][ ERROR][PID:622344] Giving up for unstable SSH, failures: 1769620261.378033, 1769620341.458709, 1769620421.5469663, 1769620501.6367958
[2026-01-28 17:15:01,639][ ERROR][PID:622344] Re-try request for task on 'ResallocHost, ticket_id=1074235, hostname=2620:52:6:1161:dead:beef:cafe:c10a, name=vmhost_p09_02_prod_01709122_20260128_170210, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-01-28 17:15:01,640][  INFO][PID:622344] Releasing VM back to pool
[2026-01-28 17:15:01,656][  INFO][PID:622344] Retry #1 (on other host)
[2026-01-28 17:15:01,657][  INFO][PID:622344] Checking that /var/lib/copr/public_html/results/@meta/drgn/centos-stream+epel-next-9-ppc64le/repodata/repomd.xml exists
[2026-01-28 17:15:01,658][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:15:01,659][  INFO][PID:622344] VM allocation process starts
[2026-01-28 17:15:01,674][  INFO][PID:622344] Trying to allocate VM: ResallocHost, ticket_id=1074421, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2026-01-28 17:15:04,699][  INFO][PID:622344] Allocated host ResallocHost, ticket_id=1074421, hostname=2620:52:6:1161:dead:beef:cafe:c115, name=vmhost_p09_02_prod_01708685_20260128_165657, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2026-01-28 17:15:04,700][  INFO][PID:622344] Allocating ssh connection to builder
[2026-01-28 17:15:04,700][  INFO][PID:622344] Checking that builder machine is OK
[2026-01-28 17:16:04,777][ ERROR][PID:622344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c115 port 22: Connection timed out

[2026-01-28 17:16:14,779][ ERROR][PID:622344] Re-try request for task on 'ResallocHost, ticket_id=1074421, hostname=2620:52:6:1161:dead:beef:cafe:c115, name=vmhost_p09_02_prod_01708685_20260128_165657, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': Unable to finish after 1 SSH attempts
[2026-01-28 17:16:14,780][  INFO][PID:622344] Releasing VM back to pool
[2026-01-28 17:16:14,801][  INFO][PID:622344] Retry #2 (on other host)
[2026-01-28 17:16:14,802][  INFO][PID:622344] Checking that /var/lib/copr/public_html/results/@meta/drgn/centos-stream+epel-next-9-ppc64le/repodata/repomd.xml exists
[2026-01-28 17:16:14,805][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:16:14,806][  INFO][PID:622344] VM allocation process starts
[2026-01-28 17:16:14,823][  INFO][PID:622344] Trying to allocate VM: ResallocHost, ticket_id=1074451, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2026-01-28 17:16:17,848][  INFO][PID:622344] Allocated host ResallocHost, ticket_id=1074451, hostname=2620:52:6:1161:dead:beef:cafe:c108, name=vmhost_p09_02_prod_01709878_20260128_170954, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']
[2026-01-28 17:16:17,850][  INFO][PID:622344] Allocating ssh connection to builder
[2026-01-28 17:16:17,851][  INFO][PID:622344] Checking that builder machine is OK
[2026-01-28 17:16:19,077][  INFO][PID:622344] Running remote command: copr-builder-ready centos-stream+epel-next-9-ppc64le
[2026-01-28 17:16:19,679][  INFO][PID:622344] Red Hat subscription not needed for centos-stream+epel-next-9-ppc64le
Builder is ready to be used

[2026-01-28 17:16:19,680][  INFO][PID:622344] Filling build.info file with builder info
[2026-01-28 17:16:19,697][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:16:19,699][  INFO][PID:622344] Not re-notifying FE and msg buses for the new host.
[2026-01-28 17:16:19,700][  INFO][PID:622344] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10072774-centos-stream+epel-next-9-ppc64le --chroot centos-stream+epel-next-9-ppc64le --detached
[2026-01-28 17:16:20,694][  INFO][PID:622344] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c108' (ED25519) to the list of known hosts.


[2026-01-28 17:16:20,696][  INFO][PID:622344] Downloading the builder-live.log file, attempt 1
[2026-01-28 17:16:20,699][  INFO][PID:622344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c108 copr-rpmbuild-log
[2026-01-28 17:16:25,704][  INFO][PID:622344] Periodic builder liveness probe: alive
[2026-01-28 17:16:25,705][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:16:30,707][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:16:35,712][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:16:40,714][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:16:45,721][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:16:50,723][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:16:55,728][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:17:00,730][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:17:05,732][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:17:10,735][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:17:15,741][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:17:20,743][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:17:25,745][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:17:30,746][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:17:35,748][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:17:40,749][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:17:45,750][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:17:50,754][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:17:55,759][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:18:00,761][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:18:05,762][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:18:10,765][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:18:15,767][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:18:20,770][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:18:25,772][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:18:30,775][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:18:35,780][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:18:40,781][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:18:45,784][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:18:50,816][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:18:55,820][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:19:00,859][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:19:05,894][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:19:10,895][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:19:15,898][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:19:20,933][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:19:25,941][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:19:30,949][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:19:35,960][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:19:40,970][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:19:45,973][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:19:50,980][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:19:55,994][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:20:01,040][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:20:06,064][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:20:11,072][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:20:16,088][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:20:21,102][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:20:26,150][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:20:31,160][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:20:36,162][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:20:41,167][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:20:46,197][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:20:51,207][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:20:56,231][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:21:01,234][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:21:06,288][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:21:11,300][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:21:16,311][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:21:21,324][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:21:26,330][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:21:31,332][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:21:36,335][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:21:41,360][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:21:46,373][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:21:51,417][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:21:56,504][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:22:01,534][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:22:04,372][ ERROR][PID:622344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-28 17:22:06,590][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:22:11,636][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:22:14,373][WARNING][PID:622344] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-28 17:22:16,640][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:22:21,670][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:22:24,376][  INFO][PID:622344] Downloading the builder-live.log file, attempt 2
[2026-01-28 17:22:24,380][  INFO][PID:622344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c108 copr-rpmbuild-log
[2026-01-28 17:22:26,672][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:22:31,675][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:22:36,685][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:22:41,698][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:22:46,705][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:22:51,707][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:22:56,708][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:23:01,709][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:23:06,818][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:23:11,819][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:23:16,820][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:23:21,821][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:23:24,458][ ERROR][PID:622344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-28 17:23:26,823][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:23:31,960][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:23:34,469][WARNING][PID:622344] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-28 17:23:36,961][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:23:41,963][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:23:44,470][  INFO][PID:622344] Downloading the builder-live.log file, attempt 3
[2026-01-28 17:23:44,471][  INFO][PID:622344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c108 copr-rpmbuild-log
[2026-01-28 17:23:46,967][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:23:51,968][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:23:56,971][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:24:02,342][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:24:07,344][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:24:12,345][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:24:17,352][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:24:22,358][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:24:27,360][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:24:32,361][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:24:37,363][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:24:42,364][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:24:44,539][ ERROR][PID:622344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-28 17:24:47,367][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:24:52,369][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:24:54,547][WARNING][PID:622344] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-28 17:24:57,370][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:25:02,375][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:25:04,549][  INFO][PID:622344] Downloading the builder-live.log file, attempt 4
[2026-01-28 17:25:04,550][  INFO][PID:622344] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c108 copr-rpmbuild-log
[2026-01-28 17:25:07,377][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:25:12,383][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:25:17,385][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:25:22,399][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:25:27,401][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:25:32,402][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:25:37,470][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:25:42,478][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:25:47,482][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:25:52,486][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:25:57,491][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:26:02,497][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:26:04,618][ ERROR][PID:622344] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-28 17:26:07,499][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:26:12,501][  INFO][PID:622344] Checking for cancel request
[2026-01-28 17:26:14,628][ ERROR][PID:622344] Giving up for unstable SSH, failures: 1769620934.3736923, 1769621014.4698944, 1769621094.5477395, 1769621174.6283798
[2026-01-28 17:26:14,629][ ERROR][PID:622344] Re-try request for task on 'ResallocHost, ticket_id=1074451, hostname=2620:52:6:1161:dead:beef:cafe:c108, name=vmhost_p09_02_prod_01709878_20260128_170954, requested_tags=['copr_builder', 'arch_ppc64le', 'arch_power9']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-01-28 17:26:14,629][  INFO][PID:622344] Releasing VM back to pool
[2026-01-28 17:26:14,637][ ERROR][PID:622344] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c10a', '2620:52:6:1161:dead:beef:cafe:c115', '2620:52:6:1161:dead:beef:cafe:c108'}
[2026-01-28 17:26:14,637][WARNING][PID:622344] Switching not-finished job state to 'failed'
[2026-01-28 17:26:14,638][  INFO][PID:622344] Worker failed build, took 594.9387378692627
[2026-01-28 17:26:14,638][  INFO][PID:622344] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@meta",
            "project_name": "drgn",
            "project_dirname": "drgn",
            "submitter": "packit",
            "ended_on": 1769621174.6377845,
            "started_on": 1769620579.6990466,
            "submitted_on": null,
            "status": 0,
            "chroot": "centos-stream+epel-next-9-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "10072774-centos-stream+epel-next-9-ppc64le",
            "build_id": 10072774,
            "package_name": "python-drgn",
            "package_version": "0.0.33+158.g8adc2264-1.20260128165741536433.main.158.g8adc2264",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@meta/drgn/python-drgn",
            "git_hash": "392742313a0e3913c7e6ac46ee4b2441749392db",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@meta/drgn--packit",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@meta/drgn/centos-stream+epel-next-9-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://kojihub.stream.centos.org/kojifiles/repos/c9s-build/latest/ppc64le/",
                    "id": "https_kojihub_stream_centos_org_kojifiles_repos_c9s_build_latest_ppc64le",
                    "name": "Additional repo https_kojihub_stream_centos_org_kojifiles_repos_c9s_build_latest_ppc64le"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le",
                "arch_power9"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@meta/drgn",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@meta/drgn",
            "result_dir": "10072774-python-drgn",
            "built_packages": "",
            "id": 10072774,
            "mockchain_macros": {
                "copr_username": "@meta",
                "copr_projectname": "drgn",
                "vendor": "Fedora Project COPR (@meta/drgn)"
            }
        }
    ]
}
[2026-01-28 17:26:14,697][  INFO][PID:622344] Sending fedora-messaging bus message in build.end
[2026-01-28 17:26:14,724][  INFO][PID:622344] Compressing /var/lib/copr/public_html/results/@meta/drgn/centos-stream+epel-next-9-ppc64le/10072774-python-drgn/builder-live.log by gzip
[2026-01-28 17:26:14,725][  INFO][PID:622344] Running command 'gzip /var/lib/copr/public_html/results/@meta/drgn/centos-stream+epel-next-9-ppc64le/10072774-python-drgn/builder-live.log' as PID 783489
[2026-01-28 17:26:14,727][  INFO][PID:622344] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@meta/drgn/centos-stream+epel-next-9-ppc64le/10072774-python-drgn/builder-live.log)
[2026-01-28 17:26:14,727][  INFO][PID:622344] Compressing /var/lib/copr/public_html/results/@meta/drgn/centos-stream+epel-next-9-ppc64le/10072774-python-drgn/backend.log by gzip
[2026-01-28 17:26:14,728][  INFO][PID:622344] Running command 'gzip /var/lib/copr/public_html/results/@meta/drgn/centos-stream+epel-next-9-ppc64le/10072774-python-drgn/backend.log' as PID 783490