[2026-01-01 08:55:12,487][  INFO][PID:2298795] Marking build as starting
[2026-01-01 08:55:13,003][  INFO][PID:2298795] Checking that /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/repodata/repomd.xml exists
[2026-01-01 08:55:13,004][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:55:13,005][  INFO][PID:2298795] VM allocation process starts
[2026-01-01 08:55:13,019][  INFO][PID:2298795] Trying to allocate VM: ResallocHost, ticket_id=609562, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-01 08:55:16,038][  INFO][PID:2298795] Allocated host ResallocHost, ticket_id=609562, hostname=2620:52:6:1161:dead:beef:cafe:c149, name=vmhost_p09_03_prod_00879541_20260101_084147, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-01 08:55:16,039][  INFO][PID:2298795] Allocating ssh connection to builder
[2026-01-01 08:55:16,039][  INFO][PID:2298795] Checking that builder machine is OK
[2026-01-01 08:56:16,110][ ERROR][PID:2298795] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c149 port 22: Connection timed out

[2026-01-01 08:56:26,111][ ERROR][PID:2298795] Re-try request for task on 'ResallocHost, ticket_id=609562, hostname=2620:52:6:1161:dead:beef:cafe:c149, name=vmhost_p09_03_prod_00879541_20260101_084147, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2026-01-01 08:56:26,111][  INFO][PID:2298795] Releasing VM back to pool
[2026-01-01 08:56:26,120][  INFO][PID:2298795] Retry #1 (on other host)
[2026-01-01 08:56:26,121][  INFO][PID:2298795] Checking that /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/repodata/repomd.xml exists
[2026-01-01 08:56:26,121][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:56:26,122][  INFO][PID:2298795] VM allocation process starts
[2026-01-01 08:56:26,177][  INFO][PID:2298795] Trying to allocate VM: ResallocHost, ticket_id=609585, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-01 08:56:29,256][  INFO][PID:2298795] Allocated host ResallocHost, ticket_id=609585, hostname=2620:52:6:1161:dead:beef:cafe:c18a, name=vmhost_p09_04_prod_00879553_20260101_084331, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-01 08:56:29,256][  INFO][PID:2298795] Allocating ssh connection to builder
[2026-01-01 08:56:29,257][  INFO][PID:2298795] Checking that builder machine is OK
[2026-01-01 08:56:30,197][  INFO][PID:2298795] Running remote command: copr-builder-ready fedora-42-ppc64le
[2026-01-01 08:56:30,724][  INFO][PID:2298795] Red Hat subscription not needed for fedora-42-ppc64le
Builder is ready to be used

[2026-01-01 08:56:30,725][  INFO][PID:2298795] Filling build.info file with builder info
[2026-01-01 08:56:30,727][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:56:30,727][  INFO][PID:2298795] Not re-notifying FE and msg buses for the new host.
[2026-01-01 08:56:30,728][  INFO][PID:2298795] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9967873-fedora-42-ppc64le --chroot fedora-42-ppc64le --detached
[2026-01-01 08:57:30,797][ ERROR][PID:2298795] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c18a port 22: Connection timed out

[2026-01-01 08:57:40,799][ ERROR][PID:2298795] Re-try request for task on 'ResallocHost, ticket_id=609585, hostname=2620:52:6:1161:dead:beef:cafe:c18a, name=vmhost_p09_04_prod_00879553_20260101_084331, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2026-01-01 08:57:40,800][  INFO][PID:2298795] Releasing VM back to pool
[2026-01-01 08:57:40,813][  INFO][PID:2298795] Retry #2 (on other host)
[2026-01-01 08:57:40,814][  INFO][PID:2298795] Checking that /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/repodata/repomd.xml exists
[2026-01-01 08:57:40,814][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:57:40,815][  INFO][PID:2298795] VM allocation process starts
[2026-01-01 08:57:40,827][  INFO][PID:2298795] Trying to allocate VM: ResallocHost, ticket_id=609603, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-01 08:57:43,855][  INFO][PID:2298795] Allocated host ResallocHost, ticket_id=609603, hostname=2620:52:6:1161:dead:beef:cafe:c148, name=vmhost_p09_03_prod_00879584_20260101_085110, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-01 08:57:43,863][  INFO][PID:2298795] Allocating ssh connection to builder
[2026-01-01 08:57:43,866][  INFO][PID:2298795] Checking that builder machine is OK
[2026-01-01 08:57:44,802][  INFO][PID:2298795] Running remote command: copr-builder-ready fedora-42-ppc64le
[2026-01-01 08:57:45,279][  INFO][PID:2298795] Red Hat subscription not needed for fedora-42-ppc64le
Builder is ready to be used

[2026-01-01 08:57:45,279][  INFO][PID:2298795] Filling build.info file with builder info
[2026-01-01 08:57:45,291][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:57:45,295][  INFO][PID:2298795] Not re-notifying FE and msg buses for the new host.
[2026-01-01 08:57:45,296][  INFO][PID:2298795] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9967873-fedora-42-ppc64le --chroot fedora-42-ppc64le --detached
[2026-01-01 08:57:46,245][  INFO][PID:2298795] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c148' (ED25519) to the list of known hosts.


[2026-01-01 08:57:46,246][  INFO][PID:2298795] Downloading the builder-live.log file, attempt 1
[2026-01-01 08:57:46,254][  INFO][PID:2298795] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c148 copr-rpmbuild-log
[2026-01-01 08:57:51,276][  INFO][PID:2298795] Periodic builder liveness probe: alive
[2026-01-01 08:57:51,278][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:57:56,280][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:58:01,284][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:58:06,286][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:58:11,287][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:58:16,290][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:58:21,292][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:58:26,293][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:58:31,295][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:58:36,298][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:58:41,300][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:58:46,302][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:58:51,303][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:58:56,305][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:59:01,306][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:59:06,307][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:59:11,309][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:59:16,311][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:59:21,312][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:59:26,313][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:59:31,314][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:59:36,316][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:59:41,318][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:59:46,319][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:59:51,320][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 08:59:56,321][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:00:01,327][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:00:06,328][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:00:11,332][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:00:16,334][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:00:21,337][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:00:26,338][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:00:31,339][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:00:36,341][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:00:41,342][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:00:46,343][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:00:51,344][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:00:56,345][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:01:01,346][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:01:06,347][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:01:11,348][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:01:16,349][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:01:21,351][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:01:26,358][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:01:31,360][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:01:36,361][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:01:41,362][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:01:46,365][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:01:51,366][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:01:56,368][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:02:01,373][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:02:06,375][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:02:11,376][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:02:16,377][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:02:21,378][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:02:26,379][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:02:31,380][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:02:36,382][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:02:41,383][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:02:46,385][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:02:51,387][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:02:56,388][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:03:01,389][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:03:06,390][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:03:11,392][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:03:16,396][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:03:21,398][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:03:26,399][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:03:31,400][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:03:36,403][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:03:41,404][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:03:46,405][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:03:51,407][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:03:56,408][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:04:01,409][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:04:06,411][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:04:11,412][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:04:16,413][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:04:21,415][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:04:26,419][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:04:31,420][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:04:36,421][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:04:41,422][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:04:46,424][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:04:51,425][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:04:56,426][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:05:01,427][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:05:06,428][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:05:11,430][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:05:16,431][  INFO][PID:2298795] Checking for cancel request
[2026-01-01 09:05:19,567][  INFO][PID:2298795] Downloading results from builder
[2026-01-01 09:05:19,567][  INFO][PID:2298795] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c148]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty started
[2026-01-01 09:05:19,568][  INFO][PID:2298795] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c148]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty/ &> /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty/build-09967873.rsync.log
[2026-01-01 09:06:19,644][  INFO][PID:2298795] rsyncing finished.
[2026-01-01 09:06:19,645][ ERROR][PID:2298795] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-01 09:06:19,645][ ERROR][PID:2298795] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-01 09:06:29,652][  INFO][PID:2298795] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c148]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty started
[2026-01-01 09:06:29,654][  INFO][PID:2298795] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c148]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty/ &> /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty/build-09967873.rsync.log
[2026-01-01 09:07:29,782][  INFO][PID:2298795] rsyncing finished.
[2026-01-01 09:07:29,803][ ERROR][PID:2298795] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-01 09:07:29,804][ ERROR][PID:2298795] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-01 09:07:39,804][  INFO][PID:2298795] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c148]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty started
[2026-01-01 09:07:39,805][  INFO][PID:2298795] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c148]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty/ &> /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty/build-09967873.rsync.log
[2026-01-01 09:08:39,893][  INFO][PID:2298795] rsyncing finished.
[2026-01-01 09:08:39,915][ ERROR][PID:2298795] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-01 09:08:39,916][ ERROR][PID:2298795] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-01 09:08:49,918][ ERROR][PID:2298795] Re-try request for task on 'ResallocHost, ticket_id=609603, hostname=2620:52:6:1161:dead:beef:cafe:c148, name=vmhost_p09_03_prod_00879584_20260101_085110, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 3 SSH attempts
[2026-01-01 09:08:49,918][  INFO][PID:2298795] Releasing VM back to pool
[2026-01-01 09:08:49,941][ ERROR][PID:2298795] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c18a', '2620:52:6:1161:dead:beef:cafe:c149', '2620:52:6:1161:dead:beef:cafe:c148'}
[2026-01-01 09:08:49,945][WARNING][PID:2298795] Switching not-finished job state to 'failed'
[2026-01-01 09:08:49,950][  INFO][PID:2298795] Worker failed build, took 664.6472411155701
[2026-01-01 09:08:49,951][  INFO][PID:2298795] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "topazus",
            "project_name": "test",
            "project_dirname": "test",
            "submitter": "topazus",
            "ended_on": 1767258529.9425952,
            "started_on": 1767257865.2953541,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-42-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9967873-fedora-42-ppc64le",
            "build_id": 9967873,
            "package_name": "python-duty",
            "package_version": "1.6.3-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/topazus/test/python-duty",
            "git_hash": "ab23b88dfce871f1d26627cd344acbc597513bc2",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "topazus/test--topazus",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test/fedora-42-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test/fedora-42-ppc64le/",
                    "id": "copr_topazus_test",
                    "name": "Additional repo copr_topazus_test"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/topazus/test",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/topazus/test",
            "result_dir": "09967873-python-duty",
            "built_packages": "",
            "id": 9967873,
            "mockchain_macros": {
                "copr_username": "topazus",
                "copr_projectname": "test",
                "vendor": "Fedora Project COPR (topazus/test)"
            }
        }
    ]
}
[2026-01-01 09:08:50,084][  INFO][PID:2298795] Sending fedora-messaging bus message in build.end
[2026-01-01 09:08:50,807][  INFO][PID:2298795] Compressing /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty/builder-live.log by gzip
[2026-01-01 09:08:50,809][  INFO][PID:2298795] Running command 'gzip /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty/builder-live.log' as PID 2387849
[2026-01-01 09:08:50,896][  INFO][PID:2298795] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty/builder-live.log)
[2026-01-01 09:08:50,899][  INFO][PID:2298795] Compressing /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty/backend.log by gzip
[2026-01-01 09:08:50,902][  INFO][PID:2298795] Running command 'gzip /var/lib/copr/public_html/results/topazus/test/fedora-42-ppc64le/09967873-python-duty/backend.log' as PID 2387862