[2026-01-03 12:51:50,955][  INFO][PID:2293504] Marking build as starting
[2026-01-03 12:51:51,069][  INFO][PID:2293504] Checking that /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2026-01-03 12:51:51,069][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:51:51,070][  INFO][PID:2293504] VM allocation process starts
[2026-01-03 12:51:51,080][  INFO][PID:2293504] Trying to allocate VM: ResallocHost, ticket_id=639850, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-03 12:51:54,091][  INFO][PID:2293504] Allocated host ResallocHost, ticket_id=639850, hostname=2620:52:6:1161:dead:beef:cafe:c14b, name=vmhost_p09_03_prod_00903692_20260103_123823, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-03 12:51:54,092][  INFO][PID:2293504] Allocating ssh connection to builder
[2026-01-03 12:51:54,092][  INFO][PID:2293504] Checking that builder machine is OK
[2026-01-03 12:51:55,112][  INFO][PID:2293504] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2026-01-03 12:51:55,641][  INFO][PID:2293504] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2026-01-03 12:51:55,642][  INFO][PID:2293504] Filling build.info file with builder info
[2026-01-03 12:51:55,644][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:51:55,646][  INFO][PID:2293504] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "topazus",
            "project_name": "test",
            "project_dirname": "test",
            "submitter": "topazus",
            "ended_on": null,
            "started_on": 1767444715.6460836,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9971827-fedora-rawhide-ppc64le",
            "build_id": 9971827,
            "package_name": "deepin-util-dfm",
            "package_version": "1.3.43-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/topazus/test/deepin-util-dfm",
            "git_hash": "e841e8fae5456731f1699e146916e042ac3911ed",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "topazus/test--topazus",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test/fedora-rawhide-ppc64le/",
                    "id": "copr_topazus_test",
                    "name": "Additional repo copr_topazus_test"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/topazus/test",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/topazus/test",
            "result_dir": "09971827-deepin-util-dfm",
            "built_packages": "",
            "id": 9971827,
            "mockchain_macros": {
                "copr_username": "topazus",
                "copr_projectname": "test",
                "vendor": "Fedora Project COPR (topazus/test)"
            }
        }
    ]
}
[2026-01-03 12:51:55,759][  INFO][PID:2293504] Sending fedora-messaging bus message in build.start
[2026-01-03 12:51:56,324][  INFO][PID:2293504] Sending fedora-messaging bus message in chroot.start
[2026-01-03 12:51:56,352][  INFO][PID:2293504] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9971827-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2026-01-03 12:52:56,423][ ERROR][PID:2293504] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c14b port 22: Connection timed out

[2026-01-03 12:53:06,423][ ERROR][PID:2293504] Re-try request for task on 'ResallocHost, ticket_id=639850, hostname=2620:52:6:1161:dead:beef:cafe:c14b, name=vmhost_p09_03_prod_00903692_20260103_123823, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2026-01-03 12:53:06,424][  INFO][PID:2293504] Releasing VM back to pool
[2026-01-03 12:53:06,433][  INFO][PID:2293504] Retry #1 (on other host)
[2026-01-03 12:53:06,433][  INFO][PID:2293504] Checking that /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2026-01-03 12:53:06,434][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:53:06,434][  INFO][PID:2293504] VM allocation process starts
[2026-01-03 12:53:06,495][  INFO][PID:2293504] Trying to allocate VM: ResallocHost, ticket_id=639860, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-03 12:53:09,531][  INFO][PID:2293504] Allocated host ResallocHost, ticket_id=639860, hostname=2620:52:6:1161:dead:beef:cafe:c14c, name=vmhost_p09_03_prod_00903672_20260103_123355, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-03 12:53:09,531][  INFO][PID:2293504] Allocating ssh connection to builder
[2026-01-03 12:53:09,532][  INFO][PID:2293504] Checking that builder machine is OK
[2026-01-03 12:53:09,564][ ERROR][PID:2293504] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c14c port 22: Permission denied

[2026-01-03 12:53:19,565][ ERROR][PID:2293504] Re-try request for task on 'ResallocHost, ticket_id=639860, hostname=2620:52:6:1161:dead:beef:cafe:c14c, name=vmhost_p09_03_prod_00903672_20260103_123355, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2026-01-03 12:53:19,565][  INFO][PID:2293504] Releasing VM back to pool
[2026-01-03 12:53:19,573][  INFO][PID:2293504] Retry #2 (on other host)
[2026-01-03 12:53:19,574][  INFO][PID:2293504] Checking that /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2026-01-03 12:53:19,574][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:53:19,575][  INFO][PID:2293504] VM allocation process starts
[2026-01-03 12:53:19,621][  INFO][PID:2293504] Trying to allocate VM: ResallocHost, ticket_id=639862, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-03 12:53:22,652][  INFO][PID:2293504] Allocated host ResallocHost, ticket_id=639862, hostname=2620:52:6:1161:dead:beef:cafe:c150, name=vmhost_p09_03_prod_00903698_20260103_124026, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-03 12:53:22,652][  INFO][PID:2293504] Allocating ssh connection to builder
[2026-01-03 12:53:22,653][  INFO][PID:2293504] Checking that builder machine is OK
[2026-01-03 12:53:23,549][  INFO][PID:2293504] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2026-01-03 12:53:24,014][  INFO][PID:2293504] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2026-01-03 12:53:24,015][  INFO][PID:2293504] Filling build.info file with builder info
[2026-01-03 12:53:24,024][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:53:24,024][  INFO][PID:2293504] Not re-notifying FE and msg buses for the new host.
[2026-01-03 12:53:24,025][  INFO][PID:2293504] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9971827-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2026-01-03 12:53:25,029][  INFO][PID:2293504] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c150' (ED25519) to the list of known hosts.


[2026-01-03 12:53:25,029][  INFO][PID:2293504] Downloading the builder-live.log file, attempt 1
[2026-01-03 12:53:25,031][  INFO][PID:2293504] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c150 copr-rpmbuild-log
[2026-01-03 12:53:30,034][  INFO][PID:2293504] Periodic builder liveness probe: alive
[2026-01-03 12:53:30,035][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:53:35,035][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:53:40,036][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:53:45,037][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:53:50,038][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:53:55,039][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:54:00,040][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:54:05,041][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:54:10,042][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:54:15,043][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:54:20,044][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:54:25,046][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:54:30,047][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:54:35,048][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:54:40,049][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:54:45,049][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:54:50,050][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:54:55,051][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:55:00,052][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:55:05,053][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:55:10,054][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:55:15,055][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:55:20,056][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:55:25,057][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:55:30,058][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:55:35,059][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:55:40,060][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:55:45,061][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:55:50,062][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:55:55,063][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:56:00,064][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:56:05,065][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:56:10,066][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:56:15,067][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:56:20,068][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:56:25,069][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:56:30,070][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:56:35,071][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:56:40,072][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:56:45,072][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:56:50,073][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:56:55,074][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:57:00,075][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:57:05,077][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:57:10,078][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:57:15,079][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:57:20,080][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:57:25,081][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:57:30,082][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:57:35,083][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:57:40,084][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:57:45,085][  INFO][PID:2293504] Checking for cancel request
[2026-01-03 12:57:49,313][  INFO][PID:2293504] Downloading results from builder
[2026-01-03 12:57:49,314][  INFO][PID:2293504] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c150]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm started
[2026-01-03 12:57:49,315][  INFO][PID:2293504] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c150]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm/ &> /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm/build-09971827.rsync.log
[2026-01-03 12:58:49,370][  INFO][PID:2293504] rsyncing finished.
[2026-01-03 12:58:49,371][ ERROR][PID:2293504] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-03 12:58:49,371][ ERROR][PID:2293504] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-03 12:58:59,372][  INFO][PID:2293504] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c150]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm started
[2026-01-03 12:58:59,373][  INFO][PID:2293504] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c150]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm/ &> /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm/build-09971827.rsync.log
[2026-01-03 12:59:59,455][  INFO][PID:2293504] rsyncing finished.
[2026-01-03 12:59:59,456][ ERROR][PID:2293504] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-03 12:59:59,456][ ERROR][PID:2293504] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-03 13:00:09,456][  INFO][PID:2293504] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c150]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm started
[2026-01-03 13:00:09,457][  INFO][PID:2293504] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c150]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm/ &> /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm/build-09971827.rsync.log
[2026-01-03 13:01:09,536][  INFO][PID:2293504] rsyncing finished.
[2026-01-03 13:01:09,555][ ERROR][PID:2293504] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-03 13:01:09,555][ ERROR][PID:2293504] SSH connection lost on #3 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-03 13:01:19,556][ ERROR][PID:2293504] Re-try request for task on 'ResallocHost, ticket_id=639862, hostname=2620:52:6:1161:dead:beef:cafe:c150, name=vmhost_p09_03_prod_00903698_20260103_124026, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 3 SSH attempts
[2026-01-03 13:01:19,556][  INFO][PID:2293504] Releasing VM back to pool
[2026-01-03 13:01:19,565][ ERROR][PID:2293504] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c14b', '2620:52:6:1161:dead:beef:cafe:c150', '2620:52:6:1161:dead:beef:cafe:c14c'}
[2026-01-03 13:01:19,566][WARNING][PID:2293504] Switching not-finished job state to 'failed'
[2026-01-03 13:01:19,566][  INFO][PID:2293504] Worker failed build, took 475.54155111312866
[2026-01-03 13:01:19,567][  INFO][PID:2293504] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "topazus",
            "project_name": "test",
            "project_dirname": "test",
            "submitter": "topazus",
            "ended_on": 1767445279.5661945,
            "started_on": 1767444804.0246434,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9971827-fedora-rawhide-ppc64le",
            "build_id": 9971827,
            "package_name": "deepin-util-dfm",
            "package_version": "1.3.43-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/topazus/test/deepin-util-dfm",
            "git_hash": "e841e8fae5456731f1699e146916e042ac3911ed",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "topazus/test--topazus",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test/fedora-rawhide-ppc64le/",
                    "id": "copr_topazus_test",
                    "name": "Additional repo copr_topazus_test"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/topazus/test",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/topazus/test",
            "result_dir": "09971827-deepin-util-dfm",
            "built_packages": "",
            "id": 9971827,
            "mockchain_macros": {
                "copr_username": "topazus",
                "copr_projectname": "test",
                "vendor": "Fedora Project COPR (topazus/test)"
            }
        }
    ]
}
[2026-01-03 13:01:19,627][  INFO][PID:2293504] Sending fedora-messaging bus message in build.end
[2026-01-03 13:01:19,654][  INFO][PID:2293504] Compressing /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm/builder-live.log by gzip
[2026-01-03 13:01:19,655][  INFO][PID:2293504] Running command 'gzip /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm/builder-live.log' as PID 2330887
[2026-01-03 13:01:19,703][  INFO][PID:2293504] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm/builder-live.log)
[2026-01-03 13:01:19,704][  INFO][PID:2293504] Compressing /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm/backend.log by gzip
[2026-01-03 13:01:19,705][  INFO][PID:2293504] Running command 'gzip /var/lib/copr/public_html/results/topazus/test/fedora-rawhide-ppc64le/09971827-deepin-util-dfm/backend.log' as PID 2330893