[2026-01-02 08:34:12,508][  INFO][PID:4146414] Marking build as starting
[2026-01-02 08:34:12,569][  INFO][PID:4146414] Checking that /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/repodata/repomd.xml exists
[2026-01-02 08:34:12,570][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:34:12,571][  INFO][PID:4146414] VM allocation process starts
[2026-01-02 08:34:12,585][  INFO][PID:4146414] Trying to allocate VM: ResallocHost, ticket_id=624838, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-02 08:34:15,603][  INFO][PID:4146414] Allocated host ResallocHost, ticket_id=624838, hostname=2620:52:6:1161:dead:beef:cafe:c151, name=vmhost_p09_03_prod_00889090_20260102_082307, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-02 08:34:15,603][  INFO][PID:4146414] Allocating ssh connection to builder
[2026-01-02 08:34:15,604][  INFO][PID:4146414] Checking that builder machine is OK
[2026-01-02 08:34:16,657][  INFO][PID:4146414] Running remote command: copr-builder-ready fedora-41-ppc64le
[2026-01-02 08:34:17,218][  INFO][PID:4146414] Red Hat subscription not needed for fedora-41-ppc64le
Builder is ready to be used

[2026-01-02 08:34:17,218][  INFO][PID:4146414] Filling build.info file with builder info
[2026-01-02 08:34:17,219][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:34:17,220][  INFO][PID:4146414] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "topazus",
            "project_name": "test",
            "project_dirname": "test",
            "submitter": "topazus",
            "ended_on": null,
            "started_on": 1767342857.220106,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-41-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9969575-fedora-41-ppc64le",
            "build_id": 9969575,
            "package_name": "f3d",
            "package_version": "3.3.0-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/topazus/test/f3d",
            "git_hash": "ed0c4b611e2f4cd4e3a5208c2d4e05dae93de283",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "topazus/test--topazus",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test/fedora-41-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test/fedora-41-ppc64le/",
                    "id": "copr_topazus_test",
                    "name": "Additional repo copr_topazus_test"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/topazus/test",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/topazus/test",
            "result_dir": "09969575-f3d",
            "built_packages": "",
            "id": 9969575,
            "mockchain_macros": {
                "copr_username": "topazus",
                "copr_projectname": "test",
                "vendor": "Fedora Project COPR (topazus/test)"
            }
        }
    ]
}
[2026-01-02 08:34:17,279][  INFO][PID:4146414] Sending fedora-messaging bus message in build.start
[2026-01-02 08:34:17,832][  INFO][PID:4146414] Sending fedora-messaging bus message in chroot.start
[2026-01-02 08:34:17,859][  INFO][PID:4146414] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9969575-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached
[2026-01-02 08:34:18,813][  INFO][PID:4146414] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c151' (ED25519) to the list of known hosts.


[2026-01-02 08:34:18,881][  INFO][PID:4146414] Downloading the builder-live.log file, attempt 1
[2026-01-02 08:34:18,882][  INFO][PID:4146414] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c151 copr-rpmbuild-log
[2026-01-02 08:34:23,886][  INFO][PID:4146414] Periodic builder liveness probe: alive
[2026-01-02 08:34:23,887][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:34:28,887][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:34:33,888][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:34:38,889][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:34:43,890][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:34:48,891][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:34:53,919][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:34:58,921][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:35:03,922][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:35:08,923][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:35:13,924][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:35:18,925][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:35:23,926][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:35:28,927][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:35:33,928][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:35:38,929][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:35:43,930][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:35:48,931][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:35:53,935][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:35:58,936][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:36:03,937][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:36:08,953][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:36:13,954][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:36:18,955][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:36:23,957][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:36:28,962][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:36:33,963][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:36:38,983][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:36:43,984][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:36:48,985][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:36:53,986][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:36:58,987][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:37:03,988][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:37:08,988][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:37:14,072][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:37:19,073][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:37:24,074][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:37:29,075][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:37:34,076][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:37:39,078][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:37:44,079][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:37:49,095][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:37:54,095][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:37:59,096][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:38:04,097][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:38:09,098][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:38:14,299][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:38:19,300][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:38:24,301][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:38:29,302][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:38:34,303][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:38:39,304][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:38:44,307][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:38:49,308][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:38:54,309][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:38:59,311][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:39:04,312][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:39:09,313][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:39:14,314][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:39:19,315][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:39:24,316][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:39:29,318][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:39:34,319][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:39:39,320][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:39:44,321][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:39:49,322][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:39:54,323][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:39:59,324][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:40:04,325][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:40:09,326][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:40:14,327][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:40:19,327][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:40:24,328][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:40:29,329][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:40:34,330][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:40:39,331][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:40:44,332][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:40:49,333][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:40:54,334][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:40:59,335][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:41:04,336][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:41:09,337][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:41:14,338][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:41:19,339][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:41:24,340][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:41:29,341][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:41:34,342][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:41:39,343][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:41:44,345][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:41:49,346][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:41:54,347][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:41:59,348][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:42:04,349][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:42:09,350][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:42:14,351][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:42:19,352][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:42:24,354][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:42:29,354][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:42:34,355][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:42:39,356][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:42:44,357][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:42:49,358][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:42:54,359][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:42:59,360][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:43:04,361][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:43:09,362][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:43:14,363][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:43:19,364][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:43:24,365][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:43:29,366][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:43:34,367][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:43:39,368][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:43:44,370][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:43:49,371][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:43:54,372][  INFO][PID:4146414] Checking for cancel request
[2026-01-02 08:43:56,441][  INFO][PID:4146414] Downloading results from builder
[2026-01-02 08:43:56,441][  INFO][PID:4146414] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c151]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d started
[2026-01-02 08:43:56,442][  INFO][PID:4146414] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c151]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d/ &> /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d/build-09969575.rsync.log
[2026-01-02 08:44:56,509][  INFO][PID:4146414] rsyncing finished.
[2026-01-02 08:44:56,510][ ERROR][PID:4146414] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-02 08:44:56,510][ ERROR][PID:4146414] SSH connection lost on #1 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-02 08:45:06,511][  INFO][PID:4146414] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c151]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d started
[2026-01-02 08:45:06,512][  INFO][PID:4146414] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c151]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d/ &> /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d/build-09969575.rsync.log
[2026-01-02 08:46:06,713][  INFO][PID:4146414] rsyncing finished.
[2026-01-02 08:46:06,790][ ERROR][PID:4146414] Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-02 08:46:06,791][ ERROR][PID:4146414] SSH connection lost on #2 attempt, let's retry after 10s, Failed to download data from builder due to rsync error, see the rsync log file for details.
[2026-01-02 08:46:16,792][  INFO][PID:4146414] rsyncing of mockbuilder@[2620:52:6:1161:dead:beef:cafe:c151]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d started
[2026-01-02 08:46:16,793][  INFO][PID:4146414] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:6:1161:dead:beef:cafe:c151]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d/ &> /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d/build-09969575.rsync.log
[2026-01-02 08:46:18,055][  INFO][PID:4146414] rsyncing finished.
[2026-01-02 08:46:18,056][  INFO][PID:4146414] Releasing VM back to pool
[2026-01-02 08:46:18,064][  INFO][PID:4146414] Searching for 'success' file in resultdir
[2026-01-02 08:46:18,065][ ERROR][PID:4146414] Build failed: Backend process error: No success file => build failure
[2026-01-02 08:46:18,065][  INFO][PID:4146414] Finished build: id=9969575 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/topazus/test chroot=fedora-41-ppc64le 
[2026-01-02 08:46:18,067][ ERROR][PID:4146414] Backend process error: No success file => build failure
[2026-01-02 08:46:18,067][  INFO][PID:4146414] Worker failed build, took 720.8473708629608
[2026-01-02 08:46:18,067][  INFO][PID:4146414] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "topazus",
            "project_name": "test",
            "project_dirname": "test",
            "submitter": "topazus",
            "ended_on": 1767343578.0674767,
            "started_on": 1767342857.220106,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-41-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9969575-fedora-41-ppc64le",
            "build_id": 9969575,
            "package_name": "f3d",
            "package_version": "3.3.0-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/topazus/test/f3d",
            "git_hash": "ed0c4b611e2f4cd4e3a5208c2d4e05dae93de283",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "topazus/test--topazus",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test/fedora-41-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/topazus/test/fedora-41-ppc64le/",
                    "id": "copr_topazus_test",
                    "name": "Additional repo copr_topazus_test"
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/topazus/test",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/topazus/test",
            "result_dir": "09969575-f3d",
            "built_packages": "",
            "id": 9969575,
            "mockchain_macros": {
                "copr_username": "topazus",
                "copr_projectname": "test",
                "vendor": "Fedora Project COPR (topazus/test)"
            }
        }
    ]
}
[2026-01-02 08:46:18,128][  INFO][PID:4146414] Sending fedora-messaging bus message in build.end
[2026-01-02 08:46:18,158][  INFO][PID:4146414] Compressing /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d/builder-live.log by gzip
[2026-01-02 08:46:18,159][  INFO][PID:4146414] Running command 'gzip /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d/builder-live.log' as PID 6108
[2026-01-02 08:46:18,191][  INFO][PID:4146414] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d/builder-live.log)
[2026-01-02 08:46:18,192][  INFO][PID:4146414] Compressing /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d/backend.log by gzip
[2026-01-02 08:46:18,193][  INFO][PID:4146414] Running command 'gzip /var/lib/copr/public_html/results/topazus/test/fedora-41-ppc64le/09969575-f3d/backend.log' as PID 6109