[2025-03-24 11:30:04,357][  INFO][PID:1958889] Marking build as starting
[2025-03-24 11:30:04,460][  INFO][PID:1958889] Trying to allocate VM
[2025-03-24 11:30:07,533][  INFO][PID:1958889] Allocated host ResallocHost, ticket_id=2003640, hostname=2620:52:3:1:dead:beef:cafe:c294, name=copr_p09_01_prod_02246324_20250324_112103
[2025-03-24 11:30:07,534][  INFO][PID:1958889] Allocating ssh connection to builder
[2025-03-24 11:30:07,534][  INFO][PID:1958889] Checking that builder machine is OK
[2025-03-24 11:30:08,441][  INFO][PID:1958889] Running remote command: copr-builder-ready fedora-eln-ppc64le
[2025-03-24 11:30:08,944][  INFO][PID:1958889] Red Hat subscription not needed for fedora-eln-ppc64le
Builder is ready to be used

[2025-03-24 11:30:08,946][  INFO][PID:1958889] Filling build.info file with builder info
[2025-03-24 11:30:08,947][  INFO][PID:1958889] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "rhcontainerbot",
            "project_name": "podman-next",
            "project_dirname": "podman-next",
            "submitter": "packit",
            "ended_on": null,
            "started_on": 1742815808.9476526,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-eln-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8812228-fedora-eln-ppc64le",
            "build_id": 8812228,
            "package_name": "netavark",
            "package_version": "102:1.14.0-1.20250324112914455606.main.52.g73a9eea",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/podman-next/netavark",
            "git_hash": "14036c502dd7fe7ec794eb6509df653d949e511f",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "rhcontainerbot/podman-next--packit",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": false,
            "bootstrap": "on",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/podman-next/fedora-eln-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://kojipkgs.fedoraproject.org/repos/eln-build/latest/$basearch/",
                    "id": "https_kojipkgs_fedoraproject_org_repos_eln_build_latest_basearch",
                    "name": "Additional repo https_kojipkgs_fedoraproject_org_repos_eln_build_latest_basearch"
                }
            ],
            "tags": [
                "arch_ppc64le",
                "arch_power9"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/rhcontainerbot/podman-next",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/podman-next",
            "result_dir": "08812228-netavark",
            "built_packages": "",
            "id": 8812228,
            "mockchain_macros": {
                "copr_username": "rhcontainerbot",
                "copr_projectname": "podman-next",
                "vendor": "Fedora Project COPR (rhcontainerbot/podman-next)"
            }
        }
    ]
}
[2025-03-24 11:30:09,028][  INFO][PID:1958889] Sending fedora-messaging bus message in build.start
[2025-03-24 11:30:09,749][  INFO][PID:1958889] Sending fedora-messaging bus message in chroot.start
[2025-03-24 11:30:09,769][  INFO][PID:1958889] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8812228-fedora-eln-ppc64le --chroot fedora-eln-ppc64le --detached
[2025-03-24 11:30:10,924][  INFO][PID:1958889] Downloading the builder-live.log file, attempt 1
[2025-03-24 11:30:10,927][  INFO][PID:1958889] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c294 copr-rpmbuild-log
[2025-03-24 11:30:15,932][  INFO][PID:1958889] Periodic builder liveness probe: alive
[2025-03-24 11:34:26,229][ ERROR][PID:1958889] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-24 11:34:36,232][WARNING][PID:1958889] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-24 11:34:46,233][  INFO][PID:1958889] Downloading the builder-live.log file, attempt 2
[2025-03-24 11:34:46,235][  INFO][PID:1958889] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c294 copr-rpmbuild-log
[2025-03-24 11:35:46,304][ ERROR][PID:1958889] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-24 11:35:56,309][WARNING][PID:1958889] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-24 11:36:06,310][  INFO][PID:1958889] Downloading the builder-live.log file, attempt 3
[2025-03-24 11:36:06,311][  INFO][PID:1958889] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c294 copr-rpmbuild-log
[2025-03-24 11:37:06,379][ ERROR][PID:1958889] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-24 11:37:16,380][WARNING][PID:1958889] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-24 11:37:26,384][  INFO][PID:1958889] Downloading the builder-live.log file, attempt 4
[2025-03-24 11:37:26,385][  INFO][PID:1958889] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c294 copr-rpmbuild-log
[2025-03-24 11:38:26,454][ ERROR][PID:1958889] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-24 11:38:36,457][ ERROR][PID:1958889] Giving up for unstable SSH, failures: 1742816076.232044, 1742816156.3094223, 1742816236.3807793, 1742816316.4567697
[2025-03-24 11:38:36,459][ ERROR][PID:1958889] Re-try request for task on 'ResallocHost, ticket_id=2003640, hostname=2620:52:3:1:dead:beef:cafe:c294, name=copr_p09_01_prod_02246324_20250324_112103': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-03-24 11:38:36,459][  INFO][PID:1958889] Releasing VM back to pool
[2025-03-24 11:38:36,477][  INFO][PID:1958889] Retry #1 (on other host)
[2025-03-24 11:38:36,478][  INFO][PID:1958889] Trying to allocate VM
[2025-03-24 11:41:54,765][  INFO][PID:1958889] Allocated host ResallocHost, ticket_id=2003976, hostname=140.211.168.48, name=copr_osuosl_p09_prod_02246794_20250324_113727
[2025-03-24 11:41:54,767][  INFO][PID:1958889] Allocating ssh connection to builder
[2025-03-24 11:41:54,769][  INFO][PID:1958889] Checking that builder machine is OK
[2025-03-24 11:41:56,085][  INFO][PID:1958889] Running remote command: copr-builder-ready fedora-eln-ppc64le
[2025-03-24 11:41:57,090][  INFO][PID:1958889] Red Hat subscription not needed for fedora-eln-ppc64le
Builder is ready to be used

[2025-03-24 11:41:57,091][  INFO][PID:1958889] Filling build.info file with builder info
[2025-03-24 11:41:57,094][  INFO][PID:1958889] Not re-notifying FE and msg buses for the new host.
[2025-03-24 11:41:57,095][  INFO][PID:1958889] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8812228-fedora-eln-ppc64le --chroot fedora-eln-ppc64le --detached
[2025-03-24 11:41:58,388][  INFO][PID:1958889] Downloading the builder-live.log file, attempt 1
[2025-03-24 11:41:58,389][  INFO][PID:1958889] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.48 copr-rpmbuild-log
[2025-03-24 11:42:03,399][  INFO][PID:1958889] Periodic builder liveness probe: alive
[2025-03-24 11:52:03,658][  INFO][PID:1958889] Periodic builder liveness probe: alive
[2025-03-24 12:02:03,832][  INFO][PID:1958889] Periodic builder liveness probe: alive
[2025-03-24 12:12:04,028][  INFO][PID:1958889] Periodic builder liveness probe: alive
[2025-03-24 12:22:01,234][ ERROR][PID:1958889] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-24 12:22:04,118][  INFO][PID:1958889] Periodic builder liveness probe: dead
[2025-03-24 12:22:11,235][WARNING][PID:1958889] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-24 12:22:21,236][  INFO][PID:1958889] Downloading the builder-live.log file, attempt 2
[2025-03-24 12:22:21,237][  INFO][PID:1958889] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.48 copr-rpmbuild-log
[2025-03-24 12:23:04,201][ ERROR][PID:1958889] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.48 port 22: Connection timed out

[2025-03-24 12:23:21,306][ ERROR][PID:1958889] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-24 12:23:31,312][WARNING][PID:1958889] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-24 12:23:41,313][  INFO][PID:1958889] Downloading the builder-live.log file, attempt 3
[2025-03-24 12:23:41,314][  INFO][PID:1958889] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.48 copr-rpmbuild-log
[2025-03-24 12:24:14,271][ ERROR][PID:1958889] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.48 port 22: Connection timed out

[2025-03-24 12:24:41,382][ ERROR][PID:1958889] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-24 12:24:51,386][WARNING][PID:1958889] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-24 12:25:01,387][  INFO][PID:1958889] Downloading the builder-live.log file, attempt 4
[2025-03-24 12:25:01,389][  INFO][PID:1958889] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.48 copr-rpmbuild-log
[2025-03-24 12:25:24,342][ ERROR][PID:1958889] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.48 port 22: Connection timed out

[2025-03-24 12:26:01,458][ ERROR][PID:1958889] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-24 12:26:11,460][ ERROR][PID:1958889] Giving up for unstable SSH, failures: 1742818931.2353652, 1742819011.3124878, 1742819091.386657, 1742819171.4605966
[2025-03-24 12:26:34,412][ ERROR][PID:1958889] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.48 port 22: Connection timed out

[2025-03-24 12:26:44,413][ ERROR][PID:1958889] Can't ssh to cancel build.
[2025-03-24 12:26:44,414][ ERROR][PID:1958889] Re-try request for task on 'ResallocHost, ticket_id=2003976, hostname=140.211.168.48, name=copr_osuosl_p09_prod_02246794_20250324_113727': SSH problems when downloading live log: Resalloc ticket FAILED
[2025-03-24 12:26:44,414][  INFO][PID:1958889] Releasing VM back to pool
[2025-03-24 12:26:44,427][  INFO][PID:1958889] Retry #2 (on other host)
[2025-03-24 12:26:44,427][  INFO][PID:1958889] Trying to allocate VM
[2025-03-24 12:26:47,453][  INFO][PID:1958889] Allocated host ResallocHost, ticket_id=2005263, hostname=2620:52:3:1:dead:beef:cafe:c291, name=copr_p09_01_prod_02247896_20250324_120717
[2025-03-24 12:26:47,453][  INFO][PID:1958889] Allocating ssh connection to builder
[2025-03-24 12:26:47,454][  INFO][PID:1958889] Checking that builder machine is OK
[2025-03-24 12:26:48,341][  INFO][PID:1958889] Running remote command: copr-builder-ready fedora-eln-ppc64le
[2025-03-24 12:26:48,736][  INFO][PID:1958889] Red Hat subscription not needed for fedora-eln-ppc64le
Builder is ready to be used

[2025-03-24 12:26:48,737][  INFO][PID:1958889] Filling build.info file with builder info
[2025-03-24 12:26:48,742][  INFO][PID:1958889] Not re-notifying FE and msg buses for the new host.
[2025-03-24 12:26:48,743][  INFO][PID:1958889] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8812228-fedora-eln-ppc64le --chroot fedora-eln-ppc64le --detached
[2025-03-24 12:26:49,444][  INFO][PID:1958889] Downloading the builder-live.log file, attempt 1
[2025-03-24 12:26:49,446][  INFO][PID:1958889] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c291 copr-rpmbuild-log
[2025-03-24 12:26:58,088][  INFO][PID:1958889] Cancel request succeeded
out:
success
err:
Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c291' (ED25519) to the list of known hosts.

[2025-03-24 12:26:58,089][  INFO][PID:1958889] Downloading results from builder
[2025-03-24 12:26:58,089][  INFO][PID:1958889] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c291]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-eln-ppc64le/08812228-netavark started
[2025-03-24 12:26:58,090][  INFO][PID:1958889] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c291]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-eln-ppc64le/08812228-netavark/ &> /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-eln-ppc64le/08812228-netavark/build-08812228.rsync.log
[2025-03-24 12:26:58,547][  INFO][PID:1958889] rsyncing finished.
[2025-03-24 12:26:58,547][  INFO][PID:1958889] Releasing VM back to pool
[2025-03-24 12:26:58,560][  INFO][PID:1958889] Searching for 'success' file in resultdir
[2025-03-24 12:26:58,561][ ERROR][PID:1958889] Build failed: Backend process error: No success file => build failure
[2025-03-24 12:26:58,561][  INFO][PID:1958889] Finished build: id=8812228 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/rhcontainerbot/podman-next chroot=fedora-eln-ppc64le 
[2025-03-24 12:26:58,563][ ERROR][PID:1958889] Backend process error: No success file => build failure
[2025-03-24 12:26:58,564][  INFO][PID:1958889] Worker failed build, took 9.821326971054077
[2025-03-24 12:26:58,564][  INFO][PID:1958889] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "rhcontainerbot",
            "project_name": "podman-next",
            "project_dirname": "podman-next",
            "submitter": "packit",
            "ended_on": 1742819218.5640204,
            "started_on": 1742819208.7426934,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-eln-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8812228-fedora-eln-ppc64le",
            "build_id": 8812228,
            "package_name": "netavark",
            "package_version": "102:1.14.0-1.20250324112914455606.main.52.g73a9eea",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rhcontainerbot/podman-next/netavark",
            "git_hash": "14036c502dd7fe7ec794eb6509df653d949e511f",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "rhcontainerbot/podman-next--packit",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": false,
            "bootstrap": "on",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/podman-next/fedora-eln-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://kojipkgs.fedoraproject.org/repos/eln-build/latest/$basearch/",
                    "id": "https_kojipkgs_fedoraproject_org_repos_eln_build_latest_basearch",
                    "name": "Additional repo https_kojipkgs_fedoraproject_org_repos_eln_build_latest_basearch"
                }
            ],
            "tags": [
                "arch_ppc64le",
                "arch_power9"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/rhcontainerbot/podman-next",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/podman-next",
            "result_dir": "08812228-netavark",
            "built_packages": "",
            "id": 8812228,
            "mockchain_macros": {
                "copr_username": "rhcontainerbot",
                "copr_projectname": "podman-next",
                "vendor": "Fedora Project COPR (rhcontainerbot/podman-next)"
            }
        }
    ]
}
[2025-03-24 12:26:58,643][  INFO][PID:1958889] Sending fedora-messaging bus message in build.end
[2025-03-24 12:26:58,659][  INFO][PID:1958889] Compressing /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-eln-ppc64le/08812228-netavark/builder-live.log by gzip
[2025-03-24 12:26:58,660][  INFO][PID:1958889] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-eln-ppc64le/08812228-netavark/builder-live.log' as PID 2506909
[2025-03-24 12:26:58,662][  INFO][PID:1958889] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-eln-ppc64le/08812228-netavark/builder-live.log)
[2025-03-24 12:26:58,663][  INFO][PID:1958889] Compressing /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-eln-ppc64le/08812228-netavark/backend.log by gzip
[2025-03-24 12:26:58,664][  INFO][PID:1958889] Running command 'gzip /var/lib/copr/public_html/results/rhcontainerbot/podman-next/fedora-eln-ppc64le/08812228-netavark/backend.log' as PID 2506910