[2025-04-28 06:27:37,040][  INFO][PID:2611118] Marking build as starting
[2025-04-28 06:27:37,119][  INFO][PID:2611118] Trying to allocate VM: ResallocHost, ticket_id=2689284, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-28 06:34:55,392][  INFO][PID:2611118] Allocated host ResallocHost, ticket_id=2689284, hostname=140.211.168.243, name=copr_osuosl_p09_prod_03447439_20250428_062959, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-28 06:34:55,393][  INFO][PID:2611118] Allocating ssh connection to builder
[2025-04-28 06:34:55,394][  INFO][PID:2611118] Checking that builder machine is OK
[2025-04-28 06:34:56,624][  INFO][PID:2611118] Running remote command: copr-builder-ready fedora-40-ppc64le
[2025-04-28 06:34:57,624][  INFO][PID:2611118] Red Hat subscription not needed for fedora-40-ppc64le
Builder is ready to be used

[2025-04-28 06:34:57,625][  INFO][PID:2611118] Filling build.info file with builder info
[2025-04-28 06:34:57,627][  INFO][PID:2611118] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "osloup",
            "project_name": "postgresql_demodularization",
            "project_dirname": "postgresql_demodularization",
            "submitter": "https://src.fedoraproject.org/user/jgrulich",
            "ended_on": null,
            "started_on": 1745822097.6269782,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-40-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8971984-fedora-40-ppc64le",
            "build_id": 8971984,
            "package_name": "qt6-qtbase",
            "package_version": "6.9.0-2",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osloup/postgresql_demodularization/qt6-qtbase",
            "git_hash": "a06659a4dcf6c721b1c1a4d7337939ee593f7597",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "osloup/postgresql_demodularization--https://src.fedoraproject.org/user/jgrulich",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_demodularization/fedora-40-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/osloup/postgresql_demodularization",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_demodularization",
            "result_dir": "08971984-qt6-qtbase",
            "built_packages": "",
            "id": 8971984,
            "mockchain_macros": {
                "copr_username": "osloup",
                "copr_projectname": "postgresql_demodularization",
                "vendor": "Fedora Project COPR (osloup/postgresql_demodularization)"
            }
        }
    ]
}
[2025-04-28 06:34:57,661][  INFO][PID:2611118] Sending fedora-messaging bus message in build.start
[2025-04-28 06:34:58,252][  INFO][PID:2611118] Sending fedora-messaging bus message in chroot.start
[2025-04-28 06:34:58,270][  INFO][PID:2611118] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971984-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2025-04-28 06:34:59,549][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 1
[2025-04-28 06:34:59,551][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log
[2025-04-28 06:35:04,558][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 06:45:04,682][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 06:55:04,842][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 07:05:04,967][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 07:09:44,140][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 07:09:54,149][WARNING][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 07:10:04,151][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 2
[2025-04-28 07:10:04,156][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log
[2025-04-28 07:11:04,241][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 07:11:14,246][WARNING][PID:2611118] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 07:11:24,251][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 3
[2025-04-28 07:11:24,255][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log
[2025-04-28 07:12:24,340][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 07:12:34,344][WARNING][PID:2611118] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 07:12:44,346][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 4
[2025-04-28 07:12:44,352][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log
[2025-04-28 07:13:44,428][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 07:13:54,435][ ERROR][PID:2611118] Giving up for unstable SSH, failures: 1745824194.1491325, 1745824274.2465234, 1745824354.3447716, 1745824434.4353778
[2025-04-28 07:13:54,437][ ERROR][PID:2611118] Re-try request for task on 'ResallocHost, ticket_id=2689284, hostname=140.211.168.243, name=copr_osuosl_p09_prod_03447439_20250428_062959, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-04-28 07:13:54,438][  INFO][PID:2611118] Releasing VM back to pool
[2025-04-28 07:13:54,451][  INFO][PID:2611118] Retry #1 (on other host)
[2025-04-28 07:13:54,466][  INFO][PID:2611118] Trying to allocate VM: ResallocHost, ticket_id=2689680, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-28 07:49:12,884][  INFO][PID:2611118] Allocated host ResallocHost, ticket_id=2689680, hostname=140.211.168.243, name=copr_osuosl_p09_prod_03449297_20250428_074440, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-28 07:49:12,891][  INFO][PID:2611118] Allocating ssh connection to builder
[2025-04-28 07:49:12,892][  INFO][PID:2611118] Checking that builder machine is OK
[2025-04-28 07:49:14,157][  INFO][PID:2611118] Running remote command: copr-builder-ready fedora-40-ppc64le
[2025-04-28 07:49:15,117][  INFO][PID:2611118] Red Hat subscription not needed for fedora-40-ppc64le
Builder is ready to be used

[2025-04-28 07:49:15,118][  INFO][PID:2611118] Filling build.info file with builder info
[2025-04-28 07:49:15,120][  INFO][PID:2611118] Not re-notifying FE and msg buses for the new host.
[2025-04-28 07:49:15,121][  INFO][PID:2611118] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971984-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2025-04-28 07:49:16,421][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 1
[2025-04-28 07:49:16,423][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log
[2025-04-28 07:49:21,430][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 07:59:21,651][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 08:09:22,023][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 08:18:07,435][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 08:18:17,437][WARNING][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 08:18:27,438][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 2
[2025-04-28 08:18:27,441][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log
[2025-04-28 08:18:28,298][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 08:18:38,299][WARNING][PID:2611118] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 08:18:48,301][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 3
[2025-04-28 08:18:48,303][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log
[2025-04-28 08:18:49,168][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 08:18:59,172][WARNING][PID:2611118] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 08:19:09,173][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 4
[2025-04-28 08:19:09,175][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log
[2025-04-28 08:19:10,037][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 08:19:20,043][ ERROR][PID:2611118] Giving up for unstable SSH, failures: 1745828297.437286, 1745828318.299914, 1745828339.1726594, 1745828360.04338
[2025-04-28 08:19:20,044][ ERROR][PID:2611118] Re-try request for task on 'ResallocHost, ticket_id=2689680, hostname=140.211.168.243, name=copr_osuosl_p09_prod_03449297_20250428_074440, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-04-28 08:19:20,045][  INFO][PID:2611118] Releasing VM back to pool
[2025-04-28 08:19:20,057][  INFO][PID:2611118] Retry #2 (on other host)
[2025-04-28 08:19:20,075][  INFO][PID:2611118] Trying to allocate VM: ResallocHost, ticket_id=2690586, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-28 09:43:39,864][  INFO][PID:2611118] Allocated host ResallocHost, ticket_id=2690586, hostname=140.211.168.229, name=copr_osuosl_p09_prod_03453831_20250428_093858, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-28 09:43:39,883][  INFO][PID:2611118] Allocating ssh connection to builder
[2025-04-28 09:43:39,885][  INFO][PID:2611118] Checking that builder machine is OK
[2025-04-28 09:43:41,270][  INFO][PID:2611118] Running remote command: copr-builder-ready fedora-40-ppc64le
[2025-04-28 09:43:42,380][  INFO][PID:2611118] Red Hat subscription not needed for fedora-40-ppc64le
Builder is ready to be used

[2025-04-28 09:43:42,381][  INFO][PID:2611118] Filling build.info file with builder info
[2025-04-28 09:43:42,387][  INFO][PID:2611118] Not re-notifying FE and msg buses for the new host.
[2025-04-28 09:43:42,389][  INFO][PID:2611118] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971984-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2025-04-28 09:43:43,753][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 1
[2025-04-28 09:43:43,766][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.229 copr-rpmbuild-log
[2025-04-28 09:43:48,779][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 09:53:49,208][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 10:03:49,643][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 10:13:50,100][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 10:23:50,493][  INFO][PID:2611118] Periodic builder liveness probe: dead
[2025-04-28 10:24:17,140][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 10:24:27,142][WARNING][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 10:24:37,143][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 2
[2025-04-28 10:24:37,148][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.229 copr-rpmbuild-log
[2025-04-28 10:24:50,583][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.229 port 22: Connection timed out

[2025-04-28 10:25:37,239][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 10:25:47,245][WARNING][PID:2611118] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 10:25:57,248][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 3
[2025-04-28 10:25:57,253][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.229 copr-rpmbuild-log
[2025-04-28 10:26:00,659][ ERROR][PID:2611118] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.229 port 22: Connection timed out

[2025-04-28 10:26:57,336][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 10:27:07,342][WARNING][PID:2611118] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 10:27:10,740][ ERROR][PID:2611118] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.229 port 22: Connection timed out

[2025-04-28 10:27:17,350][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 4
[2025-04-28 10:27:17,355][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.229 copr-rpmbuild-log
[2025-04-28 10:28:17,440][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 10:28:20,834][ ERROR][PID:2611118] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.229 port 22: Connection timed out

[2025-04-28 10:28:27,456][ ERROR][PID:2611118] Giving up for unstable SSH, failures: 1745835867.1422129, 1745835947.2455447, 1745836027.3426178, 1745836107.4562976
[2025-04-28 10:28:30,836][ ERROR][PID:2611118] Can't ssh to cancel build.
[2025-04-28 10:28:30,846][ ERROR][PID:2611118] Re-try request for task on 'ResallocHost, ticket_id=2690586, hostname=140.211.168.229, name=copr_osuosl_p09_prod_03453831_20250428_093858, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED
[2025-04-28 10:28:30,855][  INFO][PID:2611118] Releasing VM back to pool
[2025-04-28 10:28:30,902][  INFO][PID:2611118] Retry #3 (on other host)
[2025-04-28 10:28:30,949][  INFO][PID:2611118] Trying to allocate VM: ResallocHost, ticket_id=2696269, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-28 13:11:53,378][  INFO][PID:2611118] Allocated host ResallocHost, ticket_id=2696269, hostname=140.211.168.229, name=copr_osuosl_p09_prod_03462952_20250428_130646, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-28 13:11:53,385][  INFO][PID:2611118] Allocating ssh connection to builder
[2025-04-28 13:11:53,387][  INFO][PID:2611118] Checking that builder machine is OK
[2025-04-28 13:11:54,724][  INFO][PID:2611118] Running remote command: copr-builder-ready fedora-40-ppc64le
[2025-04-28 13:11:55,797][  INFO][PID:2611118] Red Hat subscription not needed for fedora-40-ppc64le
Builder is ready to be used

[2025-04-28 13:11:55,799][  INFO][PID:2611118] Filling build.info file with builder info
[2025-04-28 13:11:55,826][  INFO][PID:2611118] Not re-notifying FE and msg buses for the new host.
[2025-04-28 13:11:55,831][  INFO][PID:2611118] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971984-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2025-04-28 13:11:57,244][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 1
[2025-04-28 13:11:57,266][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.229 copr-rpmbuild-log
[2025-04-28 13:12:02,266][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 13:22:02,568][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 13:29:00,139][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 13:29:10,142][WARNING][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 13:29:20,144][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 2
[2025-04-28 13:29:20,154][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.229 copr-rpmbuild-log
[2025-04-28 13:30:20,230][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 13:30:30,233][WARNING][PID:2611118] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 13:30:40,235][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 3
[2025-04-28 13:30:40,237][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.229 copr-rpmbuild-log
[2025-04-28 13:31:40,306][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 13:31:50,310][WARNING][PID:2611118] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 13:32:00,314][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 4
[2025-04-28 13:32:00,316][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.229 copr-rpmbuild-log
[2025-04-28 13:32:02,849][  INFO][PID:2611118] Periodic builder liveness probe: dead
[2025-04-28 13:33:00,384][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 13:33:02,926][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.229 port 22: Connection timed out

[2025-04-28 13:33:10,390][ ERROR][PID:2611118] Giving up for unstable SSH, failures: 1745846950.1420596, 1745847030.2333605, 1745847110.3105447, 1745847190.3907976
[2025-04-28 13:34:13,012][ ERROR][PID:2611118] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.229 port 22: Connection timed out

[2025-04-28 13:34:23,090][ ERROR][PID:2611118] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.229 port 22: Connection refused

[2025-04-28 13:34:34,019][ ERROR][PID:2611118] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Warning: Permanently added '140.211.168.229' (ED25519) to the list of known hosts.
Permission denied, please try again.
Permission denied, please try again.
mockbuilder@140.211.168.229: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[2025-04-28 13:34:44,022][ ERROR][PID:2611118] Can't ssh to cancel build.
[2025-04-28 13:34:44,023][ ERROR][PID:2611118] Re-try request for task on 'ResallocHost, ticket_id=2696269, hostname=140.211.168.229, name=copr_osuosl_p09_prod_03462952_20250428_130646, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED
[2025-04-28 13:34:44,024][  INFO][PID:2611118] Releasing VM back to pool
[2025-04-28 13:34:44,040][  INFO][PID:2611118] Retry #4 (on other host)
[2025-04-28 13:34:44,054][  INFO][PID:2611118] Trying to allocate VM: ResallocHost, ticket_id=2700609, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-28 14:41:03,130][  INFO][PID:2611118] Allocated host ResallocHost, ticket_id=2700609, hostname=140.211.168.83, name=copr_osuosl_p09_prod_03473913_20250428_143610, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-28 14:41:03,140][  INFO][PID:2611118] Allocating ssh connection to builder
[2025-04-28 14:41:03,143][  INFO][PID:2611118] Checking that builder machine is OK
[2025-04-28 14:41:04,427][  INFO][PID:2611118] Running remote command: copr-builder-ready fedora-40-ppc64le
[2025-04-28 14:41:05,444][  INFO][PID:2611118] Red Hat subscription not needed for fedora-40-ppc64le
Builder is ready to be used

[2025-04-28 14:41:05,446][  INFO][PID:2611118] Filling build.info file with builder info
[2025-04-28 14:41:05,470][  INFO][PID:2611118] Not re-notifying FE and msg buses for the new host.
[2025-04-28 14:41:05,472][  INFO][PID:2611118] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971984-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2025-04-28 14:41:06,773][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 1
[2025-04-28 14:41:06,776][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.83 copr-rpmbuild-log
[2025-04-28 14:41:11,807][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 14:51:12,109][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 15:01:12,409][  INFO][PID:2611118] Periodic builder liveness probe: alive
[2025-04-28 15:11:12,776][  INFO][PID:2611118] Periodic builder liveness probe: dead
[2025-04-28 15:11:12,977][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Connection closed by 140.211.168.83 port 22

[2025-04-28 15:12:23,142][ ERROR][PID:2611118] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Connection timed out during banner exchange
Connection to 140.211.168.83 port 22 timed out

[2025-04-28 15:13:33,229][ ERROR][PID:2611118] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.83 port 22: Connection timed out

[2025-04-28 15:14:18,147][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 15:14:28,150][WARNING][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 15:14:38,156][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 2
[2025-04-28 15:14:38,177][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.83 copr-rpmbuild-log
[2025-04-28 15:14:43,301][ ERROR][PID:2611118] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.83 port 22: Connection timed out

[2025-04-28 15:14:53,312][ ERROR][PID:2611118] Can't ssh to cancel build.
[2025-04-28 15:15:38,277][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 15:15:48,282][WARNING][PID:2611118] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 15:15:58,283][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 3
[2025-04-28 15:15:58,289][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.83 copr-rpmbuild-log
[2025-04-28 15:16:58,364][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 15:17:08,376][WARNING][PID:2611118] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-28 15:17:18,378][  INFO][PID:2611118] Downloading the builder-live.log file, attempt 4
[2025-04-28 15:17:18,381][  INFO][PID:2611118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.83 copr-rpmbuild-log
[2025-04-28 15:18:18,459][ ERROR][PID:2611118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-28 15:18:28,469][ ERROR][PID:2611118] Giving up for unstable SSH, failures: 1745853268.1506362, 1745853348.2821777, 1745853428.376857, 1745853508.4692929
[2025-04-28 15:18:28,472][ ERROR][PID:2611118] Re-try request for task on 'ResallocHost, ticket_id=2700609, hostname=140.211.168.83, name=copr_osuosl_p09_prod_03473913_20250428_143610, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED
[2025-04-28 15:18:28,473][  INFO][PID:2611118] Releasing VM back to pool
[2025-04-28 15:18:28,505][ ERROR][PID:2611118] Backend process error: Three host tried without success: {'140.211.168.243', '140.211.168.83', '140.211.168.229'}
[2025-04-28 15:18:28,513][WARNING][PID:2611118] Switching not-finished job state to 'failed'
[2025-04-28 15:18:28,515][  INFO][PID:2611118] Worker failed build, took 2243.0435876846313
[2025-04-28 15:18:28,536][  INFO][PID:2611118] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "osloup",
            "project_name": "postgresql_demodularization",
            "project_dirname": "postgresql_demodularization",
            "submitter": "https://src.fedoraproject.org/user/jgrulich",
            "ended_on": 1745853508.5125663,
            "started_on": 1745851265.4689786,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-40-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8971984-fedora-40-ppc64le",
            "build_id": 8971984,
            "package_name": "qt6-qtbase",
            "package_version": "6.9.0-2",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osloup/postgresql_demodularization/qt6-qtbase",
            "git_hash": "a06659a4dcf6c721b1c1a4d7337939ee593f7597",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "osloup/postgresql_demodularization--https://src.fedoraproject.org/user/jgrulich",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_demodularization/fedora-40-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/osloup/postgresql_demodularization",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_demodularization",
            "result_dir": "08971984-qt6-qtbase",
            "built_packages": "",
            "id": 8971984,
            "mockchain_macros": {
                "copr_username": "osloup",
                "copr_projectname": "postgresql_demodularization",
                "vendor": "Fedora Project COPR (osloup/postgresql_demodularization)"
            }
        }
    ]
}
[2025-04-28 15:18:29,300][  INFO][PID:2611118] Sending fedora-messaging bus message in build.end
[2025-04-28 15:18:29,677][  INFO][PID:2611118] Compressing /var/lib/copr/public_html/results/osloup/postgresql_demodularization/fedora-40-ppc64le/08971984-qt6-qtbase/builder-live.log by gzip
[2025-04-28 15:18:29,695][  INFO][PID:2611118] Running command 'gzip /var/lib/copr/public_html/results/osloup/postgresql_demodularization/fedora-40-ppc64le/08971984-qt6-qtbase/builder-live.log' as PID 2813420
[2025-04-28 15:18:29,709][  INFO][PID:2611118] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/osloup/postgresql_demodularization/fedora-40-ppc64le/08971984-qt6-qtbase/builder-live.log)
[2025-04-28 15:18:29,716][  INFO][PID:2611118] Compressing /var/lib/copr/public_html/results/osloup/postgresql_demodularization/fedora-40-ppc64le/08971984-qt6-qtbase/backend.log by gzip
[2025-04-28 15:18:29,719][  INFO][PID:2611118] Running command 'gzip /var/lib/copr/public_html/results/osloup/postgresql_demodularization/fedora-40-ppc64le/08971984-qt6-qtbase/backend.log' as PID 2813421