[2025-12-22 15:17:03,418][  INFO][PID:3696595] Marking build as starting
[2025-12-22 15:17:03,471][  INFO][PID:3696595] Checking that /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2025-12-22 15:17:03,554][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:17:03,555][  INFO][PID:3696595] VM allocation process starts
[2025-12-22 15:17:03,565][  INFO][PID:3696595] Trying to allocate VM: ResallocHost, ticket_id=473485, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-22 15:17:06,580][  INFO][PID:3696595] Allocated host ResallocHost, ticket_id=473485, hostname=2620:52:6:1161:dead:beef:cafe:c14f, name=vmhost_p09_03_prod_00731690_20251222_151103, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-22 15:17:06,580][  INFO][PID:3696595] Allocating ssh connection to builder
[2025-12-22 15:17:06,581][  INFO][PID:3696595] Checking that builder machine is OK
[2025-12-22 15:17:07,427][  INFO][PID:3696595] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-12-22 15:17:07,912][  INFO][PID:3696595] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-12-22 15:17:07,913][  INFO][PID:3696595] Filling build.info file with builder info
[2025-12-22 15:17:08,009][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:17:08,010][  INFO][PID:3696595] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "osloup",
            "project_name": "postgresql_update_15_2",
            "project_dirname": "postgresql_update_15_2",
            "submitter": "https://src.fedoraproject.org/user/jgrulich",
            "ended_on": null,
            "started_on": 1766416628.0105941,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9941303-fedora-rawhide-ppc64le",
            "build_id": 9941303,
            "package_name": "qt6-qtbase",
            "package_version": "6.10.1-3",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osloup/postgresql_update_15_2/qt6-qtbase",
            "git_hash": "e9f0c7f0e0e871b42385f3efb2a7dda51583abab",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "osloup/postgresql_update_15_2--https://src.fedoraproject.org/user/jgrulich",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_update_15_2/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/osloup/postgresql_update_15_2",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_update_15_2",
            "result_dir": "09941303-qt6-qtbase",
            "built_packages": "",
            "id": 9941303,
            "mockchain_macros": {
                "copr_username": "osloup",
                "copr_projectname": "postgresql_update_15_2",
                "vendor": "Fedora Project COPR (osloup/postgresql_update_15_2)"
            }
        }
    ]
}
[2025-12-22 15:17:08,063][  INFO][PID:3696595] Sending fedora-messaging bus message in build.start
[2025-12-22 15:17:08,548][  INFO][PID:3696595] Sending fedora-messaging bus message in chroot.start
[2025-12-22 15:17:08,573][  INFO][PID:3696595] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9941303-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-12-22 15:17:09,508][  INFO][PID:3696595] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c14f' (ED25519) to the list of known hosts.


[2025-12-22 15:17:09,508][  INFO][PID:3696595] Downloading the builder-live.log file, attempt 1
[2025-12-22 15:17:09,510][  INFO][PID:3696595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14f copr-rpmbuild-log
[2025-12-22 15:17:14,513][  INFO][PID:3696595] Periodic builder liveness probe: alive
[2025-12-22 15:17:14,514][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:17:19,515][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:17:24,516][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:17:29,517][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:17:34,518][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:17:39,519][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:17:44,520][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:17:49,521][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:17:54,522][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:17:59,523][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:18:04,524][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:18:09,525][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:18:14,526][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:18:19,527][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:18:24,527][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:18:29,529][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:18:34,530][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:18:39,530][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:18:44,531][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:18:49,532][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:18:54,533][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:18:59,534][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:19:04,535][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:19:09,536][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:19:14,537][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:19:19,538][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:19:24,539][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:19:29,540][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:19:34,541][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:19:39,542][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:19:44,543][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:19:49,544][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:19:54,546][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:19:59,547][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:20:04,547][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:20:09,549][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:20:14,550][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:20:17,370][ ERROR][PID:3696595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-22 15:20:19,551][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:20:24,552][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:20:27,371][WARNING][PID:3696595] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-22 15:20:29,553][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:20:34,554][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:20:37,372][  INFO][PID:3696595] Downloading the builder-live.log file, attempt 2
[2025-12-22 15:20:37,373][  INFO][PID:3696595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14f copr-rpmbuild-log
[2025-12-22 15:20:39,555][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:20:44,556][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:20:49,557][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:20:54,559][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:20:59,561][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:21:04,563][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:21:09,563][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:21:14,565][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:21:19,566][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:21:24,567][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:21:29,568][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:21:34,569][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:21:37,435][ ERROR][PID:3696595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-22 15:21:39,570][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:21:44,571][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:21:47,449][WARNING][PID:3696595] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-22 15:21:49,572][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:21:54,573][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:21:57,450][  INFO][PID:3696595] Downloading the builder-live.log file, attempt 3
[2025-12-22 15:21:57,451][  INFO][PID:3696595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14f copr-rpmbuild-log
[2025-12-22 15:21:59,574][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:22:04,575][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:22:09,576][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:22:14,578][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:22:19,579][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:22:24,580][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:22:29,581][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:22:34,582][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:22:39,584][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:22:44,585][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:22:49,587][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:22:54,588][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:22:57,516][ ERROR][PID:3696595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-22 15:22:59,589][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:23:04,590][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:23:07,521][WARNING][PID:3696595] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-22 15:23:09,591][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:23:14,593][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:23:17,522][  INFO][PID:3696595] Downloading the builder-live.log file, attempt 4
[2025-12-22 15:23:17,523][  INFO][PID:3696595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14f copr-rpmbuild-log
[2025-12-22 15:23:19,595][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:23:24,596][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:23:29,597][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:23:34,598][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:23:39,599][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:23:44,600][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:23:49,601][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:23:54,602][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:23:59,603][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:24:04,604][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:24:09,605][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:24:14,606][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:24:17,591][ ERROR][PID:3696595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-22 15:24:19,607][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:24:24,608][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:24:27,593][ ERROR][PID:3696595] Giving up for unstable SSH, failures: 1766416827.3717668, 1766416907.4496357, 1766416987.5212295, 1766417067.5936904
[2025-12-22 15:24:27,594][ ERROR][PID:3696595] Re-try request for task on 'ResallocHost, ticket_id=473485, hostname=2620:52:6:1161:dead:beef:cafe:c14f, name=vmhost_p09_03_prod_00731690_20251222_151103, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-12-22 15:24:27,595][  INFO][PID:3696595] Releasing VM back to pool
[2025-12-22 15:24:27,603][  INFO][PID:3696595] Retry #1 (on other host)
[2025-12-22 15:24:27,604][  INFO][PID:3696595] Checking that /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2025-12-22 15:24:27,604][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:24:27,605][  INFO][PID:3696595] VM allocation process starts
[2025-12-22 15:24:27,649][  INFO][PID:3696595] Trying to allocate VM: ResallocHost, ticket_id=473553, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-22 15:24:30,698][  INFO][PID:3696595] Allocated host ResallocHost, ticket_id=473553, hostname=2620:52:6:1161:dead:beef:cafe:c18b, name=vmhost_p09_04_prod_00731726_20251222_151709, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-22 15:24:30,699][  INFO][PID:3696595] Allocating ssh connection to builder
[2025-12-22 15:24:30,699][  INFO][PID:3696595] Checking that builder machine is OK
[2025-12-22 15:24:31,714][  INFO][PID:3696595] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-12-22 15:24:32,245][  INFO][PID:3696595] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-12-22 15:24:32,245][  INFO][PID:3696595] Filling build.info file with builder info
[2025-12-22 15:24:32,262][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:24:32,262][  INFO][PID:3696595] Not re-notifying FE and msg buses for the new host.
[2025-12-22 15:24:32,263][  INFO][PID:3696595] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9941303-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-12-22 15:24:33,223][  INFO][PID:3696595] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18b' (ED25519) to the list of known hosts.


[2025-12-22 15:24:33,224][  INFO][PID:3696595] Downloading the builder-live.log file, attempt 1
[2025-12-22 15:24:33,225][  INFO][PID:3696595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18b copr-rpmbuild-log
[2025-12-22 15:24:38,224][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:24:43,225][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:24:48,226][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:24:53,227][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:24:58,228][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:25:03,229][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:25:08,230][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:25:13,231][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:25:18,232][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:25:23,233][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:25:28,234][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:25:33,235][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:25:38,236][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:25:43,237][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:25:48,237][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:25:53,238][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:25:58,239][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:26:03,240][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:26:08,241][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:26:13,242][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:26:18,244][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:26:23,245][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:26:28,246][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:26:33,247][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:26:38,249][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:26:43,250][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:26:48,250][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:26:53,251][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:26:58,252][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:27:03,253][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:27:08,254][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:27:13,255][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:27:18,261][  INFO][PID:3696595] Periodic builder liveness probe: alive
[2025-12-22 15:27:18,261][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:27:23,262][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:27:28,263][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:27:33,264][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:27:38,265][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:27:43,266][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:27:48,267][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:27:53,267][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:27:58,268][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:28:03,269][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:28:08,270][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:28:13,271][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:28:18,272][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:28:23,302][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:28:28,303][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:28:33,611][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:28:38,611][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:28:43,612][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:28:48,613][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:28:53,614][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:28:58,615][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:29:03,692][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:29:08,693][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:29:13,693][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:29:18,694][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:29:23,695][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:29:28,696][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:29:33,697][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:29:38,698][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:29:43,699][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:29:48,700][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:29:53,701][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:29:58,702][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:30:03,703][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:30:08,704][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:30:13,705][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:30:18,706][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:30:23,706][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:30:28,707][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:30:33,708][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:30:38,709][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:30:43,710][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:30:48,711][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:30:53,712][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:30:58,712][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:31:03,713][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:31:08,754][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:31:13,773][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:31:18,774][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:31:23,776][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:31:28,779][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:31:33,780][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:31:38,781][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:31:43,782][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:31:48,783][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:31:53,784][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:31:58,884][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:32:03,886][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:32:08,887][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:32:13,888][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:32:18,889][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:32:23,891][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:32:28,892][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:32:33,893][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:32:38,894][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:32:43,897][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:32:48,899][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:32:53,958][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:32:58,975][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:33:03,976][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:33:09,028][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:33:14,029][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:33:19,030][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:33:24,031][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:33:29,032][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:33:34,034][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:33:39,035][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:33:44,036][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:33:49,040][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:33:54,041][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:33:59,042][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:34:04,043][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:34:09,044][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:34:14,125][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:34:19,126][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:34:24,127][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:34:29,128][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:34:34,129][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:34:39,130][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:34:44,132][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:34:49,133][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:34:54,134][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:34:59,134][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:35:04,136][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:35:09,296][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:35:14,297][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:35:19,298][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:35:24,299][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:35:29,300][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:35:34,301][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:35:39,302][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:35:44,303][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:35:49,325][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:35:54,326][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:35:59,344][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:36:04,346][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:36:09,347][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:36:14,348][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:36:19,349][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:36:24,351][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:36:29,352][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:36:34,353][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:36:39,354][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:36:44,356][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:36:49,357][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:36:55,046][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:37:00,047][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:37:05,048][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:37:10,049][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:37:11,369][ ERROR][PID:3696595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-22 15:37:15,050][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:37:20,058][  INFO][PID:3696595] Periodic builder liveness probe: dead
[2025-12-22 15:37:20,059][  INFO][PID:3696595] calling cancel callback, and waiting
[2025-12-22 15:37:21,370][WARNING][PID:3696595] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-22 15:37:31,371][  INFO][PID:3696595] Downloading the builder-live.log file, attempt 2
[2025-12-22 15:37:31,373][  INFO][PID:3696595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18b copr-rpmbuild-log
[2025-12-22 15:38:20,130][ ERROR][PID:3696595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c18b port 22: Connection timed out

[2025-12-22 15:38:31,440][ ERROR][PID:3696595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-22 15:38:41,445][WARNING][PID:3696595] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-22 15:38:51,446][  INFO][PID:3696595] Downloading the builder-live.log file, attempt 3
[2025-12-22 15:38:51,447][  INFO][PID:3696595] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c18b copr-rpmbuild-log
[2025-12-22 15:39:06,814][WARNING][PID:3696595] Can't cancel build
out:

err:
Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c18b' (ED25519) to the list of known hosts.
Traceback (most recent call last):
  File "/usr/bin/copr-rpmbuild-cancel", line 85, in <module>
    _main()
    ~~~~~^^
  File "/usr/bin/copr-rpmbuild-cancel", line 69, in _main
    with open(pidfile, "r") as pidfd:
         ~~~~^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/copr-rpmbuild/pid'

[2025-12-22 15:40:55,645][ ERROR][PID:3696595] Re-try request for task on 'ResallocHost, ticket_id=473553, hostname=2620:52:6:1161:dead:beef:cafe:c18b, name=vmhost_p09_04_prod_00731726_20251222_151709, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED
[2025-12-22 15:40:55,646][  INFO][PID:3696595] Releasing VM back to pool
[2025-12-22 15:40:55,654][  INFO][PID:3696595] Retry #2 (on other host)
[2025-12-22 15:40:55,654][  INFO][PID:3696595] Checking that /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2025-12-22 15:40:55,654][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:40:55,655][  INFO][PID:3696595] VM allocation process starts
[2025-12-22 15:40:55,702][  INFO][PID:3696595] Trying to allocate VM: ResallocHost, ticket_id=473678, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-22 15:40:58,744][  INFO][PID:3696595] Allocated host ResallocHost, ticket_id=473678, hostname=2620:52:6:1161:dead:beef:cafe:c14c, name=vmhost_p09_03_prod_00731806_20251222_152851, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-22 15:40:58,745][  INFO][PID:3696595] Allocating ssh connection to builder
[2025-12-22 15:40:58,745][  INFO][PID:3696595] Checking that builder machine is OK
[2025-12-22 15:40:59,854][  INFO][PID:3696595] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-12-22 15:41:00,370][  INFO][PID:3696595] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-12-22 15:41:00,371][  INFO][PID:3696595] Filling build.info file with builder info
[2025-12-22 15:41:00,380][  INFO][PID:3696595] Checking for cancel request
[2025-12-22 15:41:00,381][  INFO][PID:3696595] Not re-notifying FE and msg buses for the new host.
[2025-12-22 15:41:00,382][  INFO][PID:3696595] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9941303-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-12-22 15:42:00,450][ ERROR][PID:3696595] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c14c port 22: Connection timed out

[2025-12-22 15:42:10,459][ ERROR][PID:3696595] Re-try request for task on 'ResallocHost, ticket_id=473678, hostname=2620:52:6:1161:dead:beef:cafe:c14c, name=vmhost_p09_03_prod_00731806_20251222_152851, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-22 15:42:10,460][  INFO][PID:3696595] Releasing VM back to pool
[2025-12-22 15:42:10,519][ ERROR][PID:3696595] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c18b', '2620:52:6:1161:dead:beef:cafe:c14c', '2620:52:6:1161:dead:beef:cafe:c14f'}
[2025-12-22 15:42:10,520][WARNING][PID:3696595] Switching not-finished job state to 'failed'
[2025-12-22 15:42:10,520][  INFO][PID:3696595] Worker failed build, took 70.13877177238464
[2025-12-22 15:42:10,521][  INFO][PID:3696595] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "osloup",
            "project_name": "postgresql_update_15_2",
            "project_dirname": "postgresql_update_15_2",
            "submitter": "https://src.fedoraproject.org/user/jgrulich",
            "ended_on": 1766418130.5202825,
            "started_on": 1766418060.3815107,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9941303-fedora-rawhide-ppc64le",
            "build_id": 9941303,
            "package_name": "qt6-qtbase",
            "package_version": "6.10.1-3",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osloup/postgresql_update_15_2/qt6-qtbase",
            "git_hash": "e9f0c7f0e0e871b42385f3efb2a7dda51583abab",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "osloup/postgresql_update_15_2--https://src.fedoraproject.org/user/jgrulich",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_update_15_2/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/osloup/postgresql_update_15_2",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_update_15_2",
            "result_dir": "09941303-qt6-qtbase",
            "built_packages": "",
            "id": 9941303,
            "mockchain_macros": {
                "copr_username": "osloup",
                "copr_projectname": "postgresql_update_15_2",
                "vendor": "Fedora Project COPR (osloup/postgresql_update_15_2)"
            }
        }
    ]
}
[2025-12-22 15:42:10,588][  INFO][PID:3696595] Sending fedora-messaging bus message in build.end
[2025-12-22 15:42:10,622][  INFO][PID:3696595] Compressing /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-rawhide-ppc64le/09941303-qt6-qtbase/builder-live.log by gzip
[2025-12-22 15:42:10,624][  INFO][PID:3696595] Running command 'gzip /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-rawhide-ppc64le/09941303-qt6-qtbase/builder-live.log' as PID 3803828
[2025-12-22 15:42:10,659][  INFO][PID:3696595] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-rawhide-ppc64le/09941303-qt6-qtbase/builder-live.log)
[2025-12-22 15:42:10,660][  INFO][PID:3696595] Compressing /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-rawhide-ppc64le/09941303-qt6-qtbase/backend.log by gzip
[2025-12-22 15:42:10,661][  INFO][PID:3696595] Running command 'gzip /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-rawhide-ppc64le/09941303-qt6-qtbase/backend.log' as PID 3803847