[2025-07-23 09:18:52,774][  INFO][PID:1262243] Marking build as starting
[2025-07-23 09:18:52,820][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:18:52,822][  INFO][PID:1262243] VM allocation process starts
[2025-07-23 09:18:52,838][  INFO][PID:1262243] Trying to allocate VM: ResallocHost, ticket_id=4061575, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-23 09:18:57,843][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:18:58,890][  INFO][PID:1262243] Allocated host ResallocHost, ticket_id=4061575, hostname=140.211.168.190, name=copr_osuosl_p09_prod_05755509_20250723_090625, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-23 09:18:58,891][  INFO][PID:1262243] Allocating ssh connection to builder
[2025-07-23 09:18:58,893][  INFO][PID:1262243] Checking that builder machine is OK
[2025-07-23 09:19:00,562][  INFO][PID:1262243] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-07-23 09:19:01,611][  INFO][PID:1262243] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-07-23 09:19:01,612][  INFO][PID:1262243] Filling build.info file with builder info
[2025-07-23 09:19:01,613][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:19:01,615][  INFO][PID:1262243] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "dreibh",
            "project_name": "ppa",
            "project_dirname": "ppa",
            "submitter": "dreibh",
            "ended_on": null,
            "started_on": 1753262341.6149092,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9311508-fedora-rawhide-ppc64le",
            "build_id": 9311508,
            "package_name": "neat-stuff",
            "package_version": "1.2.4-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/dreibh/ppa/neat-stuff",
            "git_hash": "634619ea2ddc20ea128a40cdb11e78fd5221dceb",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "dreibh/ppa--dreibh",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/dreibh/ppa/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/dreibh/ppa",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/dreibh/ppa",
            "result_dir": "09311508-neat-stuff",
            "built_packages": "",
            "id": 9311508,
            "mockchain_macros": {
                "copr_username": "dreibh",
                "copr_projectname": "ppa",
                "vendor": "Fedora Project COPR (dreibh/ppa)"
            }
        }
    ]
}
[2025-07-23 09:19:01,659][  INFO][PID:1262243] Sending fedora-messaging bus message in build.start
[2025-07-23 09:19:02,579][  INFO][PID:1262243] Sending fedora-messaging bus message in chroot.start
[2025-07-23 09:19:02,608][  INFO][PID:1262243] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9311508-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-07-23 09:19:04,025][  INFO][PID:1262243] Downloading the builder-live.log file, attempt 1
[2025-07-23 09:19:04,027][  INFO][PID:1262243] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.190 copr-rpmbuild-log
[2025-07-23 09:19:09,031][  INFO][PID:1262243] Periodic builder liveness probe: alive
[2025-07-23 09:19:09,031][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:19:14,032][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:19:19,034][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:19:24,035][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:19:29,036][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:19:34,037][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:19:39,038][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:19:44,039][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:19:49,041][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:19:54,042][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:19:59,045][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:20:04,046][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:20:09,053][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:20:14,054][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:20:19,056][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:20:24,060][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:20:29,062][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:20:34,064][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:20:39,065][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:20:44,067][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:20:49,069][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:20:54,071][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:20:59,072][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:21:04,073][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:21:09,080][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:21:14,083][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:21:19,084][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:21:24,085][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:21:29,087][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:21:34,088][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:21:39,089][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:21:44,091][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:21:49,092][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:21:54,093][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:21:59,094][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:22:04,096][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:22:09,097][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:22:14,098][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:22:19,099][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:22:24,100][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:22:29,102][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:22:34,103][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:22:39,104][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:22:44,105][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:22:49,107][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:22:54,110][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:22:59,112][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:23:04,114][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:23:09,115][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:23:14,116][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:23:17,139][ ERROR][PID:1262243] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-23 09:23:19,117][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:23:24,118][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:23:27,140][WARNING][PID:1262243] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-23 09:23:29,119][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:23:34,120][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:23:37,141][  INFO][PID:1262243] Downloading the builder-live.log file, attempt 2
[2025-07-23 09:23:37,142][  INFO][PID:1262243] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.190 copr-rpmbuild-log
[2025-07-23 09:23:39,121][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:23:44,123][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:23:49,124][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:23:54,125][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:23:59,126][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:24:04,127][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:24:09,128][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:24:14,129][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:24:19,130][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:24:24,132][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:24:29,133][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:24:34,134][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:24:37,278][ ERROR][PID:1262243] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-23 09:24:39,135][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:24:44,140][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:24:47,279][WARNING][PID:1262243] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-23 09:24:49,143][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:24:54,144][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:24:57,282][  INFO][PID:1262243] Downloading the builder-live.log file, attempt 3
[2025-07-23 09:24:57,284][  INFO][PID:1262243] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.190 copr-rpmbuild-log
[2025-07-23 09:24:59,145][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:25:04,147][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:25:09,148][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:25:14,149][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:25:19,151][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:25:24,154][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:25:29,155][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:25:34,156][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:25:39,157][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:25:44,159][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:25:49,161][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:25:54,162][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:25:57,426][ ERROR][PID:1262243] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-23 09:25:59,165][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:26:04,166][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:26:07,427][WARNING][PID:1262243] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-23 09:26:09,168][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:26:14,171][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:26:17,428][  INFO][PID:1262243] Downloading the builder-live.log file, attempt 4
[2025-07-23 09:26:17,430][  INFO][PID:1262243] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.190 copr-rpmbuild-log
[2025-07-23 09:26:19,174][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:26:24,177][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:26:29,179][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:26:34,181][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:26:39,185][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:26:44,189][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:26:49,193][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:26:54,199][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:26:59,201][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:27:04,205][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:27:09,213][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:27:14,214][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:27:17,572][ ERROR][PID:1262243] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-23 09:27:19,219][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:27:24,222][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:27:27,581][ ERROR][PID:1262243] Giving up for unstable SSH, failures: 1753262607.140396, 1753262687.279208, 1753262767.427235, 1753262847.5811865
[2025-07-23 09:27:27,585][ ERROR][PID:1262243] Re-try request for task on 'ResallocHost, ticket_id=4061575, hostname=140.211.168.190, name=copr_osuosl_p09_prod_05755509_20250723_090625, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-07-23 09:27:27,593][  INFO][PID:1262243] Releasing VM back to pool
[2025-07-23 09:27:27,636][  INFO][PID:1262243] Retry #1 (on other host)
[2025-07-23 09:27:27,643][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:27:27,645][  INFO][PID:1262243] VM allocation process starts
[2025-07-23 09:27:27,698][  INFO][PID:1262243] Trying to allocate VM: ResallocHost, ticket_id=4061774, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-23 09:27:32,711][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:27:37,713][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:27:42,716][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:27:47,736][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:27:52,737][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:27:57,738][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:28:02,741][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:28:07,747][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:28:12,758][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:28:17,761][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:28:22,771][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:28:27,773][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:28:32,777][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:28:37,785][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:28:42,786][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:28:47,790][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:28:52,797][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:28:57,799][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:29:02,801][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:29:07,802][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:29:12,805][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:29:17,808][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:29:22,810][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:29:27,811][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:29:32,815][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:29:37,818][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:29:42,820][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:29:47,823][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:29:52,824][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:29:57,830][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:30:02,831][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:30:07,833][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:30:12,835][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:30:17,843][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:30:22,845][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:30:27,848][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:30:32,849][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:30:37,851][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:30:42,861][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:30:45,970][  INFO][PID:1262243] Allocated host ResallocHost, ticket_id=4061774, hostname=140.211.168.2, name=copr_osuosl_p09_prod_05755889_20250723_092647, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-23 09:30:45,972][  INFO][PID:1262243] Allocating ssh connection to builder
[2025-07-23 09:30:45,973][  INFO][PID:1262243] Checking that builder machine is OK
[2025-07-23 09:30:47,571][  INFO][PID:1262243] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-07-23 09:30:48,602][  INFO][PID:1262243] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-07-23 09:30:48,603][  INFO][PID:1262243] Filling build.info file with builder info
[2025-07-23 09:30:48,607][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:30:48,612][  INFO][PID:1262243] Not re-notifying FE and msg buses for the new host.
[2025-07-23 09:30:48,612][  INFO][PID:1262243] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9311508-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-07-23 09:30:49,974][  INFO][PID:1262243] Downloading the builder-live.log file, attempt 1
[2025-07-23 09:30:49,975][  INFO][PID:1262243] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.2 copr-rpmbuild-log
[2025-07-23 09:30:54,994][  INFO][PID:1262243] Periodic builder liveness probe: alive
[2025-07-23 09:30:54,996][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:30:59,997][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:31:04,999][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:31:10,001][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:31:15,009][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:31:20,010][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:31:25,012][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:31:30,017][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:31:35,019][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:31:40,021][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:31:45,022][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:31:50,027][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:31:55,030][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:32:00,036][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:32:05,038][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:32:10,039][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:32:15,042][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:32:20,047][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:32:25,048][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:32:30,051][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:32:35,054][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:32:40,059][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:32:45,062][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:32:50,067][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:32:55,069][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:33:00,071][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:33:05,072][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:33:10,076][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:33:15,078][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:33:20,083][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:33:25,085][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:33:30,086][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:33:35,089][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:33:40,094][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:33:45,097][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:33:50,104][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:33:55,108][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:34:00,109][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:34:05,110][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:34:10,112][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:34:15,113][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:34:20,117][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:34:25,118][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:34:25,154][ ERROR][PID:1262243] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-23 09:34:30,120][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:34:35,123][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:34:35,155][WARNING][PID:1262243] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-23 09:34:40,127][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:34:45,129][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:34:45,156][  INFO][PID:1262243] Downloading the builder-live.log file, attempt 2
[2025-07-23 09:34:45,158][  INFO][PID:1262243] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.2 copr-rpmbuild-log
[2025-07-23 09:34:50,130][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:34:55,132][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:00,133][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:05,135][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:10,136][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:15,137][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:20,139][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:25,140][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:30,141][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:35,143][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:40,145][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:45,146][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:45,295][ ERROR][PID:1262243] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-23 09:35:50,147][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:55,148][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:35:55,303][WARNING][PID:1262243] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-23 09:36:00,151][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:36:05,153][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:36:05,303][  INFO][PID:1262243] Downloading the builder-live.log file, attempt 3
[2025-07-23 09:36:05,305][  INFO][PID:1262243] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.2 copr-rpmbuild-log
[2025-07-23 09:36:10,154][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:36:15,155][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:36:20,156][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:36:25,157][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:36:30,158][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:36:35,159][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:36:40,161][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:36:45,162][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:36:50,163][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:36:55,164][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:37:00,165][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:37:05,166][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:37:05,443][ ERROR][PID:1262243] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-23 09:37:10,168][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:37:15,169][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:37:15,446][WARNING][PID:1262243] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-07-23 09:37:20,170][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:37:25,171][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:37:25,447][  INFO][PID:1262243] Downloading the builder-live.log file, attempt 4
[2025-07-23 09:37:25,448][  INFO][PID:1262243] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.2 copr-rpmbuild-log
[2025-07-23 09:37:30,172][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:37:35,173][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:37:40,174][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:37:45,176][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:37:50,179][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:37:55,180][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:00,181][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:05,182][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:10,184][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:15,185][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:20,186][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:25,187][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:25,584][ ERROR][PID:1262243] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-07-23 09:38:30,188][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:35,189][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:35,587][ ERROR][PID:1262243] Giving up for unstable SSH, failures: 1753263275.154947, 1753263355.3030705, 1753263435.4465296, 1753263515.5871637
[2025-07-23 09:38:35,588][ ERROR][PID:1262243] Re-try request for task on 'ResallocHost, ticket_id=4061774, hostname=140.211.168.2, name=copr_osuosl_p09_prod_05755889_20250723_092647, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-07-23 09:38:35,588][  INFO][PID:1262243] Releasing VM back to pool
[2025-07-23 09:38:35,596][  INFO][PID:1262243] Retry #2 (on other host)
[2025-07-23 09:38:35,596][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:35,597][  INFO][PID:1262243] VM allocation process starts
[2025-07-23 09:38:35,645][  INFO][PID:1262243] Trying to allocate VM: ResallocHost, ticket_id=4062088, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-23 09:38:38,679][  INFO][PID:1262243] Allocated host ResallocHost, ticket_id=4062088, hostname=2620:52:3:1:dead:beef:cafe:c249, name=copr_hv_ppc64le_02_prod_05755952_20250723_092929, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-07-23 09:38:38,680][  INFO][PID:1262243] Allocating ssh connection to builder
[2025-07-23 09:38:38,680][  INFO][PID:1262243] Checking that builder machine is OK
[2025-07-23 09:38:41,612][  INFO][PID:1262243] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-07-23 09:38:42,541][  INFO][PID:1262243] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-07-23 09:38:42,542][  INFO][PID:1262243] Filling build.info file with builder info
[2025-07-23 09:38:42,545][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:42,546][  INFO][PID:1262243] Not re-notifying FE and msg buses for the new host.
[2025-07-23 09:38:42,546][  INFO][PID:1262243] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9311508-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-07-23 09:38:44,580][  INFO][PID:1262243] Downloading the builder-live.log file, attempt 1
[2025-07-23 09:38:44,581][  INFO][PID:1262243] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c249 copr-rpmbuild-log
[2025-07-23 09:38:49,580][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:54,581][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:38:59,582][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:39:04,583][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:39:09,584][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:39:14,586][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:39:19,587][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:39:24,588][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:39:29,589][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:39:34,590][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:39:39,591][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:39:44,592][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:39:49,593][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:39:54,595][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:39:59,596][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:40:04,597][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:40:09,598][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:40:14,599][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:40:19,604][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:40:24,608][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:40:29,610][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:40:34,611][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:40:39,613][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:40:44,616][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:40:49,617][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:40:54,618][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:40:59,639][  INFO][PID:1262243] Periodic builder liveness probe: alive
[2025-07-23 09:40:59,643][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:41:04,649][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:41:09,655][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:41:14,656][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:41:19,658][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:41:24,659][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:41:29,660][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:41:34,662][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:41:39,665][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:41:44,668][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:41:49,669][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:41:54,675][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:41:59,676][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:42:04,678][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:42:09,679][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:42:14,680][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:42:19,683][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:42:24,684][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:42:29,685][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:42:34,689][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:42:39,694][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:42:44,696][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:42:49,697][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:42:54,698][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:42:59,699][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:43:04,702][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:43:09,705][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:43:14,710][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:43:19,712][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:43:24,714][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:43:29,716][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:43:34,717][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:43:39,718][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:43:44,721][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:43:49,723][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:43:54,724][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:43:59,725][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:44:04,726][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:44:09,728][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:44:14,732][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:44:19,735][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:44:24,737][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:44:29,739][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:44:34,740][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:44:39,741][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:44:44,742][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:44:49,744][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:44:54,745][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:44:59,746][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:45:04,747][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:45:09,751][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:45:14,753][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:45:19,754][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:45:24,756][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:45:29,757][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:45:34,758][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:45:39,760][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:45:44,761][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:45:49,762][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:45:54,763][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:45:59,764][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:46:04,766][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:46:09,767][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:46:14,769][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:46:19,771][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:46:24,773][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:46:29,774][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:46:34,775][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:46:39,776][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:46:44,778][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:46:49,779][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:46:54,780][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:46:59,781][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:47:04,782][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:47:09,783][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:47:14,784][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:47:19,785][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:47:24,786][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:47:29,789][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:47:34,790][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:47:39,791][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:47:44,792][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:47:49,793][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:47:54,794][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:47:59,795][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:48:04,796][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:48:09,797][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:48:14,799][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:48:19,800][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:48:24,801][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:48:29,804][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:48:34,805][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:48:39,806][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:48:44,807][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:48:49,809][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:48:54,812][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:48:59,813][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:49:04,814][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:49:09,817][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:49:14,818][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:49:19,822][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:49:24,827][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:49:29,832][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:49:34,836][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:49:39,838][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:49:44,839][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:49:49,841][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:49:54,843][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:49:59,845][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:50:04,848][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:50:09,850][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:50:14,854][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:50:19,855][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:50:24,859][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:50:29,865][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:50:34,868][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:50:39,871][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:50:44,876][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:50:49,880][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:50:54,882][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:50:59,920][  INFO][PID:1262243] Periodic builder liveness probe: alive
[2025-07-23 09:50:59,923][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:51:04,933][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:51:09,936][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:51:14,947][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:51:19,963][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:51:24,966][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:51:29,968][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:51:34,971][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:51:39,973][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:51:44,978][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:51:49,981][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:51:54,986][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:51:59,988][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:52:04,992][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:52:09,993][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:52:14,995][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:52:19,997][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:52:24,999][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:52:30,003][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:52:35,004][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:52:40,005][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:52:45,007][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:52:50,008][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:52:55,010][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:53:00,012][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:53:05,015][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:53:10,017][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:53:15,018][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:53:20,019][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:53:25,020][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:53:30,021][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:53:35,022][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:53:40,024][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:53:45,030][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:53:50,031][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:53:55,032][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:54:00,033][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:54:05,035][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:54:10,036][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:54:15,037][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:54:20,038][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:54:25,039][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:54:30,041][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:54:35,042][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:54:40,044][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:54:45,046][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:54:50,047][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:54:55,048][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:55:00,049][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:55:05,050][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:55:10,051][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:55:15,052][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:55:20,054][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:55:25,055][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:55:30,056][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:55:35,057][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:55:40,058][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:55:45,059][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:55:50,060][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:55:55,062][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:56:00,063][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:56:05,065][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:56:10,066][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:56:15,068][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:56:20,073][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:56:25,074][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:56:30,078][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:56:35,080][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:56:40,081][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:56:45,082][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:56:50,084][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:56:55,085][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:57:00,086][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:57:05,087][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:57:10,088][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:57:15,089][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:57:20,091][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:57:25,092][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:57:30,094][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:57:35,095][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:57:40,096][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:57:45,097][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:57:50,098][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:57:55,099][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:58:00,101][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:58:05,102][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:58:10,104][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:58:15,107][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:58:20,114][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:58:25,115][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:58:30,117][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:58:35,118][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:58:40,120][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:58:45,121][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:58:50,123][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:58:55,127][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:59:00,128][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:59:05,129][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:59:10,131][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:59:15,133][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:59:20,135][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:59:25,136][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:59:30,138][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:59:35,146][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:59:40,151][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:59:45,152][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:59:50,153][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 09:59:55,155][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:00:00,156][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:00:05,157][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:00:10,159][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:00:15,160][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:00:20,162][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:00:25,165][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:00:30,168][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:00:35,169][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:00:40,171][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:00:45,172][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:00:50,174][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:00:55,176][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:01:00,183][  INFO][PID:1262243] Periodic builder liveness probe: alive
[2025-07-23 10:01:00,184][  INFO][PID:1262243] Checking for cancel request
[2025-07-23 10:01:03,763][  INFO][PID:1262243] Downloading results from builder
[2025-07-23 10:01:03,763][  INFO][PID:1262243] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c249]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff started
[2025-07-23 10:01:03,764][  INFO][PID:1262243] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c249]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/ &> /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/build-09311508.rsync.log
[2025-07-23 10:01:27,513][  INFO][PID:1262243] rsyncing finished.
[2025-07-23 10:01:27,513][  INFO][PID:1262243] Releasing VM back to pool
[2025-07-23 10:01:27,553][  INFO][PID:1262243] Searching for 'success' file in resultdir
[2025-07-23 10:01:27,554][  INFO][PID:1262243] Going to sign pkgs from source: 9311508-fedora-rawhide-ppc64le in chroot: /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le
[2025-07-23 10:01:27,555][  INFO][PID:1262243] Calling '/bin/sign -u dreibh#ppa@copr.fedorahosted.org -p' (attempt #1)
[2025-07-23 10:01:27,884][  INFO][PID:1262243] Calling '/bin/sign -4 -h sha256 -u dreibh#ppa@copr.fedorahosted.org -r /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/neat-stuff-development-1.2.4-1.noarch.rpm' (attempt #1)
[2025-07-23 10:01:28,644][  INFO][PID:1262243] signed rpm: /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/neat-stuff-development-1.2.4-1.noarch.rpm
[2025-07-23 10:01:28,645][  INFO][PID:1262243] Calling '/bin/sign -4 -h sha256 -u dreibh#ppa@copr.fedorahosted.org -r /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/neat-stuff-1.2.4-1.src.rpm' (attempt #1)
[2025-07-23 10:01:30,263][  INFO][PID:1262243] signed rpm: /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/neat-stuff-1.2.4-1.src.rpm
[2025-07-23 10:01:30,264][  INFO][PID:1262243] Calling '/bin/sign -4 -h sha256 -u dreibh#ppa@copr.fedorahosted.org -r /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/neat-stuff-desktop-1.2.4-1.noarch.rpm' (attempt #1)
[2025-07-23 10:01:36,711][  INFO][PID:1262243] signed rpm: /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/neat-stuff-desktop-1.2.4-1.noarch.rpm
[2025-07-23 10:01:36,712][  INFO][PID:1262243] Calling '/bin/sign -4 -h sha256 -u dreibh#ppa@copr.fedorahosted.org -r /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/neat-stuff-management-1.2.4-1.noarch.rpm' (attempt #1)
[2025-07-23 10:01:37,775][  INFO][PID:1262243] signed rpm: /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/neat-stuff-management-1.2.4-1.noarch.rpm
[2025-07-23 10:01:37,777][  INFO][PID:1262243] Sign done
[2025-07-23 10:01:37,777][  INFO][PID:1262243] Incremental createrepo run, adding 09311508-neat-stuff into https://download.copr.fedorainfracloud.org/results/dreibh/ppa/fedora-rawhide-ppc64le, (auto-create-repo=True)
[2025-07-23 10:01:37,779][  INFO][PID:1262243] Running command 'copr-repo --batched /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le --add 09311508-neat-stuff --no-appstream-metadata' as PID 1604148
[2025-07-23 10:01:39,965][  INFO][PID:1262243] Finished after 2 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le --add 09311508-neat-stuff --no-appstream-metadata)
[2025-07-23 10:01:39,965][  INFO][PID:1262243] Getting build details
[2025-07-23 10:01:39,967][  INFO][PID:1262243] Listing built binary packages in /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff
[2025-07-23 10:01:39,967][  INFO][PID:1262243] Built packages:
['neat-stuff-desktop 1.2.4', 'neat-stuff-management 1.2.4', 'neat-stuff-development 1.2.4']
[2025-07-23 10:01:39,968][  INFO][PID:1262243] build details: {'built_packages': 'neat-stuff-desktop 1.2.4\nneat-stuff-management 1.2.4\nneat-stuff-development 1.2.4'}
[2025-07-23 10:01:39,968][  INFO][PID:1262243] Retrieving pubkey
[2025-07-23 10:01:39,968][  INFO][PID:1262243] Calling '/bin/sign -u dreibh#ppa@copr.fedorahosted.org -p' (attempt #1)
[2025-07-23 10:01:40,380][  INFO][PID:1262243] Added pubkey for user dreibh project ppa into: /var/lib/copr/public_html/results/dreibh/ppa/pubkey.gpg
[2025-07-23 10:01:40,381][  INFO][PID:1262243] Finished build: id=9311508 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/dreibh/ppa chroot=fedora-rawhide-ppc64le 
[2025-07-23 10:01:40,383][  INFO][PID:1262243] Worker succeeded build, took 1377.837427854538
[2025-07-23 10:01:40,384][  INFO][PID:1262243] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "dreibh",
            "project_name": "ppa",
            "project_dirname": "ppa",
            "submitter": "dreibh",
            "ended_on": 1753264900.383626,
            "started_on": 1753263522.5461981,
            "submitted_on": null,
            "status": 1,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9311508-fedora-rawhide-ppc64le",
            "build_id": 9311508,
            "package_name": "neat-stuff",
            "package_version": "1.2.4-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/dreibh/ppa/neat-stuff",
            "git_hash": "634619ea2ddc20ea128a40cdb11e78fd5221dceb",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "dreibh/ppa--dreibh",
            "results": {
                "packages": [
                    {
                        "name": "neat-stuff-desktop",
                        "epoch": null,
                        "version": "1.2.4",
                        "release": "1",
                        "arch": "noarch"
                    },
                    {
                        "name": "neat-stuff-management",
                        "epoch": null,
                        "version": "1.2.4",
                        "release": "1",
                        "arch": "noarch"
                    },
                    {
                        "name": "neat-stuff-development",
                        "epoch": null,
                        "version": "1.2.4",
                        "release": "1",
                        "arch": "noarch"
                    },
                    {
                        "name": "neat-stuff",
                        "epoch": null,
                        "version": "1.2.4",
                        "release": "1",
                        "arch": "src"
                    }
                ]
            },
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/dreibh/ppa/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/dreibh/ppa",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/dreibh/ppa",
            "result_dir": "09311508-neat-stuff",
            "built_packages": "neat-stuff-desktop 1.2.4\nneat-stuff-management 1.2.4\nneat-stuff-development 1.2.4",
            "id": 9311508,
            "mockchain_macros": {
                "copr_username": "dreibh",
                "copr_projectname": "ppa",
                "vendor": "Fedora Project COPR (dreibh/ppa)"
            }
        }
    ]
}
[2025-07-23 10:01:40,421][  INFO][PID:1262243] Sending fedora-messaging bus message in build.end
[2025-07-23 10:01:40,531][  INFO][PID:1262243] Compressing /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/builder-live.log by gzip
[2025-07-23 10:01:40,532][  INFO][PID:1262243] Running command 'gzip /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/builder-live.log' as PID 1604523
[2025-07-23 10:01:40,581][  INFO][PID:1262243] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/builder-live.log)
[2025-07-23 10:01:40,582][  INFO][PID:1262243] Compressing /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/backend.log by gzip
[2025-07-23 10:01:40,583][  INFO][PID:1262243] Running command 'gzip /var/lib/copr/public_html/results/dreibh/ppa/fedora-rawhide-ppc64le/09311508-neat-stuff/backend.log' as PID 1604529