[2025-07-26 02:52:55,416][ INFO][PID:3142408] Marking build as starting [2025-07-26 02:52:55,520][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:52:55,521][ INFO][PID:3142408] VM allocation process starts [2025-07-26 02:52:55,540][ INFO][PID:3142408] Trying to allocate VM: ResallocHost, ticket_id=4170316, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-07-26 02:53:00,542][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:53:05,544][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:53:10,545][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:53:15,546][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:53:20,547][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:53:25,548][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:53:30,549][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:53:35,550][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:53:40,552][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:53:45,553][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:53:50,554][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:53:55,555][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:54:00,556][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:54:05,557][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:54:10,559][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:54:15,560][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:54:20,561][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:54:25,562][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:54:30,563][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:54:35,564][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:54:40,565][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:54:45,566][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:54:50,567][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:54:55,569][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:55:00,573][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:55:05,575][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:55:10,576][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:55:15,577][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:55:20,578][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:55:25,579][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:55:30,580][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:55:35,581][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:55:40,582][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:55:45,583][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:55:50,584][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:55:55,585][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:56:00,586][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:56:05,587][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:56:10,589][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:56:15,590][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:56:20,591][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:56:25,592][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:56:30,593][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:56:35,597][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:56:40,598][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:56:45,599][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:56:50,600][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:56:55,602][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:57:00,603][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:57:05,604][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:57:10,605][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:57:15,606][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:57:20,608][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:57:25,609][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:57:30,610][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:57:35,611][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:57:40,612][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:57:45,613][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:57:50,614][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:57:55,615][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:58:00,616][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:58:05,617][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:58:10,619][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:58:15,620][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:58:20,621][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:58:25,622][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:58:30,624][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:58:35,625][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:58:40,626][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:58:45,627][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:58:50,628][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:58:55,629][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:59:00,630][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:59:05,631][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:59:10,632][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:59:15,633][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:59:20,634][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:59:25,635][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:59:30,636][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:59:35,637][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:59:40,639][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:59:45,641][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:59:50,642][ INFO][PID:3142408] Checking for cancel request [2025-07-26 02:59:55,643][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:00:00,644][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:00:05,645][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:00:10,647][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:00:15,648][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:00:20,649][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:00:25,650][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:00:30,651][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:00:35,652][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:00:40,653][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:00:45,654][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:00:50,655][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:00:55,656][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:01:00,658][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:01:05,659][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:01:10,660][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:01:15,661][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:01:20,663][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:01:25,664][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:01:30,665][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:01:35,666][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:01:40,668][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:01:45,669][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:01:50,670][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:01:55,671][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:02:00,672][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:02:05,673][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:02:10,674][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:02:15,676][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:02:20,678][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:02:25,679][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:02:30,680][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:02:35,682][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:02:40,683][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:02:45,684][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:02:50,685][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:02:55,686][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:03:00,687][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:03:05,689][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:03:10,690][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:03:13,742][ INFO][PID:3142408] Allocated host ResallocHost, ticket_id=4170316, hostname=140.211.168.83, name=copr_osuosl_p09_prod_05889635_20250726_025948, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-07-26 03:03:13,743][ INFO][PID:3142408] Allocating ssh connection to builder [2025-07-26 03:03:13,744][ INFO][PID:3142408] Checking that builder machine is OK [2025-07-26 03:03:15,173][ INFO][PID:3142408] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-07-26 03:03:16,172][ INFO][PID:3142408] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-07-26 03:03:16,172][ INFO][PID:3142408] Filling build.info file with builder info [2025-07-26 03:03:16,173][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:03:16,174][ INFO][PID:3142408] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "rpmsoftwaremanagement", "project_name": "dnf-nightly", "project_dirname": "dnf-nightly", "submitter": "rpmsoftwaremanagement", "ended_on": null, "started_on": 1753498996.1746833, "submitted_on": null, "status": 3, "chroot": "fedora-41-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9339915-fedora-41-ppc64le", "build_id": 9339915, "package_name": "dnf5", "package_version": "5.2.15.0-20250726011825.22.g2463666a", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rpmsoftwaremanagement/dnf-nightly/dnf5", "git_hash": "98502a8370bf1c36f829586ef008d5b905d6a4bf", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "rpmsoftwaremanagement/dnf-nightly--rpmsoftwaremanagement", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rpmsoftwaremanagement/dnf-nightly", "result_dir": "09339915-dnf5", "built_packages": "", "id": 9339915, "mockchain_macros": { "copr_username": "rpmsoftwaremanagement", "copr_projectname": "dnf-nightly", "vendor": "Fedora Project COPR (rpmsoftwaremanagement/dnf-nightly)" } } ] } [2025-07-26 03:03:16,428][ INFO][PID:3142408] Sending fedora-messaging bus message in build.start [2025-07-26 03:03:17,139][ INFO][PID:3142408] Sending fedora-messaging bus message in chroot.start [2025-07-26 03:03:17,164][ INFO][PID:3142408] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9339915-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-07-26 03:03:18,517][ INFO][PID:3142408] Downloading the builder-live.log file, attempt 1 [2025-07-26 03:03:18,519][ INFO][PID:3142408] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.83 copr-rpmbuild-log [2025-07-26 03:03:23,530][ INFO][PID:3142408] Periodic builder liveness probe: alive [2025-07-26 03:03:23,531][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:03:28,532][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:03:33,533][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:03:38,534][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:03:43,535][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:03:48,537][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:03:53,539][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:03:58,540][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:04:03,541][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:04:08,542][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:04:13,543][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:04:18,544][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:04:23,546][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:04:28,547][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:04:33,548][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:04:38,549][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:04:43,550][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:04:48,552][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:04:53,553][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:04:58,554][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:05:03,557][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:05:08,558][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:05:13,559][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:05:18,560][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:05:23,561][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:05:28,562][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:05:33,564][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:05:38,565][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:05:43,566][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:05:48,567][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:05:53,568][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:05:58,569][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:06:03,571][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:06:08,574][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:06:13,575][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:06:18,577][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:06:23,579][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:06:28,584][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:06:33,585][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:06:38,586][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:06:43,588][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:06:48,589][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:06:53,591][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:06:58,592][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:07:03,594][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:07:08,595][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:07:13,596][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:07:18,597][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:07:23,599][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:07:28,601][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:07:33,603][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:07:38,607][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:07:43,608][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:07:48,609][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:07:53,610][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:07:58,611][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:08:03,614][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:08:08,615][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:08:13,616][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:08:18,617][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:08:23,618][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:08:28,619][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:08:33,621][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:08:38,622][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:08:43,625][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:08:48,626][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:08:53,627][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:08:58,628][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:09:03,629][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:09:08,630][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:09:13,633][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:09:18,640][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:09:23,642][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:09:28,643][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:09:33,644][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:09:38,645][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:09:43,646][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:09:48,647][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:09:53,650][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:09:58,654][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:10:03,655][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:10:08,656][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:10:13,657][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:10:18,659][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:10:23,660][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:10:28,661][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:10:33,662][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:10:38,663][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:10:43,664][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:10:48,666][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:10:53,669][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:10:58,671][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:11:03,672][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:11:08,673][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:11:13,674][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:11:18,675][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:11:23,677][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:11:28,678][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:11:33,679][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:11:38,680][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:11:43,681][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:11:48,683][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:11:53,684][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:11:58,685][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:12:03,686][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:12:08,688][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:12:13,690][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:12:18,692][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:12:23,693][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:12:28,695][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:12:33,696][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:12:38,697][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:12:43,698][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:12:48,699][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:12:53,701][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:12:58,703][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:13:03,704][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:13:08,705][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:13:13,709][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:13:18,710][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:13:23,724][ INFO][PID:3142408] Periodic builder liveness probe: alive [2025-07-26 03:13:23,725][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:13:28,726][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:13:33,727][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:13:38,729][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:13:43,730][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:13:48,731][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:13:53,732][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:13:58,733][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:14:03,734][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:14:08,735][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:14:13,736][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:14:18,739][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:14:23,740][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:14:28,741][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:14:33,742][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:14:38,743][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:14:43,744][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:14:48,746][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:14:53,747][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:14:58,748][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:15:03,749][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:15:08,750][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:15:13,752][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:15:18,753][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:15:23,754][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:15:28,755][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:15:33,757][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:15:38,758][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:15:43,759][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:15:48,761][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:15:53,762][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:15:58,763][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:16:03,765][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:16:08,767][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:16:13,768][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:16:18,769][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:16:23,770][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:16:28,771][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:16:33,772][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:16:38,773][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:16:43,774][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:16:48,775][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:16:53,776][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:16:58,777][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:17:03,778][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:17:08,779][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:17:13,780][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:17:18,782][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:17:23,783][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:17:28,784][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:17:33,785][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:17:38,786][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:17:43,787][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:17:48,788][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:17:53,789][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:17:58,791][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:18:03,792][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:18:08,793][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:18:13,794][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:18:18,795][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:18:23,796][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:18:28,797][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:18:33,799][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:18:38,801][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:18:43,802][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:18:48,803][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:18:53,804][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:18:58,805][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:19:03,806][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:19:08,807][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:19:13,808][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:19:18,809][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:19:23,813][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:19:28,815][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:19:33,817][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:19:38,819][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:19:43,821][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:19:48,822][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:19:53,823][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:19:58,824][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:20:03,825][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:20:08,826][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:20:13,828][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:20:18,829][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:20:23,830][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:20:28,831][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:20:33,832][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:20:38,833][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:20:43,834][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:20:48,835][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:20:53,836][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:20:58,837][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:21:03,838][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:21:08,843][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:21:13,844][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:21:18,845][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:21:23,846][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:21:28,847][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:21:33,850][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:21:38,852][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:21:43,854][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:21:48,856][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:21:53,858][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:21:58,859][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:22:03,860][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:22:08,861][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:22:13,862][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:22:18,864][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:22:23,865][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:22:28,866][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:22:33,867][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:22:38,868][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:22:43,869][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:22:48,871][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:22:53,872][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:22:58,873][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:23:03,874][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:23:08,875][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:23:13,877][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:23:18,878][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:23:23,884][ INFO][PID:3142408] Periodic builder liveness probe: alive [2025-07-26 03:23:23,885][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:23:28,886][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:23:33,888][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:23:38,890][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:23:43,891][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:23:48,893][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:23:53,894][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:23:58,896][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:24:03,897][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:24:08,898][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:24:13,900][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:24:18,901][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:24:23,902][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:24:28,903][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:24:33,906][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:24:38,907][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:24:43,908][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:24:48,910][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:24:53,911][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:24:58,913][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:25:03,918][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:25:08,919][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:25:13,921][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:25:18,926][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:25:23,927][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:25:28,929][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:25:33,933][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:25:38,941][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:25:43,942][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:25:48,944][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:25:53,945][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:25:58,946][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:26:03,947][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:26:08,949][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:26:13,951][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:26:18,953][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:26:23,955][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:26:28,960][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:26:33,962][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:26:38,964][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:26:43,977][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:26:49,049][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:26:54,051][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:26:59,052][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:27:04,053][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:27:09,054][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:27:14,055][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:27:19,057][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:27:24,058][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:27:29,059][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:27:34,061][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:27:39,062][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:27:44,063][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:27:49,064][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:27:54,067][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:27:59,068][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:28:04,069][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:28:09,070][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:28:14,071][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:28:19,072][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:28:24,208][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:28:29,209][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:28:34,211][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:28:39,212][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:28:44,214][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:28:49,215][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:28:54,216][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:28:59,217][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:29:04,275][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:29:09,276][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:29:14,277][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:29:19,330][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:29:24,331][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:29:29,332][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:29:34,399][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:29:39,400][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:29:44,401][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:29:49,403][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:29:54,404][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:29:59,405][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:30:04,406][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:30:09,430][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:30:14,431][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:30:19,436][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:30:24,437][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:30:29,439][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:30:34,440][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:30:39,445][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:30:44,446][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:30:49,447][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:30:54,448][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:30:59,450][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:31:04,451][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:31:09,456][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:31:14,463][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:31:19,465][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:31:24,469][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:31:29,471][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:31:34,595][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:31:39,598][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:31:44,601][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:31:49,603][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:31:54,605][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:31:59,607][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:32:04,608][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:32:09,610][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:32:14,611][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:32:19,612][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:32:24,613][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:32:29,615][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:32:34,616][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:32:39,617][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:32:44,618][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:32:49,619][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:32:54,620][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:32:59,621][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:33:04,623][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:33:09,624][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:33:14,626][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:33:19,627][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:33:24,636][ INFO][PID:3142408] Periodic builder liveness probe: alive [2025-07-26 03:33:24,637][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:33:29,638][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:33:34,639][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:33:39,641][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:33:44,645][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:33:49,646][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:33:54,648][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:33:59,649][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:34:04,650][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:34:09,651][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:34:14,652][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:34:19,653][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:34:24,654][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:34:29,655][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:34:34,656][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:34:39,657][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:34:44,658][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:34:49,659][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:34:54,661][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:34:59,662][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:35:04,665][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:35:09,666][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:35:14,667][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:35:19,668][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:35:24,669][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:35:29,670][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:35:34,671][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:35:39,674][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:35:44,675][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:35:49,676][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:35:54,677][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:35:59,677][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:36:04,678][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:36:09,679][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:36:14,681][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:36:19,682][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:36:24,683][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:36:29,685][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:36:34,688][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:36:39,689][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:36:44,697][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:36:49,698][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:36:54,699][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:36:59,700][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:37:04,701][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:37:09,702][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:37:14,703][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:37:19,704][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:37:24,705][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:37:29,706][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:37:34,707][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:37:39,708][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:37:44,709][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:37:49,139][ ERROR][PID:3142408] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-26 03:37:49,710][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:37:54,711][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:37:59,219][WARNING][PID:3142408] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-26 03:37:59,715][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:38:04,717][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:38:09,219][ INFO][PID:3142408] Downloading the builder-live.log file, attempt 2 [2025-07-26 03:38:09,221][ INFO][PID:3142408] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.83 copr-rpmbuild-log [2025-07-26 03:38:09,718][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:38:14,739][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:38:19,743][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:38:24,744][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:38:29,746][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:38:34,747][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:38:39,748][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:38:44,749][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:38:49,751][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:38:54,753][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:39:00,011][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:39:05,082][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:39:09,292][ ERROR][PID:3142408] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-26 03:39:10,119][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:39:15,122][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:39:19,296][WARNING][PID:3142408] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-26 03:39:20,125][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:39:25,130][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:39:29,297][ INFO][PID:3142408] Downloading the builder-live.log file, attempt 3 [2025-07-26 03:39:29,298][ INFO][PID:3142408] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.83 copr-rpmbuild-log [2025-07-26 03:39:30,134][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:39:35,138][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:39:40,141][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:39:45,145][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:39:50,151][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:39:55,182][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:40:00,188][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:40:05,190][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:40:10,193][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:40:15,194][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:40:20,196][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:40:25,197][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:40:29,366][ ERROR][PID:3142408] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-26 03:40:30,198][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:40:35,216][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:40:39,369][WARNING][PID:3142408] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-26 03:40:40,217][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:40:45,218][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:40:49,370][ INFO][PID:3142408] Downloading the builder-live.log file, attempt 4 [2025-07-26 03:40:49,371][ INFO][PID:3142408] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.83 copr-rpmbuild-log [2025-07-26 03:40:50,221][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:40:55,222][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:00,223][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:05,224][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:10,225][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:15,226][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:20,227][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:25,228][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:30,229][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:35,231][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:40,232][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:45,233][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:49,439][ ERROR][PID:3142408] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-26 03:41:50,234][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:55,235][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:59,441][ ERROR][PID:3142408] Giving up for unstable SSH, failures: 1753501079.2191107, 1753501159.2969344, 1753501239.369805, 1753501319.4409008 [2025-07-26 03:41:59,442][ ERROR][PID:3142408] Re-try request for task on 'ResallocHost, ticket_id=4170316, hostname=140.211.168.83, name=copr_osuosl_p09_prod_05889635_20250726_025948, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-07-26 03:41:59,442][ INFO][PID:3142408] Releasing VM back to pool [2025-07-26 03:41:59,451][ INFO][PID:3142408] Retry #1 (on other host) [2025-07-26 03:41:59,452][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:41:59,453][ INFO][PID:3142408] VM allocation process starts [2025-07-26 03:41:59,492][ INFO][PID:3142408] Trying to allocate VM: ResallocHost, ticket_id=4170858, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-07-26 03:42:04,493][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:42:09,494][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:42:14,495][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:42:19,496][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:42:24,497][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:42:29,498][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:42:34,499][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:42:39,500][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:42:44,501][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:42:49,502][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:42:54,504][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:42:59,505][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:43:04,506][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:43:09,507][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:43:14,508][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:43:19,510][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:43:24,511][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:43:29,512][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:43:34,514][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:43:39,516][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:43:44,517][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:43:49,518][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:43:54,522][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:43:59,528][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:44:04,531][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:44:09,534][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:44:14,539][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:44:19,540][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:44:24,547][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:44:29,550][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:44:34,554][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:44:39,556][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:44:44,559][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:44:49,561][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:44:54,563][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:44:59,564][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:45:04,566][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:45:09,573][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:45:14,577][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:45:17,698][ INFO][PID:3142408] Allocated host ResallocHost, ticket_id=4170858, hostname=140.211.168.199, name=copr_osuosl_p09_prod_05890017_20250726_034131, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-07-26 03:45:17,700][ INFO][PID:3142408] Allocating ssh connection to builder [2025-07-26 03:45:17,702][ INFO][PID:3142408] Checking that builder machine is OK [2025-07-26 03:45:19,280][ INFO][PID:3142408] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-07-26 03:45:20,428][ INFO][PID:3142408] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-07-26 03:45:20,430][ INFO][PID:3142408] Filling build.info file with builder info [2025-07-26 03:45:20,437][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:45:20,441][ INFO][PID:3142408] Not re-notifying FE and msg buses for the new host. [2025-07-26 03:45:20,445][ INFO][PID:3142408] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9339915-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-07-26 03:45:21,875][ INFO][PID:3142408] Downloading the builder-live.log file, attempt 1 [2025-07-26 03:45:21,877][ INFO][PID:3142408] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.199 copr-rpmbuild-log [2025-07-26 03:45:26,890][ INFO][PID:3142408] Periodic builder liveness probe: alive [2025-07-26 03:45:26,894][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:45:31,896][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:45:36,897][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:45:41,901][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:45:46,902][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:45:51,907][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:45:56,912][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:46:01,918][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:46:06,920][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:46:11,921][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:46:16,927][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:46:21,928][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:46:26,932][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:46:31,946][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:46:36,947][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:46:41,953][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:46:46,961][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:46:51,967][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:46:56,969][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:47:01,973][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:47:06,979][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:47:11,988][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:47:16,996][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:47:22,004][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:47:27,008][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:47:32,010][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:47:37,015][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:47:42,018][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:47:47,020][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:47:52,027][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:47:57,029][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:48:02,039][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:48:07,041][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:48:12,047][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:48:17,052][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:48:22,055][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:48:27,057][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:48:32,058][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:48:37,059][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:48:42,061][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:48:47,062][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:48:52,066][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:48:57,067][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:49:02,068][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:49:07,073][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:49:12,074][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:49:17,075][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:49:22,076][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:49:27,077][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:49:32,079][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:49:37,080][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:49:42,081][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:49:47,083][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:49:52,085][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:49:57,086][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:50:02,087][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:50:07,088][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:50:12,089][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:50:17,091][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:50:22,092][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:50:27,094][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:50:32,096][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:50:37,097][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:50:42,098][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:50:47,100][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:50:52,101][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:50:57,103][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:51:02,104][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:51:07,109][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:51:12,111][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:51:17,112][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:51:22,113][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:51:27,114][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:51:32,115][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:51:37,116][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:51:42,117][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:51:47,119][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:51:52,120][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:51:57,121][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:52:02,123][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:52:07,124][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:52:12,125][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:52:17,127][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:52:22,127][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:52:27,128][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:52:32,129][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:52:37,131][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:52:42,132][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:52:47,133][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:52:52,134][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:52:57,135][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:53:02,136][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:53:07,137][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:53:12,138][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:53:17,140][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:53:22,141][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:53:27,142][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:53:32,143][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:53:37,144][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:53:42,146][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:53:47,147][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:53:52,149][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:53:57,150][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:54:02,151][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:54:07,153][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:54:12,154][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:54:17,155][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:54:22,156][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:54:27,161][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:54:32,163][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:54:37,166][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:54:42,167][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:54:47,168][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:54:52,169][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:54:57,170][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:55:02,172][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:55:07,176][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:55:12,177][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:55:17,178][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:55:22,181][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:55:27,191][ INFO][PID:3142408] Periodic builder liveness probe: alive [2025-07-26 03:55:27,191][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:55:32,192][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:55:37,193][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:55:42,195][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:55:47,196][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:55:52,197][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:55:57,198][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:56:02,199][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:56:07,200][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:56:12,232][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:56:17,234][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:56:22,237][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:56:27,238][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:56:32,240][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:56:37,241][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:56:42,243][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:56:47,245][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:56:52,246][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:56:57,248][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:57:02,250][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:57:07,251][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:57:12,253][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:57:17,255][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:57:22,256][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:57:27,259][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:57:32,260][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:57:37,263][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:57:42,270][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:57:47,272][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:57:52,274][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:57:57,277][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:58:02,278][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:58:07,279][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:58:12,282][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:58:17,287][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:58:22,288][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:58:27,289][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:58:32,290][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:58:37,292][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:58:42,303][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:58:47,304][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:58:52,306][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:58:57,307][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:59:02,308][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:59:07,309][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:59:12,310][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:59:17,315][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:59:22,316][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:59:27,318][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:59:32,320][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:59:37,322][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:59:42,324][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:59:47,325][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:59:52,330][ INFO][PID:3142408] Checking for cancel request [2025-07-26 03:59:57,332][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:00:02,334][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:00:07,336][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:00:12,337][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:00:17,339][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:00:22,340][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:00:27,341][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:00:32,344][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:00:37,347][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:00:42,352][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:00:47,363][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:00:52,370][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:00:57,371][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:01:02,373][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:01:07,375][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:01:12,377][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:01:17,378][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:01:22,380][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:01:27,385][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:01:32,386][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:01:37,387][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:01:42,389][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:01:47,390][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:01:52,391][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:01:57,392][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:02:02,396][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:02:07,397][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:02:12,399][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:02:17,401][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:02:22,402][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:02:27,403][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:02:32,404][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:02:37,405][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:02:42,406][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:02:47,407][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:02:52,408][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:02:57,409][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:03:02,410][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:03:07,411][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:03:12,412][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:03:17,413][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:03:22,415][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:03:27,416][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:03:32,417][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:03:37,418][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:03:42,419][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:03:47,421][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:03:52,422][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:03:57,423][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:04:02,424][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:04:07,425][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:04:11,139][ ERROR][PID:3142408] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-26 04:04:12,426][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:04:17,427][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:04:21,139][WARNING][PID:3142408] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-26 04:04:22,428][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:04:27,430][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:04:31,140][ INFO][PID:3142408] Downloading the builder-live.log file, attempt 2 [2025-07-26 04:04:31,142][ INFO][PID:3142408] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.199 copr-rpmbuild-log [2025-07-26 04:04:32,430][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:04:37,432][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:04:42,433][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:04:47,434][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:04:52,435][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:04:57,436][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:05:02,437][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:05:07,438][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:05:12,440][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:05:17,442][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:05:22,444][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:05:27,451][ INFO][PID:3142408] Periodic builder liveness probe: dead [2025-07-26 04:05:31,210][ ERROR][PID:3142408] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-26 04:05:41,222][WARNING][PID:3142408] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-26 04:05:51,223][ INFO][PID:3142408] Downloading the builder-live.log file, attempt 3 [2025-07-26 04:05:51,225][ INFO][PID:3142408] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.199 copr-rpmbuild-log [2025-07-26 04:06:27,524][ ERROR][PID:3142408] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.199 port 22: Connection timed out [2025-07-26 04:06:51,297][ ERROR][PID:3142408] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-26 04:07:01,299][WARNING][PID:3142408] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-07-26 04:07:11,300][ INFO][PID:3142408] Downloading the builder-live.log file, attempt 4 [2025-07-26 04:07:11,301][ INFO][PID:3142408] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.199 copr-rpmbuild-log [2025-07-26 04:07:37,599][ ERROR][PID:3142408] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.199 port 22: Connection timed out [2025-07-26 04:08:11,375][ ERROR][PID:3142408] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-07-26 04:08:21,574][ ERROR][PID:3142408] Giving up for unstable SSH, failures: 1753502661.1398337, 1753502741.2228596, 1753502821.299094, 1753502901.574468 [2025-07-26 04:08:47,674][ ERROR][PID:3142408] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.199 port 22: Connection timed out [2025-07-26 04:09:57,763][ ERROR][PID:3142408] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.199 port 22: Connection timed out [2025-07-26 04:10:07,764][ ERROR][PID:3142408] Can't ssh to cancel build. [2025-07-26 04:10:07,766][ ERROR][PID:3142408] Re-try request for task on 'ResallocHost, ticket_id=4170858, hostname=140.211.168.199, name=copr_osuosl_p09_prod_05890017_20250726_034131, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED [2025-07-26 04:10:07,767][ INFO][PID:3142408] Releasing VM back to pool [2025-07-26 04:10:07,784][ INFO][PID:3142408] Retry #2 (on other host) [2025-07-26 04:10:07,785][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:10:07,787][ INFO][PID:3142408] VM allocation process starts [2025-07-26 04:10:07,802][ INFO][PID:3142408] Trying to allocate VM: ResallocHost, ticket_id=4171245, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-07-26 04:10:10,826][ INFO][PID:3142408] Allocated host ResallocHost, ticket_id=4171245, hostname=140.211.168.14, name=copr_osuosl_p09_prod_05890135_20250726_040243, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-07-26 04:10:10,827][ INFO][PID:3142408] Allocating ssh connection to builder [2025-07-26 04:10:10,829][ INFO][PID:3142408] Checking that builder machine is OK [2025-07-26 04:10:12,117][ INFO][PID:3142408] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-07-26 04:10:13,169][ INFO][PID:3142408] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-07-26 04:10:13,169][ INFO][PID:3142408] Filling build.info file with builder info [2025-07-26 04:10:13,173][ INFO][PID:3142408] Checking for cancel request [2025-07-26 04:10:13,178][ INFO][PID:3142408] Not re-notifying FE and msg buses for the new host. [2025-07-26 04:10:13,179][ INFO][PID:3142408] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9339915-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-07-26 04:10:14,479][ INFO][PID:3142408] Downloading the builder-live.log file, attempt 1 [2025-07-26 04:10:14,484][ INFO][PID:3142408] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log [2025-07-26 04:10:23,601][ INFO][PID:3142408] Cancel request succeeded out: success err: Warning: Permanently added '140.211.168.14' (ED25519) to the list of known hosts. [2025-07-26 04:10:23,602][ INFO][PID:3142408] Downloading results from builder [2025-07-26 04:10:23,603][ INFO][PID:3142408] rsyncing of mockbuilder@140.211.168.14:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09339915-dnf5 started [2025-07-26 04:10:23,607][ INFO][PID:3142408] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@140.211.168.14:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09339915-dnf5/ &> /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09339915-dnf5/build-09339915.rsync.log [2025-07-26 04:10:25,242][ INFO][PID:3142408] rsyncing finished. [2025-07-26 04:10:25,242][ INFO][PID:3142408] Releasing VM back to pool [2025-07-26 04:10:25,265][ INFO][PID:3142408] Searching for 'success' file in resultdir [2025-07-26 04:10:25,266][ ERROR][PID:3142408] Build failed: Backend process error: No success file => build failure [2025-07-26 04:10:25,271][ INFO][PID:3142408] Finished build: id=9339915 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly chroot=fedora-41-ppc64le [2025-07-26 04:10:25,275][ ERROR][PID:3142408] Backend process error: No success file => build failure [2025-07-26 04:10:25,276][ INFO][PID:3142408] Worker failed build, took 12.098087310791016 [2025-07-26 04:10:25,277][ INFO][PID:3142408] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "rpmsoftwaremanagement", "project_name": "dnf-nightly", "project_dirname": "dnf-nightly", "submitter": "rpmsoftwaremanagement", "ended_on": 1753503025.2763085, "started_on": 1753503013.1782212, "submitted_on": null, "status": 0, "chroot": "fedora-41-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "9339915-fedora-41-ppc64le", "build_id": 9339915, "package_name": "dnf5", "package_version": "5.2.15.0-20250726011825.22.g2463666a", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rpmsoftwaremanagement/dnf-nightly/dnf5", "git_hash": "98502a8370bf1c36f829586ef008d5b905d6a4bf", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "rpmsoftwaremanagement/dnf-nightly--rpmsoftwaremanagement", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": false, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rpmsoftwaremanagement/dnf-nightly", "result_dir": "09339915-dnf5", "built_packages": "", "id": 9339915, "mockchain_macros": { "copr_username": "rpmsoftwaremanagement", "copr_projectname": "dnf-nightly", "vendor": "Fedora Project COPR (rpmsoftwaremanagement/dnf-nightly)" } } ] } [2025-07-26 04:10:25,368][ INFO][PID:3142408] Sending fedora-messaging bus message in build.end [2025-07-26 04:10:25,431][ INFO][PID:3142408] Compressing /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09339915-dnf5/builder-live.log by gzip [2025-07-26 04:10:25,436][ INFO][PID:3142408] Running command 'gzip /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09339915-dnf5/builder-live.log' as PID 3552161 [2025-07-26 04:10:25,438][ INFO][PID:3142408] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09339915-dnf5/builder-live.log) [2025-07-26 04:10:25,441][ INFO][PID:3142408] Compressing /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09339915-dnf5/backend.log by gzip [2025-07-26 04:10:25,456][ INFO][PID:3142408] Running command 'gzip /var/lib/copr/public_html/results/rpmsoftwaremanagement/dnf-nightly/fedora-41-ppc64le/09339915-dnf5/backend.log' as PID 3552164