[2025-08-25 15:28:21,599][ INFO][PID:2268084] Marking build as starting [2025-08-25 15:28:21,685][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:28:21,686][ INFO][PID:2268084] VM allocation process starts [2025-08-25 15:28:21,700][ INFO][PID:2268084] Trying to allocate VM: ResallocHost, ticket_id=4709977, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-25 15:28:24,728][ INFO][PID:2268084] Allocated host ResallocHost, ticket_id=4709977, hostname=34.226.139.180, name=aws_x86_64_spot_prod_06414574_20250825_152344, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-25 15:28:24,728][ INFO][PID:2268084] Allocating ssh connection to builder [2025-08-25 15:28:24,729][ INFO][PID:2268084] Checking that builder machine is OK [2025-08-25 15:28:25,153][ INFO][PID:2268084] Running remote command: copr-builder-ready fedora-41-x86_64 [2025-08-25 15:28:25,349][ INFO][PID:2268084] Red Hat subscription not needed for fedora-41-x86_64 Builder is ready to be used [2025-08-25 15:28:25,350][ INFO][PID:2268084] Filling build.info file with builder info [2025-08-25 15:28:25,350][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:28:25,351][ INFO][PID:2268084] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@CoreOS", "project_name": "continuous", "project_dirname": "continuous", "submitter": "https://api.github.com/users/cgwalters", "ended_on": null, "started_on": 1756135705.3513389, "submitted_on": null, "status": 3, "chroot": "fedora-41-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9478634-fedora-41-x86_64", "build_id": 9478634, "package_name": "rpm-ostree", "package_version": "2025.10.7.g283f0897-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@CoreOS/continuous/rpm-ostree", "git_hash": "f56e62d5c47805dc3ede39ce02509ee7e40c8bc9", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@CoreOS/continuous--https://api.github.com/users/cgwalters", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@CoreOS/continuous/fedora-41-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/fedora-41-x86_64/", "id": "copr_rhcontainerbot_bootc", "name": "Additional repo copr_rhcontainerbot_bootc" } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@CoreOS/continuous", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@CoreOS/continuous", "result_dir": "09478634-rpm-ostree", "built_packages": "", "id": 9478634, "mockchain_macros": { "copr_username": "@CoreOS", "copr_projectname": "continuous", "vendor": "Fedora Project COPR (@CoreOS/continuous)" } } ] } [2025-08-25 15:28:25,404][ INFO][PID:2268084] Sending fedora-messaging bus message in build.start [2025-08-25 15:28:26,099][ INFO][PID:2268084] Sending fedora-messaging bus message in chroot.start [2025-08-25 15:28:26,125][ INFO][PID:2268084] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9478634-fedora-41-x86_64 --chroot fedora-41-x86_64 --detached [2025-08-25 15:28:26,443][ INFO][PID:2268084] Downloading the builder-live.log file, attempt 1 [2025-08-25 15:28:26,445][ INFO][PID:2268084] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.226.139.180 copr-rpmbuild-log [2025-08-25 15:28:31,452][ INFO][PID:2268084] Periodic builder liveness probe: alive [2025-08-25 15:28:31,453][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:28:36,454][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:28:41,456][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:28:46,457][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:28:51,459][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:28:56,460][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:29:01,463][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:29:06,465][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:29:11,466][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:29:16,468][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:29:21,470][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:29:26,472][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:29:31,473][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:29:36,474][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:29:41,475][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:29:46,477][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:29:51,479][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:29:56,480][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:30:01,481][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:30:06,483][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:30:11,484][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:30:16,486][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:30:21,487][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:30:26,489][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:30:31,490][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:30:36,491][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:30:41,492][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:30:46,493][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:30:51,497][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:30:56,499][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:31:01,503][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:31:06,505][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:31:11,506][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:31:16,508][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:31:21,510][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:31:26,512][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:31:31,514][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:31:36,516][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:31:41,518][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:31:46,520][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:31:51,521][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:31:56,525][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:32:01,527][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:32:06,528][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:32:11,539][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:32:16,543][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:32:21,544][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:32:26,548][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:32:31,551][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:32:36,553][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:32:41,554][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:32:46,555][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:32:51,556][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:32:56,557][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:33:01,558][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:33:06,560][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:33:11,561][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:33:16,562][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:33:21,563][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:33:26,564][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:33:31,566][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:33:36,567][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:33:41,568][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:33:46,569][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:33:51,570][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:33:56,572][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:34:01,574][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:34:06,575][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:34:11,576][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:34:15,691][ ERROR][PID:2268084] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 15:34:16,577][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:34:21,579][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:34:25,691][WARNING][PID:2268084] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-25 15:34:26,579][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:34:31,581][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:34:35,692][ INFO][PID:2268084] Downloading the builder-live.log file, attempt 2 [2025-08-25 15:34:35,694][ INFO][PID:2268084] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.226.139.180 copr-rpmbuild-log [2025-08-25 15:34:36,582][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:34:41,583][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:34:46,585][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:34:51,586][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:34:56,588][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:35:01,590][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:35:06,591][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:35:11,592][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:35:16,593][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:35:21,595][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:35:26,596][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:35:31,598][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:35:35,767][ ERROR][PID:2268084] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 15:35:36,600][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:35:41,602][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:35:45,770][WARNING][PID:2268084] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-25 15:35:46,605][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:35:51,606][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:35:55,771][ INFO][PID:2268084] Downloading the builder-live.log file, attempt 3 [2025-08-25 15:35:55,772][ INFO][PID:2268084] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.226.139.180 copr-rpmbuild-log [2025-08-25 15:35:56,608][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:36:01,610][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:36:06,612][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:36:11,615][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:36:16,616][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:36:21,617][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:36:26,618][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:36:31,620][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:36:36,621][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:36:41,622][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:36:46,623][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:36:51,624][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:36:55,842][ ERROR][PID:2268084] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 15:36:56,625][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:37:01,627][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:37:05,844][WARNING][PID:2268084] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-08-25 15:37:06,630][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:37:11,632][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:37:15,845][ INFO][PID:2268084] Downloading the builder-live.log file, attempt 4 [2025-08-25 15:37:15,847][ INFO][PID:2268084] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.226.139.180 copr-rpmbuild-log [2025-08-25 15:37:16,634][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:37:21,635][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:37:26,637][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:37:31,641][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:37:36,642][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:37:41,644][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:37:46,646][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:37:51,648][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:37:56,649][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:38:01,650][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:38:06,651][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:38:11,653][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:38:15,915][ ERROR][PID:2268084] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-08-25 15:38:16,654][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:38:21,655][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:38:25,918][ ERROR][PID:2268084] Giving up for unstable SSH, failures: 1756136065.6918497, 1756136145.7705848, 1756136225.844185, 1756136305.9185867 [2025-08-25 15:38:25,920][ ERROR][PID:2268084] Re-try request for task on 'ResallocHost, ticket_id=4709977, hostname=34.226.139.180, name=aws_x86_64_spot_prod_06414574_20250825_152344, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-08-25 15:38:25,921][ INFO][PID:2268084] Releasing VM back to pool [2025-08-25 15:38:25,938][ INFO][PID:2268084] Retry #1 (on other host) [2025-08-25 15:38:25,939][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:38:25,941][ INFO][PID:2268084] VM allocation process starts [2025-08-25 15:38:25,957][ INFO][PID:2268084] Trying to allocate VM: ResallocHost, ticket_id=4710219, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-25 15:38:30,959][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:38:35,960][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:38:38,133][ INFO][PID:2268084] Allocated host ResallocHost, ticket_id=4710219, hostname=3.80.206.6, name=aws_x86_64_spot_prod_06414056_20250825_150208, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-25 15:38:38,134][ INFO][PID:2268084] Allocating ssh connection to builder [2025-08-25 15:38:38,134][ INFO][PID:2268084] Checking that builder machine is OK [2025-08-25 15:39:38,205][ ERROR][PID:2268084] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 3.80.206.6 port 22: Connection timed out [2025-08-25 15:39:48,206][ ERROR][PID:2268084] Re-try request for task on 'ResallocHost, ticket_id=4710219, hostname=3.80.206.6, name=aws_x86_64_spot_prod_06414056_20250825_150208, requested_tags=['copr_builder', 'arch_x86_64']': Unable to finish after 1 SSH attempts [2025-08-25 15:39:48,206][ INFO][PID:2268084] Releasing VM back to pool [2025-08-25 15:39:48,243][ INFO][PID:2268084] Retry #2 (on other host) [2025-08-25 15:39:48,244][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:39:48,244][ INFO][PID:2268084] VM allocation process starts [2025-08-25 15:39:48,257][ INFO][PID:2268084] Trying to allocate VM: ResallocHost, ticket_id=4710243, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-25 15:39:51,330][ INFO][PID:2268084] Allocated host ResallocHost, ticket_id=4710243, hostname=3.85.49.107, name=aws_x86_64_spot_prod_06414303_20250825_151332, requested_tags=['copr_builder', 'arch_x86_64'] [2025-08-25 15:39:51,331][ INFO][PID:2268084] Allocating ssh connection to builder [2025-08-25 15:39:51,331][ INFO][PID:2268084] Checking that builder machine is OK [2025-08-25 15:39:51,818][ INFO][PID:2268084] Running remote command: copr-builder-ready fedora-41-x86_64 [2025-08-25 15:39:52,011][ INFO][PID:2268084] Red Hat subscription not needed for fedora-41-x86_64 Builder is ready to be used [2025-08-25 15:39:52,011][ INFO][PID:2268084] Filling build.info file with builder info [2025-08-25 15:39:52,015][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:39:52,017][ INFO][PID:2268084] Not re-notifying FE and msg buses for the new host. [2025-08-25 15:39:52,017][ INFO][PID:2268084] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9478634-fedora-41-x86_64 --chroot fedora-41-x86_64 --detached [2025-08-25 15:39:52,335][ INFO][PID:2268084] Downloading the builder-live.log file, attempt 1 [2025-08-25 15:39:52,336][ INFO][PID:2268084] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.85.49.107 copr-rpmbuild-log [2025-08-25 15:39:57,350][ INFO][PID:2268084] Periodic builder liveness probe: alive [2025-08-25 15:39:57,354][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:40:02,356][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:40:07,357][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:40:12,358][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:40:17,359][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:40:22,361][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:40:27,362][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:40:32,364][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:40:37,365][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:40:42,369][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:40:47,370][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:40:52,371][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:40:57,373][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:41:02,374][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:41:07,375][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:41:12,376][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:41:17,377][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:41:22,378][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:41:27,379][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:41:32,381][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:41:37,382][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:41:42,385][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:41:47,386][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:41:52,388][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:41:57,389][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:42:02,390][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:42:07,391][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:42:12,392][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:42:17,394][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:42:22,395][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:42:27,397][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:42:32,398][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:42:37,400][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:42:42,404][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:42:47,409][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:42:52,410][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:42:57,411][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:43:02,416][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:43:07,420][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:43:12,426][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:43:17,427][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:43:22,430][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:43:27,431][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:43:32,433][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:43:37,438][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:43:42,444][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:43:47,446][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:43:52,449][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:43:57,451][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:44:02,452][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:44:07,454][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:44:12,455][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:44:17,458][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:44:22,459][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:44:27,461][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:44:32,463][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:44:37,465][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:44:42,466][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:44:47,467][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:44:52,468][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:44:57,470][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:45:02,473][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:45:07,475][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:45:12,476][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:45:17,482][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:45:22,489][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:45:27,490][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:45:32,492][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:45:37,494][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:45:42,495][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:45:47,496][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:45:52,497][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:45:57,499][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:46:02,500][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:46:07,501][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:46:12,503][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:46:17,504][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:46:22,505][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:46:27,506][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:46:32,508][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:46:37,509][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:46:42,510][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:46:47,511][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:46:52,512][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:46:57,513][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:47:02,514][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:47:07,515][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:47:12,516][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:47:17,517][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:47:22,519][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:47:27,520][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:47:32,522][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:47:37,523][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:47:42,524][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:47:47,526][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:47:52,527][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:47:57,528][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:48:02,529][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:48:07,531][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:48:12,534][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:48:17,535][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:48:22,536][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:48:27,538][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:48:32,539][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:48:37,541][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:48:42,542][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:48:47,547][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:48:52,555][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:48:57,560][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:49:02,562][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:49:07,571][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:49:12,576][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:49:17,582][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:49:22,584][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:49:27,585][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:49:32,588][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:49:37,591][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:49:42,594][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:49:47,595][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:49:52,597][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:49:57,605][ INFO][PID:2268084] Periodic builder liveness probe: alive [2025-08-25 15:49:57,606][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:50:02,607][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:50:07,609][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:50:12,611][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:50:17,613][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:50:22,616][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:50:27,619][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:50:32,623][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:50:37,625][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:50:42,626][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:50:47,635][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:50:52,636][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:50:57,637][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:51:02,639][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:51:07,641][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:51:12,644][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:51:17,646][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:51:22,648][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:51:27,650][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:51:32,652][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:51:37,653][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:51:42,655][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:51:47,657][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:51:52,658][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:51:57,662][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:52:02,664][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:52:07,665][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:52:12,667][ INFO][PID:2268084] Checking for cancel request [2025-08-25 15:52:13,384][ INFO][PID:2268084] Downloading results from builder [2025-08-25 15:52:13,385][ INFO][PID:2268084] rsyncing of mockbuilder@3.85.49.107:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree started [2025-08-25 15:52:13,386][ INFO][PID:2268084] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@3.85.49.107:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/ &> /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/build-09478634.rsync.log [2025-08-25 15:52:14,371][ INFO][PID:2268084] rsyncing finished. [2025-08-25 15:52:14,372][ INFO][PID:2268084] Releasing VM back to pool [2025-08-25 15:52:14,388][ INFO][PID:2268084] Searching for 'success' file in resultdir [2025-08-25 15:52:14,390][ INFO][PID:2268084] Going to sign pkgs from source: 9478634-fedora-41-x86_64 in chroot: /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64 [2025-08-25 15:52:14,393][ INFO][PID:2268084] Calling '/bin/sign -u @CoreOS#continuous@copr.fedorahosted.org -p' (attempt #1) [2025-08-25 15:52:14,573][ INFO][PID:2268084] Calling '/bin/sign -4 -h sha256 -u @CoreOS#continuous@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-debugsource-2025.10.7.g283f0897-1.fc41.x86_64.rpm' (attempt #1) [2025-08-25 15:52:15,521][ INFO][PID:2268084] signed rpm: /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-debugsource-2025.10.7.g283f0897-1.fc41.x86_64.rpm [2025-08-25 15:52:15,522][ INFO][PID:2268084] Calling '/bin/sign -4 -h sha256 -u @CoreOS#continuous@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-libs-debuginfo-2025.10.7.g283f0897-1.fc41.x86_64.rpm' (attempt #1) [2025-08-25 15:52:16,441][ INFO][PID:2268084] signed rpm: /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-libs-debuginfo-2025.10.7.g283f0897-1.fc41.x86_64.rpm [2025-08-25 15:52:16,442][ INFO][PID:2268084] Calling '/bin/sign -4 -h sha256 -u @CoreOS#continuous@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-2025.10.7.g283f0897-1.fc41.x86_64.rpm' (attempt #1) [2025-08-25 15:52:17,482][ INFO][PID:2268084] signed rpm: /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-2025.10.7.g283f0897-1.fc41.x86_64.rpm [2025-08-25 15:52:17,483][ INFO][PID:2268084] Calling '/bin/sign -4 -h sha256 -u @CoreOS#continuous@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-2025.10.7.g283f0897-1.fc41.src.rpm' (attempt #1) [2025-08-25 15:52:18,677][ INFO][PID:2268084] signed rpm: /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-2025.10.7.g283f0897-1.fc41.src.rpm [2025-08-25 15:52:18,677][ INFO][PID:2268084] Calling '/bin/sign -4 -h sha256 -u @CoreOS#continuous@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-devel-2025.10.7.g283f0897-1.fc41.x86_64.rpm' (attempt #1) [2025-08-25 15:52:19,433][ INFO][PID:2268084] signed rpm: /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-devel-2025.10.7.g283f0897-1.fc41.x86_64.rpm [2025-08-25 15:52:19,434][ INFO][PID:2268084] Calling '/bin/sign -4 -h sha256 -u @CoreOS#continuous@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-libs-2025.10.7.g283f0897-1.fc41.x86_64.rpm' (attempt #1) [2025-08-25 15:52:19,880][ INFO][PID:2268084] signed rpm: /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-libs-2025.10.7.g283f0897-1.fc41.x86_64.rpm [2025-08-25 15:52:19,881][ INFO][PID:2268084] Calling '/bin/sign -4 -h sha256 -u @CoreOS#continuous@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-debuginfo-2025.10.7.g283f0897-1.fc41.x86_64.rpm' (attempt #1) [2025-08-25 15:52:21,517][ INFO][PID:2268084] signed rpm: /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/rpm-ostree-debuginfo-2025.10.7.g283f0897-1.fc41.x86_64.rpm [2025-08-25 15:52:21,519][ INFO][PID:2268084] Sign done [2025-08-25 15:52:21,520][ INFO][PID:2268084] Incremental createrepo run, adding 09478634-rpm-ostree into https://download.copr.fedorainfracloud.org/results/@CoreOS/continuous/fedora-41-x86_64, (auto-create-repo=True) [2025-08-25 15:52:21,524][ INFO][PID:2268084] Running command 'copr-repo --batched /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64 --add 09478634-rpm-ostree --no-appstream-metadata' as PID 2471619 [2025-08-25 15:52:23,409][ INFO][PID:2268084] Finished after 1 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64 --add 09478634-rpm-ostree --no-appstream-metadata) [2025-08-25 15:52:23,411][ INFO][PID:2268084] Getting build details [2025-08-25 15:52:23,412][ INFO][PID:2268084] Listing built binary packages in /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree [2025-08-25 15:52:23,413][ INFO][PID:2268084] Built packages: ['rpm-ostree-debugsource 2025.10.7.g283f0897', 'rpm-ostree-devel 2025.10.7.g283f0897', 'rpm-ostree-libs 2025.10.7.g283f0897', 'rpm-ostree 2025.10.7.g283f0897', 'rpm-ostree-debuginfo 2025.10.7.g283f0897', 'rpm-ostree-libs-debuginfo 2025.10.7.g283f0897'] [2025-08-25 15:52:23,421][ INFO][PID:2268084] build details: {'built_packages': 'rpm-ostree-debugsource 2025.10.7.g283f0897\nrpm-ostree-devel 2025.10.7.g283f0897\nrpm-ostree-libs 2025.10.7.g283f0897\nrpm-ostree 2025.10.7.g283f0897\nrpm-ostree-debuginfo 2025.10.7.g283f0897\nrpm-ostree-libs-debuginfo 2025.10.7.g283f0897'} [2025-08-25 15:52:23,423][ INFO][PID:2268084] Retrieving pubkey [2025-08-25 15:52:23,424][ INFO][PID:2268084] Calling '/bin/sign -u @CoreOS#continuous@copr.fedorahosted.org -p' (attempt #1) [2025-08-25 15:52:23,764][ INFO][PID:2268084] Added pubkey for user @CoreOS project continuous into: /var/lib/copr/public_html/results/@CoreOS/continuous/pubkey.gpg [2025-08-25 15:52:23,768][ INFO][PID:2268084] Finished build: id=9478634 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/@CoreOS/continuous chroot=fedora-41-x86_64 [2025-08-25 15:52:23,771][ INFO][PID:2268084] Worker succeeded build, took 751.7545161247253 [2025-08-25 15:52:23,772][ INFO][PID:2268084] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@CoreOS", "project_name": "continuous", "project_dirname": "continuous", "submitter": "https://api.github.com/users/cgwalters", "ended_on": 1756137143.7717018, "started_on": 1756136392.0171857, "submitted_on": null, "status": 1, "chroot": "fedora-41-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "9478634-fedora-41-x86_64", "build_id": 9478634, "package_name": "rpm-ostree", "package_version": "2025.10.7.g283f0897-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@CoreOS/continuous/rpm-ostree", "git_hash": "f56e62d5c47805dc3ede39ce02509ee7e40c8bc9", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@CoreOS/continuous--https://api.github.com/users/cgwalters", "results": { "packages": [ { "name": "rpm-ostree-debugsource", "epoch": null, "version": "2025.10.7.g283f0897", "release": "1.fc41", "arch": "x86_64" }, { "name": "rpm-ostree-devel", "epoch": null, "version": "2025.10.7.g283f0897", "release": "1.fc41", "arch": "x86_64" }, { "name": "rpm-ostree-libs", "epoch": null, "version": "2025.10.7.g283f0897", "release": "1.fc41", "arch": "x86_64" }, { "name": "rpm-ostree", "epoch": null, "version": "2025.10.7.g283f0897", "release": "1.fc41", "arch": "src" }, { "name": "rpm-ostree", "epoch": null, "version": "2025.10.7.g283f0897", "release": "1.fc41", "arch": "x86_64" }, { "name": "rpm-ostree-debuginfo", "epoch": null, "version": "2025.10.7.g283f0897", "release": "1.fc41", "arch": "x86_64" }, { "name": "rpm-ostree-libs-debuginfo", "epoch": null, "version": "2025.10.7.g283f0897", "release": "1.fc41", "arch": "x86_64" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@CoreOS/continuous/fedora-41-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/fedora-41-x86_64/", "id": "copr_rhcontainerbot_bootc", "name": "Additional repo copr_rhcontainerbot_bootc" } ], "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@CoreOS/continuous", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@CoreOS/continuous", "result_dir": "09478634-rpm-ostree", "built_packages": "rpm-ostree-debugsource 2025.10.7.g283f0897\nrpm-ostree-devel 2025.10.7.g283f0897\nrpm-ostree-libs 2025.10.7.g283f0897\nrpm-ostree 2025.10.7.g283f0897\nrpm-ostree-debuginfo 2025.10.7.g283f0897\nrpm-ostree-libs-debuginfo 2025.10.7.g283f0897", "id": 9478634, "mockchain_macros": { "copr_username": "@CoreOS", "copr_projectname": "continuous", "vendor": "Fedora Project COPR (@CoreOS/continuous)" } } ] } [2025-08-25 15:52:23,837][ INFO][PID:2268084] Sending fedora-messaging bus message in build.end [2025-08-25 15:52:23,875][ INFO][PID:2268084] Compressing /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/builder-live.log by gzip [2025-08-25 15:52:23,882][ INFO][PID:2268084] Running command 'gzip /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/builder-live.log' as PID 2471987 [2025-08-25 15:52:23,919][ INFO][PID:2268084] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/builder-live.log) [2025-08-25 15:52:23,920][ INFO][PID:2268084] Compressing /var/lib/copr/public_html/results/@CoreOS/continuous/fedora-41-x86_64/09478634-rpm-ostree/backend.log by gzip