[2026-02-12 22:19:11,756][ INFO][PID:3775240] Marking build as starting [2026-02-12 22:19:11,806][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:19:11,807][ INFO][PID:3775240] VM allocation process starts [2026-02-12 22:19:11,822][ INFO][PID:3775240] Trying to allocate VM: ResallocHost, ticket_id=1309194, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-12 22:19:14,853][ INFO][PID:3775240] Allocated host ResallocHost, ticket_id=1309194, hostname=54.160.15.44, name=aws_x86_64_spot_prod_02241187_20260212_220920, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-12 22:19:14,854][ INFO][PID:3775240] Allocating ssh connection to builder [2026-02-12 22:19:14,856][ INFO][PID:3775240] Checking that builder machine is OK [2026-02-12 22:19:15,214][ INFO][PID:3775240] Running remote command: copr-builder-ready srpm-builds [2026-02-12 22:19:15,429][ INFO][PID:3775240] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-02-12 22:19:15,430][ INFO][PID:3775240] Filling build.info file with builder info [2026-02-12 22:19:15,430][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:19:15,431][ INFO][PID:3775240] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "networkmanager", "project_name": "NetworkManager-1.48-debug", "project_dirname": "NetworkManager-1.48-debug", "submitter": null, "ended_on": null, "started_on": 1770934755.43122, "submitted_on": null, "status": 3, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10126166", "build_id": 10126166, "package_name": "NetworkManager", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#!/bin/bash\\n\\nexport GIT_REF=nm-1-48\\nexport DEBUG=1\\nexport LTO=\\ncurl https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/raw/main/contrib/scripts/nm-copr-build.sh | bash\\n\", \"chroot\": \"fedora-44-x86_64\", \"builddeps\": \"git-core curl-minimal pam\", \"resultdir\": \"\", \"repos\": \"\", \"tmp\": \"tmpnrxxq3vw\", \"hook_data\": true}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "networkmanager/NetworkManager-1.48-debug--9978ee0b-e1bd-41e7-b975-c922ea3f85d3", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": true, "destdir": "/var/lib/copr/public_html/results/networkmanager/NetworkManager-1.48-debug", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/networkmanager/NetworkManager-1.48-debug", "result_dir": "10126166", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 10126166, "mockchain_macros": { "copr_username": "networkmanager", "copr_projectname": "NetworkManager-1.48-debug", "vendor": "Fedora Project COPR (networkmanager/NetworkManager-1.48-debug)" } } ] } [2026-02-12 22:19:15,489][ INFO][PID:3775240] Sending fedora-messaging bus message in build.start [2026-02-12 22:19:16,199][ INFO][PID:3775240] Sending fedora-messaging bus message in chroot.start [2026-02-12 22:19:16,227][ INFO][PID:3775240] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10126166 --detached [2026-02-12 22:19:16,591][ INFO][PID:3775240] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.160.15.44' (ED25519) to the list of known hosts. [2026-02-12 22:19:16,592][ INFO][PID:3775240] Downloading the builder-live.log file, attempt 1 [2026-02-12 22:19:16,594][ INFO][PID:3775240] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.160.15.44 copr-rpmbuild-log [2026-02-12 22:19:21,597][ INFO][PID:3775240] Periodic builder liveness probe: alive [2026-02-12 22:19:21,598][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:19:26,599][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:19:31,601][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:19:31,656][ ERROR][PID:3775240] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-12 22:19:36,602][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:19:41,603][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:19:41,656][WARNING][PID:3775240] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-12 22:19:46,604][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:19:51,605][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:19:51,657][ INFO][PID:3775240] Downloading the builder-live.log file, attempt 2 [2026-02-12 22:19:51,659][ INFO][PID:3775240] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.160.15.44 copr-rpmbuild-log [2026-02-12 22:19:56,606][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:20:01,607][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:20:06,613][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:20:11,614][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:20:16,616][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:20:21,617][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:20:26,618][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:20:31,619][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:20:36,619][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:20:41,620][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:20:46,621][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:20:51,622][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:20:51,726][ ERROR][PID:3775240] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-12 22:20:56,623][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:21:01,624][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:21:01,735][WARNING][PID:3775240] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-12 22:21:06,625][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:21:11,626][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:21:11,735][ INFO][PID:3775240] Downloading the builder-live.log file, attempt 3 [2026-02-12 22:21:11,737][ INFO][PID:3775240] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.160.15.44 copr-rpmbuild-log [2026-02-12 22:21:16,627][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:21:21,634][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:21:26,640][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:21:31,665][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:21:36,667][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:21:41,668][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:21:46,669][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:21:51,670][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:21:56,671][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:22:01,672][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:22:06,673][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:22:11,674][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:22:11,804][ ERROR][PID:3775240] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-12 22:22:16,676][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:22:21,681][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:22:21,825][WARNING][PID:3775240] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-02-12 22:22:26,683][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:22:31,687][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:22:31,826][ INFO][PID:3775240] Downloading the builder-live.log file, attempt 4 [2026-02-12 22:22:31,829][ INFO][PID:3775240] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.160.15.44 copr-rpmbuild-log [2026-02-12 22:22:36,691][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:22:41,694][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:22:46,696][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:22:51,697][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:22:56,698][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:01,699][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:06,700][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:11,701][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:16,702][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:21,703][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:26,704][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:31,706][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:31,906][ ERROR][PID:3775240] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-02-12 22:23:36,711][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:41,712][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:41,959][ ERROR][PID:3775240] Giving up for unstable SSH, failures: 1770934781.656909, 1770934861.7353354, 1770934941.8253334, 1770935021.9593928 [2026-02-12 22:23:41,960][ ERROR][PID:3775240] Re-try request for task on 'ResallocHost, ticket_id=1309194, hostname=54.160.15.44, name=aws_x86_64_spot_prod_02241187_20260212_220920, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-02-12 22:23:41,961][ INFO][PID:3775240] Releasing VM back to pool [2026-02-12 22:23:41,976][ INFO][PID:3775240] Retry #1 (on other host) [2026-02-12 22:23:41,977][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:41,978][ INFO][PID:3775240] VM allocation process starts [2026-02-12 22:23:41,992][ INFO][PID:3775240] Trying to allocate VM: ResallocHost, ticket_id=1309333, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-12 22:23:45,009][ INFO][PID:3775240] Allocated host ResallocHost, ticket_id=1309333, hostname=54.91.180.212, name=aws_x86_64_spot_prod_02241332_20260212_222127, requested_tags=['copr_builder', 'arch_x86_64'] [2026-02-12 22:23:45,010][ INFO][PID:3775240] Allocating ssh connection to builder [2026-02-12 22:23:45,011][ INFO][PID:3775240] Checking that builder machine is OK [2026-02-12 22:23:45,339][ INFO][PID:3775240] Running remote command: copr-builder-ready srpm-builds [2026-02-12 22:23:45,558][ INFO][PID:3775240] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-02-12 22:23:45,558][ INFO][PID:3775240] Filling build.info file with builder info [2026-02-12 22:23:45,567][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:45,568][ INFO][PID:3775240] Not re-notifying FE and msg buses for the new host. [2026-02-12 22:23:45,569][ INFO][PID:3775240] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10126166 --detached [2026-02-12 22:23:45,908][ INFO][PID:3775240] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '54.91.180.212' (ED25519) to the list of known hosts. [2026-02-12 22:23:45,909][ INFO][PID:3775240] Downloading the builder-live.log file, attempt 1 [2026-02-12 22:23:45,911][ INFO][PID:3775240] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.91.180.212 copr-rpmbuild-log [2026-02-12 22:23:50,910][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:23:55,912][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:00,913][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:05,915][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:10,916][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:15,917][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:20,918][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:25,919][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:30,921][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:35,922][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:40,923][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:45,935][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:50,937][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:55,938][ INFO][PID:3775240] Checking for cancel request [2026-02-12 22:24:59,328][ INFO][PID:3775240] Downloading results from builder [2026-02-12 22:24:59,328][ INFO][PID:3775240] rsyncing of mockbuilder@54.91.180.212:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/networkmanager/NetworkManager-1.48-debug/srpm-builds/10126166 started [2026-02-12 22:24:59,329][ INFO][PID:3775240] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@54.91.180.212:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/networkmanager/NetworkManager-1.48-debug/srpm-builds/10126166/ &> /var/lib/copr/public_html/results/networkmanager/NetworkManager-1.48-debug/srpm-builds/10126166/build-10126166.rsync.log [2026-02-12 22:24:59,637][ INFO][PID:3775240] rsyncing finished. [2026-02-12 22:24:59,637][ INFO][PID:3775240] Releasing VM back to pool [2026-02-12 22:24:59,647][ INFO][PID:3775240] Searching for 'success' file in resultdir [2026-02-12 22:24:59,648][ INFO][PID:3775240] Getting build details [2026-02-12 22:24:59,649][ INFO][PID:3775240] Retrieving SRPM info from /var/lib/copr/public_html/results/networkmanager/NetworkManager-1.48-debug/srpm-builds/10126166 [2026-02-12 22:24:59,650][ INFO][PID:3775240] SRPM URL: https://download.copr.fedorainfracloud.org/results/networkmanager/NetworkManager-1.48-debug/srpm-builds/10126166/NetworkManager-1.48.17-33403.copr.247f9cfda8.src.rpm [2026-02-12 22:24:59,650][ INFO][PID:3775240] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/networkmanager/NetworkManager-1.48-debug/srpm-builds/10126166/NetworkManager-1.48.17-33403.copr.247f9cfda8.src.rpm', 'pkg_name': 'NetworkManager', 'pkg_version': '1:1.48.17-33403.copr.247f9cfda8'} [2026-02-12 22:24:59,650][ INFO][PID:3775240] Finished build: id=10126166 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/networkmanager/NetworkManager-1.48-debug chroot=srpm-builds [2026-02-12 22:24:59,652][ INFO][PID:3775240] Worker succeeded build, took 74.08370757102966 [2026-02-12 22:24:59,653][ INFO][PID:3775240] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "networkmanager", "project_name": "NetworkManager-1.48-debug", "project_dirname": "NetworkManager-1.48-debug", "submitter": null, "ended_on": 1770935099.6524508, "started_on": 1770935025.5687432, "submitted_on": null, "status": 1, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10126166", "build_id": 10126166, "package_name": "NetworkManager", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#!/bin/bash\\n\\nexport GIT_REF=nm-1-48\\nexport DEBUG=1\\nexport LTO=\\ncurl https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/raw/main/contrib/scripts/nm-copr-build.sh | bash\\n\", \"chroot\": \"fedora-44-x86_64\", \"builddeps\": \"git-core curl-minimal pam\", \"resultdir\": \"\", \"repos\": \"\", \"tmp\": \"tmpnrxxq3vw\", \"hook_data\": true}", "pkg_name": "NetworkManager", "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": "https://download.copr.fedorainfracloud.org/results/networkmanager/NetworkManager-1.48-debug/srpm-builds/10126166/NetworkManager-1.48.17-33403.copr.247f9cfda8.src.rpm", "uses_devel_repo": null, "sandbox": "networkmanager/NetworkManager-1.48-debug--9978ee0b-e1bd-41e7-b975-c922ea3f85d3", "results": { "name": "NetworkManager", "epoch": 1, "version": "1.48.17", "release": "33403.copr.247f9cfda8", "exclusivearch": [], "excludearch": [] }, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": true, "destdir": "/var/lib/copr/public_html/results/networkmanager/NetworkManager-1.48-debug", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/networkmanager/NetworkManager-1.48-debug", "result_dir": "10126166", "built_packages": "", "tags": [ "arch_x86_64" ], "pkg_version": "1:1.48.17-33403.copr.247f9cfda8", "id": 10126166, "mockchain_macros": { "copr_username": "networkmanager", "copr_projectname": "NetworkManager-1.48-debug", "vendor": "Fedora Project COPR (networkmanager/NetworkManager-1.48-debug)" } } ] } [2026-02-12 22:24:59,712][ INFO][PID:3775240] Sending fedora-messaging bus message in build.end [2026-02-12 22:24:59,743][ INFO][PID:3775240] Compressing /var/lib/copr/public_html/results/networkmanager/NetworkManager-1.48-debug/srpm-builds/10126166/builder-live.log by gzip [2026-02-12 22:24:59,744][ INFO][PID:3775240] Running command 'gzip /var/lib/copr/public_html/results/networkmanager/NetworkManager-1.48-debug/srpm-builds/10126166/builder-live.log' as PID 3822287 [2026-02-12 22:24:59,788][ INFO][PID:3775240] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/networkmanager/NetworkManager-1.48-debug/srpm-builds/10126166/builder-live.log) [2026-02-12 22:24:59,789][ INFO][PID:3775240] Compressing /var/lib/copr/public_html/results/networkmanager/NetworkManager-1.48-debug/srpm-builds/10126166/backend.log by gzip [2026-02-12 22:24:59,790][ INFO][PID:3775240] Running command 'gzip /var/lib/copr/public_html/results/networkmanager/NetworkManager-1.48-debug/srpm-builds/10126166/backend.log' as PID 3822299