[2026-05-14 15:15:51,928][ INFO][PID:3779291] Marking build as starting [2026-05-14 15:15:52,251][ INFO][PID:3779291] Checking that /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/repodata/repomd.xml exists [2026-05-14 15:15:52,287][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:15:52,299][ INFO][PID:3779291] VM allocation process starts [2026-05-14 15:15:52,483][ INFO][PID:3779291] Trying to allocate VM: ResallocHost, ticket_id=2890210, requested_tags=['arch_i386', 'copr_builder'] [2026-05-14 15:15:57,512][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:16:02,523][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:16:07,526][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:16:12,530][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:16:17,532][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:16:22,534][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:16:27,542][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:16:32,546][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:16:37,561][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:16:42,565][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:16:47,571][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:16:52,572][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:16:57,577][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:17:02,584][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:17:07,601][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:17:12,619][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:17:17,622][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:17:20,770][ INFO][PID:3779291] Allocated host ResallocHost, ticket_id=2890210, hostname=98.92.74.50, name=aws_x86_64_spot_prod_08839282_20260514_151449, requested_tags=['arch_i386', 'copr_builder'] [2026-05-14 15:17:20,772][ INFO][PID:3779291] Allocating ssh connection to builder [2026-05-14 15:17:20,774][ INFO][PID:3779291] Checking that builder machine is OK [2026-05-14 15:17:21,182][ INFO][PID:3779291] Installed copr-rpmbuild version: 1.8 [2026-05-14 15:17:21,186][ INFO][PID:3779291] Running remote command: copr-builder-ready fedora-43-i386 [2026-05-14 15:17:21,448][ INFO][PID:3779291] Red Hat subscription not needed for fedora-43-i386 Builder is ready to be used [2026-05-14 15:17:21,449][ INFO][PID:3779291] Filling build.info file with builder info [2026-05-14 15:17:21,450][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:17:21,458][ INFO][PID:3779291] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "xxmitsu", "project_name": "llvm", "project_dirname": "llvm", "submitter": null, "ended_on": null, "started_on": 1778771841.4580364, "submitted_on": null, "status": 3, "chroot": "fedora-43-i386", "arch": "i386", "buildroot_pkgs": [ "git" ], "task_id": "10460291-fedora-43-i386", "build_id": 10460291, "package_name": "libomp", "package_version": "19.0.0-0.1.20240309.git0d6f9bf", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xxmitsu/llvm/libomp", "git_hash": "e155a1177a62c7ec887c1e6756a58c17d95b1cde", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "xxmitsu/llvm--b82c6b6f-e185-4542-ade1-644434c4cbf2", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm/fedora-43-i386/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_i386" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/xxmitsu/llvm", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm", "result_dir": "10460291-libomp", "built_packages": "", "id": 10460291, "mockchain_macros": { "copr_username": "xxmitsu", "copr_projectname": "llvm", "vendor": "Fedora Project COPR (xxmitsu/llvm)" } } ] } [2026-05-14 15:17:21,590][ INFO][PID:3779291] Sending fedora-messaging bus message in build.start [2026-05-14 15:17:22,629][ INFO][PID:3779291] Sending fedora-messaging bus message in chroot.start [2026-05-14 15:17:22,685][ INFO][PID:3779291] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10460291-fedora-43-i386 --chroot fedora-43-i386 --detached [2026-05-14 15:17:23,310][ INFO][PID:3779291] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '98.92.74.50' (ED25519) to the list of known hosts. [2026-05-14 15:17:23,316][ INFO][PID:3779291] Downloading the builder-live.log file, attempt 1 [2026-05-14 15:17:23,329][ INFO][PID:3779291] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.74.50 copr-rpmbuild-log [2026-05-14 15:17:28,386][ INFO][PID:3779291] Periodic builder liveness probe: alive [2026-05-14 15:17:28,388][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:17:33,392][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:17:38,397][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:17:43,441][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:17:48,460][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:17:53,478][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:17:58,485][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:18:03,498][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:18:08,511][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:18:13,542][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:18:18,547][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:18:23,562][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:18:28,586][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:18:33,600][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:18:38,619][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:18:43,628][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:18:48,671][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:18:53,734][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:18:58,753][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:19:03,765][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:19:08,793][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:19:13,800][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:19:17,772][ ERROR][PID:3779291] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-05-14 15:19:18,816][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:19:23,872][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:19:27,863][WARNING][PID:3779291] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-05-14 15:19:28,889][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:19:33,929][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:19:37,873][ INFO][PID:3779291] Downloading the builder-live.log file, attempt 2 [2026-05-14 15:19:37,904][ INFO][PID:3779291] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.74.50 copr-rpmbuild-log [2026-05-14 15:19:38,950][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:19:43,992][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:19:49,009][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:19:54,040][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:19:59,045][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:20:04,061][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:20:09,126][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:20:14,147][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:20:19,154][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:20:24,170][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:20:29,179][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:20:34,244][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:20:38,064][ ERROR][PID:3779291] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-05-14 15:20:39,247][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:20:44,256][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:20:48,187][WARNING][PID:3779291] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-05-14 15:20:49,263][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:20:54,274][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:20:58,191][ INFO][PID:3779291] Downloading the builder-live.log file, attempt 3 [2026-05-14 15:20:58,198][ INFO][PID:3779291] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.74.50 copr-rpmbuild-log [2026-05-14 15:20:59,281][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:21:04,300][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:21:09,321][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:21:14,326][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:21:19,334][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:21:24,343][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:21:29,345][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:21:34,347][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:21:39,352][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:21:44,364][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:21:49,378][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:21:54,440][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:21:58,283][ ERROR][PID:3779291] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-05-14 15:21:59,455][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:22:04,466][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:22:08,346][WARNING][PID:3779291] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-05-14 15:22:09,474][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:22:14,518][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:22:18,351][ INFO][PID:3779291] Downloading the builder-live.log file, attempt 4 [2026-05-14 15:22:18,359][ INFO][PID:3779291] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@98.92.74.50 copr-rpmbuild-log [2026-05-14 15:22:19,555][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:22:24,572][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:22:29,584][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:22:34,616][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:22:39,631][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:22:44,664][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:22:49,682][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:22:54,720][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:22:59,737][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:23:04,764][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:23:09,800][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:23:14,819][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:23:18,450][ ERROR][PID:3779291] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-05-14 15:23:19,827][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:23:24,842][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:23:28,497][ ERROR][PID:3779291] Giving up for unstable SSH, failures: 1778771967.8639002, 1778772048.1874588, 1778772128.3466985, 1778772208.4974961 [2026-05-14 15:23:28,517][ ERROR][PID:3779291] Re-try request for task on 'ResallocHost, ticket_id=2890210, hostname=98.92.74.50, name=aws_x86_64_spot_prod_08839282_20260514_151449, requested_tags=['arch_i386', 'copr_builder']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-05-14 15:23:28,568][ INFO][PID:3779291] VM Release request [2026-05-14 15:23:28,610][ INFO][PID:3779291] Retry #1 (on other host) [2026-05-14 15:23:28,614][ INFO][PID:3779291] Checking that /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/repodata/repomd.xml exists [2026-05-14 15:23:28,617][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:23:28,628][ INFO][PID:3779291] VM allocation process starts [2026-05-14 15:23:28,663][ INFO][PID:3779291] Trying to allocate VM: ResallocHost, ticket_id=2890335, requested_tags=['arch_i386', 'copr_builder'] [2026-05-14 15:23:33,667][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:23:38,669][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:23:40,734][ INFO][PID:3779291] Allocated host ResallocHost, ticket_id=2890335, hostname=13.221.46.232, name=aws_x86_64_normal_prod_08839362_20260514_151702, requested_tags=['arch_i386', 'copr_builder'] [2026-05-14 15:23:40,736][ INFO][PID:3779291] Allocating ssh connection to builder [2026-05-14 15:23:40,737][ INFO][PID:3779291] Checking that builder machine is OK [2026-05-14 15:23:41,128][ INFO][PID:3779291] Installed copr-rpmbuild version: 1.8 [2026-05-14 15:23:41,129][ INFO][PID:3779291] Running remote command: copr-builder-ready fedora-43-i386 [2026-05-14 15:23:41,367][ INFO][PID:3779291] Red Hat subscription not needed for fedora-43-i386 Builder is ready to be used [2026-05-14 15:23:41,368][ INFO][PID:3779291] Filling build.info file with builder info [2026-05-14 15:23:41,380][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:23:41,384][ INFO][PID:3779291] Not re-notifying FE and msg buses for the new host. [2026-05-14 15:23:41,386][ INFO][PID:3779291] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10460291-fedora-43-i386 --chroot fedora-43-i386 --detached [2026-05-14 15:23:41,855][ INFO][PID:3779291] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '13.221.46.232' (ED25519) to the list of known hosts. [2026-05-14 15:23:41,856][ INFO][PID:3779291] Downloading the builder-live.log file, attempt 1 [2026-05-14 15:23:41,858][ INFO][PID:3779291] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.221.46.232 copr-rpmbuild-log [2026-05-14 15:23:46,858][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:23:51,860][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:23:56,862][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:24:01,866][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:24:06,871][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:24:11,873][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:24:16,878][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:24:21,883][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:24:26,886][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:24:31,890][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:24:36,894][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:24:41,896][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:24:46,898][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:24:51,901][ INFO][PID:3779291] Checking for cancel request [2026-05-14 15:24:54,865][ INFO][PID:3779291] Downloading results from builder [2026-05-14 15:24:54,868][ INFO][PID:3779291] rsyncing of mockbuilder@13.221.46.232:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/10460291-libomp started [2026-05-14 15:24:54,870][ INFO][PID:3779291] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@13.221.46.232:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/10460291-libomp/ &> /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/10460291-libomp/build-10460291.rsync.log [2026-05-14 15:24:57,287][ INFO][PID:3779291] rsyncing finished. [2026-05-14 15:24:57,288][ INFO][PID:3779291] VM Release request [2026-05-14 15:24:57,307][ INFO][PID:3779291] Searching for 'success' file in resultdir [2026-05-14 15:24:57,308][ ERROR][PID:3779291] Build failed: Backend process error: No success file => build failure [2026-05-14 15:24:57,329][ INFO][PID:3779291] Finished build: id=10460291 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/xxmitsu/llvm chroot=fedora-43-i386 [2026-05-14 15:24:57,337][ ERROR][PID:3779291] Backend process error: No success file => build failure [2026-05-14 15:24:57,354][ INFO][PID:3779291] Worker failed build, took 75.97054195404053 [2026-05-14 15:24:57,357][ INFO][PID:3779291] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "xxmitsu", "project_name": "llvm", "project_dirname": "llvm", "submitter": null, "ended_on": 1778772297.354631, "started_on": 1778772221.384089, "submitted_on": null, "status": 0, "chroot": "fedora-43-i386", "arch": "i386", "buildroot_pkgs": [ "git" ], "task_id": "10460291-fedora-43-i386", "build_id": 10460291, "package_name": "libomp", "package_version": "19.0.0-0.1.20240309.git0d6f9bf", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xxmitsu/llvm/libomp", "git_hash": "e155a1177a62c7ec887c1e6756a58c17d95b1cde", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "xxmitsu/llvm--b82c6b6f-e185-4542-ade1-644434c4cbf2", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm/fedora-43-i386/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_i386" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/xxmitsu/llvm", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm", "result_dir": "10460291-libomp", "built_packages": "", "id": 10460291, "mockchain_macros": { "copr_username": "xxmitsu", "copr_projectname": "llvm", "vendor": "Fedora Project COPR (xxmitsu/llvm)" } } ] } [2026-05-14 15:24:57,404][ INFO][PID:3779291] Sending fedora-messaging bus message in build.end [2026-05-14 15:24:57,435][ INFO][PID:3779291] Compressing /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/10460291-libomp/builder-live.log by gzip [2026-05-14 15:24:57,437][ INFO][PID:3779291] Running command 'gzip /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/10460291-libomp/builder-live.log' as PID 3881973 [2026-05-14 15:24:57,446][ INFO][PID:3779291] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/10460291-libomp/builder-live.log) [2026-05-14 15:24:57,447][ INFO][PID:3779291] Compressing /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/10460291-libomp/backend.log by gzip [2026-05-14 15:24:57,448][ INFO][PID:3779291] Running command 'gzip /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/10460291-libomp/backend.log' as PID 3881975