[2026-01-06 15:04:03,859][ INFO][PID:177164] Marking build as starting [2026-01-06 15:04:03,905][ INFO][PID:177164] Checking that /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/repodata/repomd.xml exists [2026-01-06 15:04:04,031][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:04:04,032][ INFO][PID:177164] VM allocation process starts [2026-01-06 15:04:04,044][ INFO][PID:177164] Trying to allocate VM: ResallocHost, ticket_id=674942, requested_tags=['copr_builder', 'arch_i386'] [2026-01-06 15:04:07,074][ INFO][PID:177164] Allocated host ResallocHost, ticket_id=674942, hostname=3.91.147.43, name=aws_x86_64_spot_prod_00956256_20260106_150132, requested_tags=['copr_builder', 'arch_i386'] [2026-01-06 15:04:07,076][ INFO][PID:177164] Allocating ssh connection to builder [2026-01-06 15:04:07,078][ INFO][PID:177164] Checking that builder machine is OK [2026-01-06 15:04:07,470][ INFO][PID:177164] Running remote command: copr-builder-ready fedora-43-i386 [2026-01-06 15:04:07,710][ INFO][PID:177164] Red Hat subscription not needed for fedora-43-i386 Builder is ready to be used [2026-01-06 15:04:07,711][ INFO][PID:177164] Filling build.info file with builder info [2026-01-06 15:04:07,712][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:04:07,714][ INFO][PID:177164] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "xxmitsu", "project_name": "llvm", "project_dirname": "llvm", "submitter": null, "ended_on": null, "started_on": 1767711847.714697, "submitted_on": null, "status": 3, "chroot": "fedora-43-i386", "arch": "i386", "buildroot_pkgs": [ "git" ], "task_id": "9978390-fedora-43-i386", "build_id": 9978390, "package_name": "compiler-rt", "package_version": "19.0.0-0.1.20240309.git0d6f9bf", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xxmitsu/llvm/compiler-rt", "git_hash": "d2f7ae387f782237cd89f42f5a485a70bc6ce99d", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "xxmitsu/llvm--6e102d6d-e625-47dc-982b-85d509ec94ad", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm/fedora-43-i386/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_i386" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/xxmitsu/llvm", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm", "result_dir": "09978390-compiler-rt", "built_packages": "", "id": 9978390, "mockchain_macros": { "copr_username": "xxmitsu", "copr_projectname": "llvm", "vendor": "Fedora Project COPR (xxmitsu/llvm)" } } ] } [2026-01-06 15:04:07,800][ INFO][PID:177164] Sending fedora-messaging bus message in build.start [2026-01-06 15:04:08,591][ INFO][PID:177164] Sending fedora-messaging bus message in chroot.start [2026-01-06 15:04:08,621][ INFO][PID:177164] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9978390-fedora-43-i386 --chroot fedora-43-i386 --detached [2026-01-06 15:04:08,966][ INFO][PID:177164] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.91.147.43' (ED25519) to the list of known hosts. [2026-01-06 15:04:08,967][ INFO][PID:177164] Downloading the builder-live.log file, attempt 1 [2026-01-06 15:04:08,974][ INFO][PID:177164] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.91.147.43 copr-rpmbuild-log [2026-01-06 15:04:13,978][ INFO][PID:177164] Periodic builder liveness probe: alive [2026-01-06 15:04:13,979][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:04:18,980][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:04:19,597][ ERROR][PID:177164] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-06 15:04:23,981][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:04:28,982][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:04:29,600][WARNING][PID:177164] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-06 15:04:33,984][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:04:38,985][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:04:39,601][ INFO][PID:177164] Downloading the builder-live.log file, attempt 2 [2026-01-06 15:04:39,602][ INFO][PID:177164] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.91.147.43 copr-rpmbuild-log [2026-01-06 15:04:43,986][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:04:48,987][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:04:53,988][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:04:58,989][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:03,990][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:08,997][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:13,998][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:18,999][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:24,000][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:29,001][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:34,002][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:39,003][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:39,669][ ERROR][PID:177164] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-06 15:05:44,004][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:49,005][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:49,690][WARNING][PID:177164] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-06 15:05:54,006][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:59,007][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:05:59,691][ INFO][PID:177164] Downloading the builder-live.log file, attempt 3 [2026-01-06 15:05:59,692][ INFO][PID:177164] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.91.147.43 copr-rpmbuild-log [2026-01-06 15:06:04,009][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:06:09,010][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:06:14,012][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:06:19,015][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:06:24,017][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:06:29,018][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:06:34,019][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:06:39,020][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:06:44,022][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:06:49,024][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:06:54,025][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:06:59,029][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:06:59,764][ ERROR][PID:177164] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-06 15:07:04,034][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:07:09,039][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:07:09,782][WARNING][PID:177164] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-06 15:07:14,041][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:07:19,045][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:07:19,786][ INFO][PID:177164] Downloading the builder-live.log file, attempt 4 [2026-01-06 15:07:19,799][ INFO][PID:177164] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.91.147.43 copr-rpmbuild-log [2026-01-06 15:07:24,055][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:07:29,061][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:07:34,064][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:07:39,065][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:07:44,067][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:07:49,070][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:07:54,078][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:07:59,080][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:08:04,089][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:08:09,090][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:08:14,093][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:08:19,106][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:08:19,911][ ERROR][PID:177164] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-06 15:08:24,112][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:08:29,113][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:08:29,927][ ERROR][PID:177164] Giving up for unstable SSH, failures: 1767711869.6000001, 1767711949.6907377, 1767712029.7822154, 1767712109.926816 [2026-01-06 15:08:29,928][ ERROR][PID:177164] Re-try request for task on 'ResallocHost, ticket_id=674942, hostname=3.91.147.43, name=aws_x86_64_spot_prod_00956256_20260106_150132, requested_tags=['copr_builder', 'arch_i386']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-06 15:08:29,931][ INFO][PID:177164] Releasing VM back to pool [2026-01-06 15:08:30,004][ INFO][PID:177164] Retry #1 (on other host) [2026-01-06 15:08:30,020][ INFO][PID:177164] Checking that /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/repodata/repomd.xml exists [2026-01-06 15:08:30,024][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:08:30,028][ INFO][PID:177164] VM allocation process starts [2026-01-06 15:08:30,107][ INFO][PID:177164] Trying to allocate VM: ResallocHost, ticket_id=675055, requested_tags=['copr_builder', 'arch_i386'] [2026-01-06 15:08:35,115][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:08:40,116][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:08:45,118][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:08:50,122][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:08:55,124][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:00,134][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:05,135][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:08,409][ INFO][PID:177164] Allocated host ResallocHost, ticket_id=675055, hostname=44.200.35.70, name=aws_x86_64_normal_prod_00956614_20260106_150656, requested_tags=['copr_builder', 'arch_i386'] [2026-01-06 15:09:08,427][ INFO][PID:177164] Allocating ssh connection to builder [2026-01-06 15:09:08,427][ INFO][PID:177164] Checking that builder machine is OK [2026-01-06 15:09:08,886][ INFO][PID:177164] Running remote command: copr-builder-ready fedora-43-i386 [2026-01-06 15:09:09,227][ INFO][PID:177164] Red Hat subscription not needed for fedora-43-i386 Builder is ready to be used [2026-01-06 15:09:09,230][ INFO][PID:177164] Filling build.info file with builder info [2026-01-06 15:09:09,252][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:09,254][ INFO][PID:177164] Not re-notifying FE and msg buses for the new host. [2026-01-06 15:09:09,256][ INFO][PID:177164] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9978390-fedora-43-i386 --chroot fedora-43-i386 --detached [2026-01-06 15:09:09,776][ INFO][PID:177164] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '44.200.35.70' (ED25519) to the list of known hosts. [2026-01-06 15:09:09,783][ INFO][PID:177164] Downloading the builder-live.log file, attempt 1 [2026-01-06 15:09:09,794][ INFO][PID:177164] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.200.35.70 copr-rpmbuild-log [2026-01-06 15:09:14,803][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:19,824][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:24,829][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:29,831][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:34,834][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:39,840][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:44,848][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:49,850][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:54,852][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:09:59,854][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:10:04,864][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:10:09,872][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:10:14,878][ INFO][PID:177164] Checking for cancel request [2026-01-06 15:10:15,428][ INFO][PID:177164] Downloading results from builder [2026-01-06 15:10:15,437][ INFO][PID:177164] rsyncing of mockbuilder@44.200.35.70:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/09978390-compiler-rt started [2026-01-06 15:10:15,453][ INFO][PID:177164] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.200.35.70:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/09978390-compiler-rt/ &> /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/09978390-compiler-rt/build-09978390.rsync.log [2026-01-06 15:10:26,445][ INFO][PID:177164] rsyncing finished. [2026-01-06 15:10:26,450][ INFO][PID:177164] Releasing VM back to pool [2026-01-06 15:10:26,523][ INFO][PID:177164] Searching for 'success' file in resultdir [2026-01-06 15:10:26,527][ ERROR][PID:177164] Build failed: Backend process error: No success file => build failure [2026-01-06 15:10:26,531][ INFO][PID:177164] Finished build: id=9978390 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/xxmitsu/llvm chroot=fedora-43-i386 [2026-01-06 15:10:26,560][ ERROR][PID:177164] Backend process error: No success file => build failure [2026-01-06 15:10:26,562][ INFO][PID:177164] Worker failed build, took 77.30792379379272 [2026-01-06 15:10:26,563][ INFO][PID:177164] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": true, "project_owner": "xxmitsu", "project_name": "llvm", "project_dirname": "llvm", "submitter": null, "ended_on": 1767712226.5620766, "started_on": 1767712149.2541528, "submitted_on": null, "status": 0, "chroot": "fedora-43-i386", "arch": "i386", "buildroot_pkgs": [ "git" ], "task_id": "9978390-fedora-43-i386", "build_id": 9978390, "package_name": "compiler-rt", "package_version": "19.0.0-0.1.20240309.git0d6f9bf", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xxmitsu/llvm/compiler-rt", "git_hash": "d2f7ae387f782237cd89f42f5a485a70bc6ce99d", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "xxmitsu/llvm--6e102d6d-e625-47dc-982b-85d509ec94ad", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm/fedora-43-i386/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "tags": [ "arch_i386" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/xxmitsu/llvm", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm", "result_dir": "09978390-compiler-rt", "built_packages": "", "id": 9978390, "mockchain_macros": { "copr_username": "xxmitsu", "copr_projectname": "llvm", "vendor": "Fedora Project COPR (xxmitsu/llvm)" } } ] } [2026-01-06 15:10:26,751][ INFO][PID:177164] Sending fedora-messaging bus message in build.end [2026-01-06 15:10:26,885][ INFO][PID:177164] Compressing /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/09978390-compiler-rt/builder-live.log by gzip [2026-01-06 15:10:26,891][ INFO][PID:177164] Running command 'gzip /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/09978390-compiler-rt/builder-live.log' as PID 227521 [2026-01-06 15:10:26,934][ INFO][PID:177164] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/09978390-compiler-rt/builder-live.log) [2026-01-06 15:10:26,938][ INFO][PID:177164] Compressing /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/09978390-compiler-rt/backend.log by gzip [2026-01-06 15:10:26,946][ INFO][PID:177164] Running command 'gzip /var/lib/copr/public_html/results/xxmitsu/llvm/fedora-43-i386/09978390-compiler-rt/backend.log' as PID 227537