[2026-03-30 02:09:04,200][ INFO][PID:633246] Marking build as starting [2026-03-30 02:09:04,264][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:09:04,269][ INFO][PID:633246] VM allocation process starts [2026-03-30 02:09:04,356][ INFO][PID:633246] Trying to allocate VM: ResallocHost, ticket_id=2084348, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-30 02:09:07,380][ INFO][PID:633246] Allocated host ResallocHost, ticket_id=2084348, hostname=35.174.11.44, name=aws_x86_64_spot_prod_04380414_20260330_020722, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-30 02:09:07,381][ INFO][PID:633246] Allocating ssh connection to builder [2026-03-30 02:09:07,381][ INFO][PID:633246] Checking that builder machine is OK [2026-03-30 02:09:07,715][ INFO][PID:633246] Running remote command: copr-builder-ready srpm-builds [2026-03-30 02:09:07,923][ INFO][PID:633246] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-03-30 02:09:07,924][ INFO][PID:633246] Filling build.info file with builder info [2026-03-30 02:09:07,925][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:09:07,926][ INFO][PID:633246] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "clarlok", "project_name": "network", "project_dirname": "network", "submitter": "clarlok", "ended_on": null, "started_on": 1774836547.9259126, "submitted_on": null, "status": 3, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10272134", "build_id": 10272134, "package_name": "sing-geosite", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#! /bin/sh -x\\npkgname=sing-geosite\\nDATE=$(curl -s https://api.github.com/repos/SagerNet/${pkgname}/releases/latest | grep -oP -m 1 '\\\"published_at\\\": \\\"\\\\K(.*)(?=T)')\\nDATE=$(date -d \\\"$DATE\\\" +'%a %b %d %Y')\\nVERSION=$(curl -s https://api.github.com/repos/SagerNet/${pkgname}/releases/latest | grep -oP '\\\"tag_name\\\": \\\"\\\\K(.*)(?=\\\")')\\n\\ngit clone https://github.com/SagerNet/${pkgname}\\nmv -v ${pkgname} ${pkgname}-${VERSION}\\ntar czvf ${pkgname}-${VERSION}.tar.gz ${pkgname}-${VERSION}\\ncurl -L -O https://raw.githubusercontent.com/calyle/rpm-spec-templates/refs/heads/main/network/${pkgname}/${pkgname}.spec\\nsed -i -e \\\"s/VERSION/$VERSION/g\\\" -e \\\"s/DATE/$DATE/g\\\" ${pkgname}.spec\\n\", \"chroot\": \"fedora-42-x86_64\", \"builddeps\": \"git\", \"resultdir\": \"\", \"repos\": \"\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "clarlok/network--clarlok", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "destdir": "/var/lib/copr/public_html/results/clarlok/network", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/clarlok/network", "result_dir": "10272134", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 10272134, "mockchain_macros": { "copr_username": "clarlok", "copr_projectname": "network", "vendor": "Fedora Project COPR (clarlok/network)" } } ] } [2026-03-30 02:09:07,971][ INFO][PID:633246] Sending fedora-messaging bus message in build.start [2026-03-30 02:09:08,951][ INFO][PID:633246] Sending fedora-messaging bus message in chroot.start [2026-03-30 02:09:08,986][ INFO][PID:633246] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10272134 --detached [2026-03-30 02:09:09,336][ INFO][PID:633246] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '35.174.11.44' (ED25519) to the list of known hosts. [2026-03-30 02:09:09,337][ INFO][PID:633246] Downloading the builder-live.log file, attempt 1 [2026-03-30 02:09:09,339][ INFO][PID:633246] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.174.11.44 copr-rpmbuild-log [2026-03-30 02:09:14,349][ INFO][PID:633246] Periodic builder liveness probe: alive [2026-03-30 02:09:14,350][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:09:19,351][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:09:24,356][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:09:29,357][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:09:31,773][ ERROR][PID:633246] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-30 02:09:34,360][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:09:39,363][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:09:41,775][WARNING][PID:633246] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-30 02:09:44,364][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:09:49,366][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:09:51,777][ INFO][PID:633246] Downloading the builder-live.log file, attempt 2 [2026-03-30 02:09:51,782][ INFO][PID:633246] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.174.11.44 copr-rpmbuild-log [2026-03-30 02:09:54,367][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:09:59,371][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:10:04,372][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:10:09,374][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:10:14,379][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:10:19,382][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:10:24,385][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:10:29,387][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:10:34,389][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:10:39,391][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:10:44,393][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:10:49,396][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:10:51,870][ ERROR][PID:633246] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-30 02:10:54,399][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:10:59,414][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:11:01,892][WARNING][PID:633246] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-30 02:11:04,415][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:11:09,422][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:11:11,894][ INFO][PID:633246] Downloading the builder-live.log file, attempt 3 [2026-03-30 02:11:11,895][ INFO][PID:633246] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.174.11.44 copr-rpmbuild-log [2026-03-30 02:11:14,423][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:11:19,424][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:11:24,426][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:11:29,427][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:11:34,428][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:11:39,429][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:11:44,430][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:11:49,431][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:11:54,432][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:11:59,434][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:12:04,435][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:12:09,436][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:12:11,963][ ERROR][PID:633246] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-30 02:12:14,437][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:12:19,438][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:12:21,964][WARNING][PID:633246] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-03-30 02:12:24,439][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:12:29,440][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:12:31,965][ INFO][PID:633246] Downloading the builder-live.log file, attempt 4 [2026-03-30 02:12:31,966][ INFO][PID:633246] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.174.11.44 copr-rpmbuild-log [2026-03-30 02:12:34,441][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:12:39,442][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:12:44,443][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:12:49,444][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:12:54,448][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:12:59,452][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:13:04,453][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:13:09,454][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:13:14,455][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:13:19,456][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:13:24,458][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:13:29,459][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:13:32,034][ ERROR][PID:633246] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-03-30 02:13:34,460][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:13:39,461][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:13:42,042][ ERROR][PID:633246] Giving up for unstable SSH, failures: 1774836581.775814, 1774836661.8927898, 1774836741.9643562, 1774836822.0426443 [2026-03-30 02:13:42,043][ ERROR][PID:633246] Re-try request for task on 'ResallocHost, ticket_id=2084348, hostname=35.174.11.44, name=aws_x86_64_spot_prod_04380414_20260330_020722, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-03-30 02:13:42,044][ INFO][PID:633246] Releasing VM back to pool [2026-03-30 02:13:42,052][ INFO][PID:633246] Retry #1 (on other host) [2026-03-30 02:13:42,052][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:13:42,053][ INFO][PID:633246] VM allocation process starts [2026-03-30 02:13:42,095][ INFO][PID:633246] Trying to allocate VM: ResallocHost, ticket_id=2084493, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-30 02:13:45,137][ INFO][PID:633246] Allocated host ResallocHost, ticket_id=2084493, hostname=107.21.161.107, name=aws_x86_64_spot_prod_04380484_20260330_021026, requested_tags=['copr_builder', 'arch_x86_64'] [2026-03-30 02:13:45,137][ INFO][PID:633246] Allocating ssh connection to builder [2026-03-30 02:13:45,138][ INFO][PID:633246] Checking that builder machine is OK [2026-03-30 02:13:45,457][ INFO][PID:633246] Running remote command: copr-builder-ready srpm-builds [2026-03-30 02:13:45,661][ INFO][PID:633246] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-03-30 02:13:45,662][ INFO][PID:633246] Filling build.info file with builder info [2026-03-30 02:13:45,664][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:13:45,665][ INFO][PID:633246] Not re-notifying FE and msg buses for the new host. [2026-03-30 02:13:45,666][ INFO][PID:633246] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10272134 --detached [2026-03-30 02:13:45,990][ INFO][PID:633246] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '107.21.161.107' (ED25519) to the list of known hosts. [2026-03-30 02:13:45,991][ INFO][PID:633246] Downloading the builder-live.log file, attempt 1 [2026-03-30 02:13:45,992][ INFO][PID:633246] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@107.21.161.107 copr-rpmbuild-log [2026-03-30 02:13:50,991][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:13:55,992][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:14:00,994][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:14:05,995][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:14:10,996][ INFO][PID:633246] Checking for cancel request [2026-03-30 02:14:14,767][ INFO][PID:633246] Downloading results from builder [2026-03-30 02:14:14,767][ INFO][PID:633246] rsyncing of mockbuilder@107.21.161.107:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/clarlok/network/srpm-builds/10272134 started [2026-03-30 02:14:14,768][ INFO][PID:633246] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@107.21.161.107:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/clarlok/network/srpm-builds/10272134/ &> /var/lib/copr/public_html/results/clarlok/network/srpm-builds/10272134/build-10272134.rsync.log [2026-03-30 02:14:15,032][ INFO][PID:633246] rsyncing finished. [2026-03-30 02:14:15,033][ INFO][PID:633246] Releasing VM back to pool [2026-03-30 02:14:15,041][ INFO][PID:633246] Searching for 'success' file in resultdir [2026-03-30 02:14:15,042][ INFO][PID:633246] Getting build details [2026-03-30 02:14:15,043][ INFO][PID:633246] Retrieving SRPM info from /var/lib/copr/public_html/results/clarlok/network/srpm-builds/10272134 [2026-03-30 02:14:15,043][ INFO][PID:633246] SRPM URL: https://download.copr.fedorainfracloud.org/results/clarlok/network/srpm-builds/10272134/sing-geosite-20260326050832-1.src.rpm [2026-03-30 02:14:15,044][ INFO][PID:633246] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/clarlok/network/srpm-builds/10272134/sing-geosite-20260326050832-1.src.rpm', 'pkg_name': 'sing-geosite', 'pkg_version': '20260326050832-1'} [2026-03-30 02:14:15,044][ INFO][PID:633246] Finished build: id=10272134 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/clarlok/network chroot=srpm-builds [2026-03-30 02:14:15,046][ INFO][PID:633246] Worker succeeded build, took 29.38012933731079 [2026-03-30 02:14:15,046][ INFO][PID:633246] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "clarlok", "project_name": "network", "project_dirname": "network", "submitter": "clarlok", "ended_on": 1774836855.0459762, "started_on": 1774836825.6658468, "submitted_on": null, "status": 1, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10272134", "build_id": 10272134, "package_name": "sing-geosite", "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"#! /bin/sh -x\\npkgname=sing-geosite\\nDATE=$(curl -s https://api.github.com/repos/SagerNet/${pkgname}/releases/latest | grep -oP -m 1 '\\\"published_at\\\": \\\"\\\\K(.*)(?=T)')\\nDATE=$(date -d \\\"$DATE\\\" +'%a %b %d %Y')\\nVERSION=$(curl -s https://api.github.com/repos/SagerNet/${pkgname}/releases/latest | grep -oP '\\\"tag_name\\\": \\\"\\\\K(.*)(?=\\\")')\\n\\ngit clone https://github.com/SagerNet/${pkgname}\\nmv -v ${pkgname} ${pkgname}-${VERSION}\\ntar czvf ${pkgname}-${VERSION}.tar.gz ${pkgname}-${VERSION}\\ncurl -L -O https://raw.githubusercontent.com/calyle/rpm-spec-templates/refs/heads/main/network/${pkgname}/${pkgname}.spec\\nsed -i -e \\\"s/VERSION/$VERSION/g\\\" -e \\\"s/DATE/$DATE/g\\\" ${pkgname}.spec\\n\", \"chroot\": \"fedora-42-x86_64\", \"builddeps\": \"git\", \"resultdir\": \"\", \"repos\": \"\"}", "pkg_name": "sing-geosite", "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": "https://download.copr.fedorainfracloud.org/results/clarlok/network/srpm-builds/10272134/sing-geosite-20260326050832-1.src.rpm", "uses_devel_repo": null, "sandbox": "clarlok/network--clarlok", "results": { "name": "sing-geosite", "epoch": null, "version": "20260326050832", "release": "1", "exclusivearch": [], "excludearch": [] }, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [], "background": false, "destdir": "/var/lib/copr/public_html/results/clarlok/network", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/clarlok/network", "result_dir": "10272134", "built_packages": "", "tags": [ "arch_x86_64" ], "pkg_version": "20260326050832-1", "id": 10272134, "mockchain_macros": { "copr_username": "clarlok", "copr_projectname": "network", "vendor": "Fedora Project COPR (clarlok/network)" } } ] } [2026-03-30 02:14:15,117][ INFO][PID:633246] Sending fedora-messaging bus message in build.end [2026-03-30 02:14:15,145][ INFO][PID:633246] Compressing /var/lib/copr/public_html/results/clarlok/network/srpm-builds/10272134/builder-live.log by gzip [2026-03-30 02:14:15,146][ INFO][PID:633246] Running command 'gzip /var/lib/copr/public_html/results/clarlok/network/srpm-builds/10272134/builder-live.log' as PID 671830 [2026-03-30 02:14:15,165][ INFO][PID:633246] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/clarlok/network/srpm-builds/10272134/builder-live.log) [2026-03-30 02:14:15,166][ INFO][PID:633246] Compressing /var/lib/copr/public_html/results/clarlok/network/srpm-builds/10272134/backend.log by gzip [2026-03-30 02:14:15,167][ INFO][PID:633246] Running command 'gzip /var/lib/copr/public_html/results/clarlok/network/srpm-builds/10272134/backend.log' as PID 671834