[2025-02-18 15:59:00,338][ INFO][PID:3200497] Marking build as starting [2025-02-18 15:59:00,539][ INFO][PID:3200497] Trying to allocate VM [2025-02-18 16:00:28,666][ INFO][PID:3200497] Allocated host ResallocHost, ticket_id=1361726, hostname=3.234.143.13, name=aws_x86_64_spot_prod_01295942_20250218_155633 [2025-02-18 16:00:28,667][ INFO][PID:3200497] Allocating ssh connection to builder [2025-02-18 16:00:28,667][ INFO][PID:3200497] Checking that builder machine is OK [2025-02-18 16:00:29,019][ INFO][PID:3200497] Running remote command: copr-builder-ready fedora-40-x86_64 [2025-02-18 16:00:29,238][ INFO][PID:3200497] Red Hat subscription not needed for fedora-40-x86_64 Builder is ready to be used [2025-02-18 16:00:29,239][ INFO][PID:3200497] Filling build.info file with builder info [2025-02-18 16:00:29,242][ INFO][PID:3200497] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@testing-farm", "project_name": "stable", "project_dirname": "stable", "submitter": "packit", "ended_on": null, "started_on": 1739894429.2420053, "submitted_on": null, "status": 3, "chroot": "fedora-40-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8668911-fedora-40-x86_64", "build_id": 8668911, "package_name": "testing-farm", "package_version": "0.0.23-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@testing-farm/stable/testing-farm", "git_hash": "bc216ad214140c702d3bbc764ae09795e63e598d", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@testing-farm/stable--packit", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@testing-farm/stable/fedora-40-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@testing-farm/stable", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@testing-farm/stable", "result_dir": "08668911-testing-farm", "built_packages": "", "id": 8668911, "mockchain_macros": { "copr_username": "@testing-farm", "copr_projectname": "stable", "vendor": "Fedora Project COPR (@testing-farm/stable)" } } ] } [2025-02-18 16:00:29,276][ INFO][PID:3200497] Sending fedora-messaging bus message in build.start [2025-02-18 16:00:30,011][ INFO][PID:3200497] Sending fedora-messaging bus message in chroot.start [2025-02-18 16:00:30,040][ INFO][PID:3200497] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8668911-fedora-40-x86_64 --chroot fedora-40-x86_64 --detached [2025-02-18 16:00:30,402][ INFO][PID:3200497] Downloading the builder-live.log file, attempt 1 [2025-02-18 16:00:30,405][ INFO][PID:3200497] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.234.143.13 copr-rpmbuild-log [2025-02-18 16:00:35,412][ INFO][PID:3200497] Periodic builder liveness probe: alive [2025-02-18 16:00:58,183][ ERROR][PID:3200497] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-02-18 16:01:08,185][WARNING][PID:3200497] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-02-18 16:01:18,186][ INFO][PID:3200497] Downloading the builder-live.log file, attempt 2 [2025-02-18 16:01:18,187][ INFO][PID:3200497] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.234.143.13 copr-rpmbuild-log [2025-02-18 16:02:18,255][ ERROR][PID:3200497] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-02-18 16:02:28,257][WARNING][PID:3200497] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-02-18 16:02:38,258][ INFO][PID:3200497] Downloading the builder-live.log file, attempt 3 [2025-02-18 16:02:38,259][ INFO][PID:3200497] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.234.143.13 copr-rpmbuild-log [2025-02-18 16:03:38,327][ ERROR][PID:3200497] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-02-18 16:03:48,328][WARNING][PID:3200497] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-02-18 16:03:58,329][ INFO][PID:3200497] Downloading the builder-live.log file, attempt 4 [2025-02-18 16:03:58,330][ INFO][PID:3200497] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.234.143.13 copr-rpmbuild-log [2025-02-18 16:04:58,399][ ERROR][PID:3200497] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-02-18 16:05:08,401][ ERROR][PID:3200497] Giving up for unstable SSH, failures: 1739894468.185305, 1739894548.2571816, 1739894628.3284187, 1739894708.4014354 [2025-02-18 16:05:08,402][ ERROR][PID:3200497] Re-try request for task on 'ResallocHost, ticket_id=1361726, hostname=3.234.143.13, name=aws_x86_64_spot_prod_01295942_20250218_155633': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-02-18 16:05:08,403][ INFO][PID:3200497] Releasing VM back to pool [2025-02-18 16:05:08,413][ INFO][PID:3200497] Retry #1 (on other host) [2025-02-18 16:05:08,414][ INFO][PID:3200497] Trying to allocate VM [2025-02-18 16:06:36,542][ INFO][PID:3200497] Allocated host ResallocHost, ticket_id=1361806, hostname=3.208.12.168, name=aws_x86_64_spot_prod_01296153_20250218_160459 [2025-02-18 16:06:36,543][ INFO][PID:3200497] Allocating ssh connection to builder [2025-02-18 16:06:36,543][ INFO][PID:3200497] Checking that builder machine is OK [2025-02-18 16:06:36,885][ INFO][PID:3200497] Running remote command: copr-builder-ready fedora-40-x86_64 [2025-02-18 16:06:37,098][ INFO][PID:3200497] Red Hat subscription not needed for fedora-40-x86_64 Builder is ready to be used [2025-02-18 16:06:37,098][ INFO][PID:3200497] Filling build.info file with builder info [2025-02-18 16:06:37,099][ INFO][PID:3200497] Not re-notifying FE and msg buses for the new host. [2025-02-18 16:06:37,100][ INFO][PID:3200497] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8668911-fedora-40-x86_64 --chroot fedora-40-x86_64 --detached [2025-02-18 16:06:37,467][ INFO][PID:3200497] Downloading the builder-live.log file, attempt 1 [2025-02-18 16:06:37,469][ INFO][PID:3200497] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.208.12.168 copr-rpmbuild-log [2025-02-18 16:07:23,234][ INFO][PID:3200497] Downloading results from builder [2025-02-18 16:07:23,235][ INFO][PID:3200497] rsyncing of mockbuilder@3.208.12.168:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm started [2025-02-18 16:07:23,241][ INFO][PID:3200497] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@3.208.12.168:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm/ &> /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm/build-08668911.rsync.log [2025-02-18 16:07:23,527][ INFO][PID:3200497] rsyncing finished. [2025-02-18 16:07:23,529][ INFO][PID:3200497] Releasing VM back to pool [2025-02-18 16:07:23,553][ INFO][PID:3200497] Searching for 'success' file in resultdir [2025-02-18 16:07:23,554][ INFO][PID:3200497] Going to sign pkgs from source: 8668911-fedora-40-x86_64 in chroot: /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64 [2025-02-18 16:07:23,555][ INFO][PID:3200497] Calling '/bin/sign -u @testing-farm#stable@copr.fedorahosted.org -p' (attempt #1) [2025-02-18 16:07:23,863][ INFO][PID:3200497] Calling '/bin/sign -4 -h sha256 -u @testing-farm#stable@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm/testing-farm-0.0.23-1.fc40.noarch.rpm' (attempt #1) [2025-02-18 16:07:24,554][ INFO][PID:3200497] signed rpm: /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm/testing-farm-0.0.23-1.fc40.noarch.rpm [2025-02-18 16:07:24,557][ INFO][PID:3200497] Calling '/bin/sign -4 -h sha256 -u @testing-farm#stable@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm/testing-farm-0.0.23-1.fc40.src.rpm' (attempt #1) [2025-02-18 16:07:25,249][ INFO][PID:3200497] signed rpm: /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm/testing-farm-0.0.23-1.fc40.src.rpm [2025-02-18 16:07:25,250][ INFO][PID:3200497] Sign done [2025-02-18 16:07:25,257][ INFO][PID:3200497] Incremental createrepo run, adding 08668911-testing-farm into https://download.copr.fedorainfracloud.org/results/@testing-farm/stable/fedora-40-x86_64, (auto-create-repo=True) [2025-02-18 16:07:25,260][ INFO][PID:3200497] Running command 'copr-repo --batched /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64 --add 08668911-testing-farm --no-appstream-metadata' as PID 3297817 [2025-02-18 16:07:26,567][ INFO][PID:3200497] Finished after 1 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64 --add 08668911-testing-farm --no-appstream-metadata) [2025-02-18 16:07:26,572][ INFO][PID:3200497] Getting build details [2025-02-18 16:07:26,573][ INFO][PID:3200497] Listing built binary packages in /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm [2025-02-18 16:07:26,576][ INFO][PID:3200497] Built packages: ['testing-farm 0.0.23'] [2025-02-18 16:07:26,578][ INFO][PID:3200497] build details: {'built_packages': 'testing-farm 0.0.23'} [2025-02-18 16:07:26,579][ INFO][PID:3200497] Retrieving pubkey [2025-02-18 16:07:26,580][ INFO][PID:3200497] Calling '/bin/sign -u @testing-farm#stable@copr.fedorahosted.org -p' (attempt #1) [2025-02-18 16:07:26,890][ INFO][PID:3200497] Added pubkey for user @testing-farm project stable into: /var/lib/copr/public_html/results/@testing-farm/stable/pubkey.gpg [2025-02-18 16:07:26,894][ INFO][PID:3200497] Finished build: id=8668911 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/@testing-farm/stable chroot=fedora-40-x86_64 [2025-02-18 16:07:26,898][ INFO][PID:3200497] Worker succeeded build, took 49.79837155342102 [2025-02-18 16:07:26,901][ INFO][PID:3200497] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "@testing-farm", "project_name": "stable", "project_dirname": "stable", "submitter": "packit", "ended_on": 1739894846.89833, "started_on": 1739894797.0999584, "submitted_on": null, "status": 1, "chroot": "fedora-40-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8668911-fedora-40-x86_64", "build_id": 8668911, "package_name": "testing-farm", "package_version": "0.0.23-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@testing-farm/stable/testing-farm", "git_hash": "bc216ad214140c702d3bbc764ae09795e63e598d", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "@testing-farm/stable--packit", "results": { "packages": [ { "name": "testing-farm", "epoch": null, "version": "0.0.23", "release": "1.fc40", "arch": "src" }, { "name": "testing-farm", "epoch": null, "version": "0.0.23", "release": "1.fc40", "arch": "noarch" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": 0, "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/@testing-farm/stable/fedora-40-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/@testing-farm/stable", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@testing-farm/stable", "result_dir": "08668911-testing-farm", "built_packages": "testing-farm 0.0.23", "id": 8668911, "mockchain_macros": { "copr_username": "@testing-farm", "copr_projectname": "stable", "vendor": "Fedora Project COPR (@testing-farm/stable)" } } ] } [2025-02-18 16:07:26,945][ INFO][PID:3200497] Sending fedora-messaging bus message in build.end [2025-02-18 16:07:26,988][ INFO][PID:3200497] Compressing /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm/builder-live.log by gzip [2025-02-18 16:07:26,993][ INFO][PID:3200497] Running command 'gzip /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm/builder-live.log' as PID 3298131 [2025-02-18 16:07:27,001][ INFO][PID:3200497] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm/builder-live.log) [2025-02-18 16:07:27,003][ INFO][PID:3200497] Compressing /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm/backend.log by gzip [2025-02-18 16:07:27,012][ INFO][PID:3200497] Running command 'gzip /var/lib/copr/public_html/results/@testing-farm/stable/fedora-40-x86_64/08668911-testing-farm/backend.log' as PID 3298134