[2025-04-29 20:23:23,688][ INFO][PID:335994] Marking build as starting [2025-04-29 20:23:23,746][ INFO][PID:335994] Trying to allocate VM: ResallocHost, ticket_id=2727049, requested_tags=['copr_builder', 'arch_x86_64'] [2025-04-29 20:24:51,967][ INFO][PID:335994] Allocated host ResallocHost, ticket_id=2727049, hostname=34.236.155.105, name=aws_x86_64_spot_prod_03523178_20250429_201125, requested_tags=['copr_builder', 'arch_x86_64'] [2025-04-29 20:24:51,969][ INFO][PID:335994] Allocating ssh connection to builder [2025-04-29 20:24:51,973][ INFO][PID:335994] Checking that builder machine is OK [2025-04-29 20:25:52,079][ ERROR][PID:335994] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 34.236.155.105 port 22: Connection timed out [2025-04-29 20:26:02,083][ ERROR][PID:335994] Re-try request for task on 'ResallocHost, ticket_id=2727049, hostname=34.236.155.105, name=aws_x86_64_spot_prod_03523178_20250429_201125, requested_tags=['copr_builder', 'arch_x86_64']': Unable to finish after 1 SSH attempts [2025-04-29 20:26:02,087][ INFO][PID:335994] Releasing VM back to pool [2025-04-29 20:26:02,148][ INFO][PID:335994] Retry #1 (on other host) [2025-04-29 20:26:02,200][ INFO][PID:335994] Trying to allocate VM: ResallocHost, ticket_id=2727109, requested_tags=['copr_builder', 'arch_x86_64'] [2025-04-29 20:28:20,381][ INFO][PID:335994] Allocated host ResallocHost, ticket_id=2727109, hostname=54.210.161.249, name=aws_x86_64_spot_prod_03524259_20250429_202627, requested_tags=['copr_builder', 'arch_x86_64'] [2025-04-29 20:28:20,382][ INFO][PID:335994] Allocating ssh connection to builder [2025-04-29 20:28:20,384][ INFO][PID:335994] Checking that builder machine is OK [2025-04-29 20:28:20,737][ INFO][PID:335994] Running remote command: copr-builder-ready alma+epel-9-x86_64 [2025-04-29 20:28:20,954][ INFO][PID:335994] Red Hat subscription not needed for alma+epel-9-x86_64 Builder is ready to be used [2025-04-29 20:28:20,956][ INFO][PID:335994] Filling build.info file with builder info [2025-04-29 20:28:20,958][ INFO][PID:335994] Not re-notifying FE and msg buses for the new host. [2025-04-29 20:28:20,959][ INFO][PID:335994] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8980623-alma+epel-9-x86_64 --chroot alma+epel-9-x86_64 --detached [2025-04-29 20:28:21,325][ INFO][PID:335994] Downloading the builder-live.log file, attempt 1 [2025-04-29 20:28:21,329][ INFO][PID:335994] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.210.161.249 copr-rpmbuild-log [2025-04-29 20:28:26,336][ INFO][PID:335994] Periodic builder liveness probe: alive [2025-04-29 20:36:51,038][ ERROR][PID:335994] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-29 20:37:01,039][WARNING][PID:335994] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-29 20:37:11,041][ INFO][PID:335994] Downloading the builder-live.log file, attempt 2 [2025-04-29 20:37:11,049][ INFO][PID:335994] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.210.161.249 copr-rpmbuild-log [2025-04-29 20:38:11,116][ ERROR][PID:335994] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-29 20:38:21,119][WARNING][PID:335994] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-29 20:38:26,484][ INFO][PID:335994] Periodic builder liveness probe: alive [2025-04-29 20:38:31,120][ INFO][PID:335994] Downloading the builder-live.log file, attempt 3 [2025-04-29 20:38:31,122][ INFO][PID:335994] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.210.161.249 copr-rpmbuild-log [2025-04-29 20:39:31,182][ ERROR][PID:335994] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-29 20:39:41,187][WARNING][PID:335994] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-29 20:39:51,189][ INFO][PID:335994] Downloading the builder-live.log file, attempt 4 [2025-04-29 20:39:51,190][ INFO][PID:335994] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.210.161.249 copr-rpmbuild-log [2025-04-29 20:40:51,258][ ERROR][PID:335994] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-29 20:41:01,263][ ERROR][PID:335994] Giving up for unstable SSH, failures: 1745959021.0394676, 1745959101.119026, 1745959181.1877367, 1745959261.2632215 [2025-04-29 20:41:01,264][ ERROR][PID:335994] Re-try request for task on 'ResallocHost, ticket_id=2727109, hostname=54.210.161.249, name=aws_x86_64_spot_prod_03524259_20250429_202627, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-04-29 20:41:01,265][ INFO][PID:335994] Releasing VM back to pool [2025-04-29 20:41:01,298][ INFO][PID:335994] Retry #2 (on other host) [2025-04-29 20:41:01,312][ INFO][PID:335994] Trying to allocate VM: ResallocHost, ticket_id=2727149, requested_tags=['copr_builder', 'arch_x86_64'] [2025-04-29 20:41:04,330][ INFO][PID:335994] Allocated host ResallocHost, ticket_id=2727149, hostname=2620:52:3:1:dead:beef:cafe:c1db, name=copr_hv_x86_64_04_prod_03524517_20250429_203021, requested_tags=['copr_builder', 'arch_x86_64'] [2025-04-29 20:41:04,331][ INFO][PID:335994] Allocating ssh connection to builder [2025-04-29 20:41:04,332][ INFO][PID:335994] Checking that builder machine is OK [2025-04-29 20:41:04,951][ INFO][PID:335994] Running remote command: copr-builder-ready alma+epel-9-x86_64 [2025-04-29 20:41:05,339][ INFO][PID:335994] Red Hat subscription not needed for alma+epel-9-x86_64 Builder is ready to be used [2025-04-29 20:41:05,340][ INFO][PID:335994] Filling build.info file with builder info [2025-04-29 20:41:05,344][ INFO][PID:335994] Not re-notifying FE and msg buses for the new host. [2025-04-29 20:41:05,345][ INFO][PID:335994] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8980623-alma+epel-9-x86_64 --chroot alma+epel-9-x86_64 --detached [2025-04-29 20:41:06,124][ INFO][PID:335994] Downloading the builder-live.log file, attempt 1 [2025-04-29 20:41:06,125][ INFO][PID:335994] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c1db copr-rpmbuild-log [2025-04-29 20:48:31,216][ INFO][PID:335994] Periodic builder liveness probe: alive [2025-04-29 20:58:31,340][ INFO][PID:335994] Periodic builder liveness probe: alive [2025-04-29 21:02:57,335][ INFO][PID:335994] Downloading results from builder [2025-04-29 21:02:57,337][ INFO][PID:335994] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c1db]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband started [2025-04-29 21:02:57,338][ INFO][PID:335994] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c1db]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/ &> /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/build-08980623.rsync.log [2025-04-29 21:02:58,945][ INFO][PID:335994] rsyncing finished. [2025-04-29 21:02:58,947][ INFO][PID:335994] Releasing VM back to pool [2025-04-29 21:02:58,958][ INFO][PID:335994] Searching for 'success' file in resultdir [2025-04-29 21:02:58,959][ INFO][PID:335994] Going to sign pkgs from source: 8980623-alma+epel-9-x86_64 in chroot: /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64 [2025-04-29 21:02:58,960][ INFO][PID:335994] Calling '/bin/sign -u whitehara#hengband@copr.fedorahosted.org -p' (attempt #1) [2025-04-29 21:02:59,279][ INFO][PID:335994] Calling '/bin/sign -4 -h sha256 -u whitehara#hengband@copr.fedorahosted.org -r /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-en-debuginfo-3.0.1.26-1.x86_64.rpm' (attempt #1) [2025-04-29 21:03:00,657][ INFO][PID:335994] signed rpm: /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-en-debuginfo-3.0.1.26-1.x86_64.rpm [2025-04-29 21:03:00,661][ INFO][PID:335994] Calling '/bin/sign -4 -h sha256 -u whitehara#hengband@copr.fedorahosted.org -r /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-data-3.0.1.26-1.x86_64.rpm' (attempt #1) [2025-04-29 21:03:01,574][ INFO][PID:335994] signed rpm: /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-data-3.0.1.26-1.x86_64.rpm [2025-04-29 21:03:01,575][ INFO][PID:335994] Calling '/bin/sign -4 -h sha256 -u whitehara#hengband@copr.fedorahosted.org -r /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-3.0.1.26-1.x86_64.rpm' (attempt #1) [2025-04-29 21:03:02,501][ INFO][PID:335994] signed rpm: /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-3.0.1.26-1.x86_64.rpm [2025-04-29 21:03:02,503][ INFO][PID:335994] Calling '/bin/sign -4 -h sha256 -u whitehara#hengband@copr.fedorahosted.org -r /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-3.0.1.26-1.src.rpm' (attempt #1) [2025-04-29 21:03:03,466][ INFO][PID:335994] signed rpm: /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-3.0.1.26-1.src.rpm [2025-04-29 21:03:03,467][ INFO][PID:335994] Calling '/bin/sign -4 -h sha256 -u whitehara#hengband@copr.fedorahosted.org -r /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-debugsource-3.0.1.26-1.x86_64.rpm' (attempt #1) [2025-04-29 21:03:04,389][ INFO][PID:335994] signed rpm: /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-debugsource-3.0.1.26-1.x86_64.rpm [2025-04-29 21:03:04,391][ INFO][PID:335994] Calling '/bin/sign -4 -h sha256 -u whitehara#hengband@copr.fedorahosted.org -r /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-en-3.0.1.26-1.x86_64.rpm' (attempt #1) [2025-04-29 21:03:05,032][ INFO][PID:335994] signed rpm: /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-en-3.0.1.26-1.x86_64.rpm [2025-04-29 21:03:05,033][ INFO][PID:335994] Calling '/bin/sign -4 -h sha256 -u whitehara#hengband@copr.fedorahosted.org -r /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-debuginfo-3.0.1.26-1.x86_64.rpm' (attempt #1) [2025-04-29 21:03:06,288][ INFO][PID:335994] signed rpm: /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/hengband-debuginfo-3.0.1.26-1.x86_64.rpm [2025-04-29 21:03:06,290][ INFO][PID:335994] Sign done [2025-04-29 21:03:06,291][ INFO][PID:335994] Incremental createrepo run, adding 08980623-hengband into https://download.copr.fedorainfracloud.org/results/whitehara/hengband/alma+epel-9-x86_64, (auto-create-repo=True) [2025-04-29 21:03:06,292][ INFO][PID:335994] Running command 'copr-repo --batched /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64 --add 08980623-hengband --no-appstream-metadata' as PID 482357 [2025-04-29 21:03:07,012][ INFO][PID:335994] Finished after 0 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64 --add 08980623-hengband --no-appstream-metadata) [2025-04-29 21:03:07,014][ INFO][PID:335994] Getting build details [2025-04-29 21:03:07,015][ INFO][PID:335994] Listing built binary packages in /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband [2025-04-29 21:03:07,016][ INFO][PID:335994] Built packages: ['hengband-data 3.0.1.26', 'hengband-en-debuginfo 3.0.1.26', 'hengband-debuginfo 3.0.1.26', 'hengband 3.0.1.26', 'hengband-debugsource 3.0.1.26', 'hengband-en 3.0.1.26'] [2025-04-29 21:03:07,017][ INFO][PID:335994] build details: {'built_packages': 'hengband-data 3.0.1.26\nhengband-en-debuginfo 3.0.1.26\nhengband-debuginfo 3.0.1.26\nhengband 3.0.1.26\nhengband-debugsource 3.0.1.26\nhengband-en 3.0.1.26'} [2025-04-29 21:03:07,018][ INFO][PID:335994] Retrieving pubkey [2025-04-29 21:03:07,019][ INFO][PID:335994] Calling '/bin/sign -u whitehara#hengband@copr.fedorahosted.org -p' (attempt #1) [2025-04-29 21:03:07,339][ INFO][PID:335994] Added pubkey for user whitehara project hengband into: /var/lib/copr/public_html/results/whitehara/hengband/pubkey.gpg [2025-04-29 21:03:07,340][ INFO][PID:335994] Finished build: id=8980623 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/whitehara/hengband chroot=alma+epel-9-x86_64 [2025-04-29 21:03:07,346][ INFO][PID:335994] Worker succeeded build, took 1322.0024800300598 [2025-04-29 21:03:07,348][ INFO][PID:335994] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "whitehara", "project_name": "hengband", "project_dirname": "hengband", "submitter": "whitehara", "ended_on": 1745960587.3464494, "started_on": 1745959265.3439693, "submitted_on": null, "status": 1, "chroot": "alma+epel-9-x86_64", "arch": "x86_64", "buildroot_pkgs": [], "task_id": "8980623-alma+epel-9-x86_64", "build_id": 8980623, "package_name": "hengband", "package_version": "3.0.1.26-1", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/whitehara/hengband/hengband", "git_hash": "884cd36bfd728df79414604f97f419c87db773b3", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "whitehara/hengband--whitehara", "results": { "packages": [ { "name": "hengband-data", "epoch": null, "version": "3.0.1.26", "release": "1", "arch": "x86_64" }, { "name": "hengband-en-debuginfo", "epoch": null, "version": "3.0.1.26", "release": "1", "arch": "x86_64" }, { "name": "hengband-debuginfo", "epoch": null, "version": "3.0.1.26", "release": "1", "arch": "x86_64" }, { "name": "hengband", "epoch": null, "version": "3.0.1.26", "release": "1", "arch": "src" }, { "name": "hengband", "epoch": null, "version": "3.0.1.26", "release": "1", "arch": "x86_64" }, { "name": "hengband-debugsource", "epoch": null, "version": "3.0.1.26", "release": "1", "arch": "x86_64" }, { "name": "hengband-en", "epoch": null, "version": "3.0.1.26", "release": "1", "arch": "x86_64" } ] }, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": false, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/whitehara/hengband/alma+epel-9-x86_64/", "id": "copr_base", "name": "Copr repository", "priority": null }, { "baseurl": "https://download.copr.fedorainfracloud.org/results/whitehara/nkf/alma+epel-9-x86_64/", "id": "copr_whitehara_nkf", "name": "Additional repo copr_whitehara_nkf" } ], "tags": [ "arch_x86_64" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/whitehara/hengband", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/whitehara/hengband", "result_dir": "08980623-hengband", "built_packages": "hengband-data 3.0.1.26\nhengband-en-debuginfo 3.0.1.26\nhengband-debuginfo 3.0.1.26\nhengband 3.0.1.26\nhengband-debugsource 3.0.1.26\nhengband-en 3.0.1.26", "id": 8980623, "mockchain_macros": { "copr_username": "whitehara", "copr_projectname": "hengband", "vendor": "Fedora Project COPR (whitehara/hengband)" } } ] } [2025-04-29 21:03:07,394][ INFO][PID:335994] Sending fedora-messaging bus message in build.end [2025-04-29 21:03:07,969][ INFO][PID:335994] Compressing /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/builder-live.log by gzip [2025-04-29 21:03:07,971][ INFO][PID:335994] Running command 'gzip /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/builder-live.log' as PID 482471 [2025-04-29 21:03:07,999][ INFO][PID:335994] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/builder-live.log) [2025-04-29 21:03:08,000][ INFO][PID:335994] Compressing /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/backend.log by gzip [2025-04-29 21:03:08,002][ INFO][PID:335994] Running command 'gzip /var/lib/copr/public_html/results/whitehara/hengband/alma+epel-9-x86_64/08980623-hengband/backend.log' as PID 482472