[2026-01-16 08:07:28,261][ INFO][PID:3155064] Marking build as starting [2026-01-16 08:07:28,343][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:07:28,345][ INFO][PID:3155064] VM allocation process starts [2026-01-16 08:07:28,367][ INFO][PID:3155064] Trying to allocate VM: ResallocHost, ticket_id=848397, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-16 08:07:31,416][ INFO][PID:3155064] Allocated host ResallocHost, ticket_id=848397, hostname=44.204.94.169, name=aws_x86_64_normal_prod_01288832_20260116_073742, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-16 08:07:31,418][ INFO][PID:3155064] Allocating ssh connection to builder [2026-01-16 08:07:31,420][ INFO][PID:3155064] Checking that builder machine is OK [2026-01-16 08:07:31,819][ INFO][PID:3155064] Running remote command: copr-builder-ready srpm-builds [2026-01-16 08:07:32,057][ INFO][PID:3155064] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-01-16 08:07:32,057][ INFO][PID:3155064] Filling build.info file with builder info [2026-01-16 08:07:32,058][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:07:32,062][ INFO][PID:3155064] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "@teemtee", "project_name": "latest", "project_dirname": "latest", "submitter": "packit", "ended_on": null, "started_on": 1768550852.0621264, "submitted_on": null, "status": 3, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10018581", "build_id": 10018581, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"\\n#!/bin/sh\\n\\ngit config --global user.email \\\"hello@packit.dev\\\"\\ngit config --global user.name \\\"Packit\\\"\\nresultdir=$PWD\\npackit -d prepare-sources --result-dir \\\"$resultdir\\\" --ref main --job-config-index 1 --release-suffix '{PACKIT_PROJECT_BRANCH}' --merged-ref '28be037500e2338cca4c9bdd4e9b13327df36df9' --no-create-symlinks https://github.com/teemtee/tmt\\n\\n\", \"chroot\": \"fedora-43-x86_64\", \"builddeps\": \"packit hatch python3-hatch-vcs\", \"resultdir\": null, \"repos\": \"copr://packit/packit-stable\"}", "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": null, "sandbox": "@teemtee/latest--packit", "results": null, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/packit-stable/fedora-43-x86_64/", "id": "copr_packit_packit_stable", "name": "Additional repo copr_packit_packit_stable" } ], "background": false, "destdir": "/var/lib/copr/public_html/results/@teemtee/latest", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@teemtee/latest", "result_dir": "10018581", "built_packages": "", "tags": [ "arch_x86_64" ], "id": 10018581, "mockchain_macros": { "copr_username": "@teemtee", "copr_projectname": "latest", "vendor": "Fedora Project COPR (@teemtee/latest)" } } ] } [2026-01-16 08:07:32,150][ INFO][PID:3155064] Sending fedora-messaging bus message in build.start [2026-01-16 08:07:33,020][ INFO][PID:3155064] Sending fedora-messaging bus message in chroot.start [2026-01-16 08:07:33,063][ INFO][PID:3155064] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10018581 --detached [2026-01-16 08:07:33,414][ INFO][PID:3155064] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '44.204.94.169' (ED25519) to the list of known hosts. [2026-01-16 08:07:33,415][ INFO][PID:3155064] Downloading the builder-live.log file, attempt 1 [2026-01-16 08:07:33,417][ INFO][PID:3155064] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.204.94.169 copr-rpmbuild-log [2026-01-16 08:07:38,424][ INFO][PID:3155064] Periodic builder liveness probe: alive [2026-01-16 08:07:38,425][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:07:43,426][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:07:48,428][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:07:53,432][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:07:56,277][ ERROR][PID:3155064] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-16 08:07:58,441][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:08:03,443][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:08:06,283][WARNING][PID:3155064] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-16 08:08:08,456][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:08:13,458][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:08:16,286][ INFO][PID:3155064] Downloading the builder-live.log file, attempt 2 [2026-01-16 08:08:16,290][ INFO][PID:3155064] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.204.94.169 copr-rpmbuild-log [2026-01-16 08:08:18,460][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:08:23,463][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:08:28,465][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:08:33,467][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:08:38,472][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:08:43,475][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:08:48,482][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:08:53,496][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:08:58,500][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:09:03,508][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:09:08,510][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:09:13,514][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:09:16,386][ ERROR][PID:3155064] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-16 08:09:18,524][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:09:23,527][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:09:26,402][WARNING][PID:3155064] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-16 08:09:28,532][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:09:33,535][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:09:36,403][ INFO][PID:3155064] Downloading the builder-live.log file, attempt 3 [2026-01-16 08:09:36,422][ INFO][PID:3155064] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.204.94.169 copr-rpmbuild-log [2026-01-16 08:09:38,544][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:09:43,550][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:09:48,561][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:09:53,563][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:09:58,572][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:10:03,581][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:10:08,585][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:10:13,589][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:10:18,591][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:10:23,594][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:10:28,598][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:10:33,599][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:10:36,529][ ERROR][PID:3155064] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-16 08:10:38,607][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:10:43,627][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:10:46,542][WARNING][PID:3155064] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2026-01-16 08:10:48,629][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:10:53,630][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:10:56,551][ INFO][PID:3155064] Downloading the builder-live.log file, attempt 4 [2026-01-16 08:10:56,557][ INFO][PID:3155064] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.204.94.169 copr-rpmbuild-log [2026-01-16 08:10:58,645][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:11:03,650][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:11:08,659][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:11:13,662][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:11:18,665][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:11:23,669][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:11:28,674][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:11:33,676][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:11:38,688][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:11:43,690][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:11:48,758][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:11:53,759][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:11:56,703][ ERROR][PID:3155064] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2026-01-16 08:11:58,763][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:12:03,817][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:12:06,769][ ERROR][PID:3155064] Giving up for unstable SSH, failures: 1768550886.2830524, 1768550966.4027016, 1768551046.5429127, 1768551126.7691822 [2026-01-16 08:12:06,778][ ERROR][PID:3155064] Re-try request for task on 'ResallocHost, ticket_id=848397, hostname=44.204.94.169, name=aws_x86_64_normal_prod_01288832_20260116_073742, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2026-01-16 08:12:06,779][ INFO][PID:3155064] Releasing VM back to pool [2026-01-16 08:12:06,833][ INFO][PID:3155064] Retry #1 (on other host) [2026-01-16 08:12:06,840][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:12:06,842][ INFO][PID:3155064] VM allocation process starts [2026-01-16 08:12:06,869][ INFO][PID:3155064] Trying to allocate VM: ResallocHost, ticket_id=848486, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-16 08:12:11,881][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:12:12,936][ INFO][PID:3155064] Allocated host ResallocHost, ticket_id=848486, hostname=54.175.1.121, name=aws_x86_64_normal_prod_01288872_20260116_073954, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-16 08:12:12,938][ INFO][PID:3155064] Allocating ssh connection to builder [2026-01-16 08:12:12,939][ INFO][PID:3155064] Checking that builder machine is OK [2026-01-16 08:13:13,036][ ERROR][PID:3155064] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 54.175.1.121 port 22: Connection timed out [2026-01-16 08:13:23,061][ ERROR][PID:3155064] Re-try request for task on 'ResallocHost, ticket_id=848486, hostname=54.175.1.121, name=aws_x86_64_normal_prod_01288872_20260116_073954, requested_tags=['copr_builder', 'arch_x86_64']': Unable to finish after 1 SSH attempts [2026-01-16 08:13:23,063][ INFO][PID:3155064] Releasing VM back to pool [2026-01-16 08:13:23,081][ INFO][PID:3155064] Retry #2 (on other host) [2026-01-16 08:13:23,082][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:13:23,084][ INFO][PID:3155064] VM allocation process starts [2026-01-16 08:13:23,106][ INFO][PID:3155064] Trying to allocate VM: ResallocHost, ticket_id=848549, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-16 08:13:28,108][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:13:33,110][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:13:38,113][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:13:41,154][ INFO][PID:3155064] Allocated host ResallocHost, ticket_id=848549, hostname=3.235.186.223, name=aws_x86_64_normal_prod_01289384_20260116_081217, requested_tags=['copr_builder', 'arch_x86_64'] [2026-01-16 08:13:41,155][ INFO][PID:3155064] Allocating ssh connection to builder [2026-01-16 08:13:41,158][ INFO][PID:3155064] Checking that builder machine is OK [2026-01-16 08:13:41,505][ INFO][PID:3155064] Running remote command: copr-builder-ready srpm-builds [2026-01-16 08:13:41,717][ INFO][PID:3155064] Red Hat subscription not needed for srpm-builds Builder is ready to be used [2026-01-16 08:13:41,720][ INFO][PID:3155064] Filling build.info file with builder info [2026-01-16 08:13:41,832][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:13:41,835][ INFO][PID:3155064] Not re-notifying FE and msg buses for the new host. [2026-01-16 08:13:41,836][ INFO][PID:3155064] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/10018581 --detached [2026-01-16 08:13:42,180][ INFO][PID:3155064] The copr-rpmbuild seems started, per: stdout: stderr: Warning: Permanently added '3.235.186.223' (ED25519) to the list of known hosts. [2026-01-16 08:13:42,183][ INFO][PID:3155064] Downloading the builder-live.log file, attempt 1 [2026-01-16 08:13:42,188][ INFO][PID:3155064] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@3.235.186.223 copr-rpmbuild-log [2026-01-16 08:13:47,183][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:13:52,189][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:13:57,191][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:14:02,197][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:14:07,203][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:14:12,207][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:14:17,226][ INFO][PID:3155064] Checking for cancel request [2026-01-16 08:14:21,670][ INFO][PID:3155064] Downloading results from builder [2026-01-16 08:14:21,675][ INFO][PID:3155064] rsyncing of mockbuilder@3.235.186.223:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@teemtee/latest/srpm-builds/10018581 started [2026-01-16 08:14:21,677][ INFO][PID:3155064] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@3.235.186.223:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@teemtee/latest/srpm-builds/10018581/ &> /var/lib/copr/public_html/results/@teemtee/latest/srpm-builds/10018581/build-10018581.rsync.log [2026-01-16 08:14:22,215][ INFO][PID:3155064] rsyncing finished. [2026-01-16 08:14:22,216][ INFO][PID:3155064] Releasing VM back to pool [2026-01-16 08:14:22,240][ INFO][PID:3155064] Searching for 'success' file in resultdir [2026-01-16 08:14:22,241][ INFO][PID:3155064] Getting build details [2026-01-16 08:14:22,244][ INFO][PID:3155064] Retrieving SRPM info from /var/lib/copr/public_html/results/@teemtee/latest/srpm-builds/10018581 [2026-01-16 08:14:22,245][ INFO][PID:3155064] SRPM URL: https://download.copr.fedorainfracloud.org/results/@teemtee/latest/srpm-builds/10018581/tmt-1.66.0.dev2+g28be03750-main.src.rpm [2026-01-16 08:14:22,246][ INFO][PID:3155064] build details: {'srpm_url': 'https://download.copr.fedorainfracloud.org/results/@teemtee/latest/srpm-builds/10018581/tmt-1.66.0.dev2+g28be03750-main.src.rpm', 'pkg_name': 'tmt', 'pkg_version': '1.66.0.dev2+g28be03750-main'} [2026-01-16 08:14:22,246][ INFO][PID:3155064] Finished build: id=10018581 failed=False timeout=108000 destdir=/var/lib/copr/public_html/results/@teemtee/latest chroot=srpm-builds [2026-01-16 08:14:22,251][ INFO][PID:3155064] Worker succeeded build, took 40.41563606262207 [2026-01-16 08:14:22,257][ INFO][PID:3155064] Sending build state back to frontend: { "builds": [ { "timeout": 108000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": null, "enable_net": true, "project_owner": "@teemtee", "project_name": "latest", "project_dirname": "latest", "submitter": "packit", "ended_on": 1768551262.2514992, "started_on": 1768551221.835863, "submitted_on": null, "status": 1, "chroot": "srpm-builds", "arch": "x86_64", "buildroot_pkgs": null, "task_id": "10018581", "build_id": 10018581, "package_name": null, "package_version": null, "git_repo": null, "git_hash": null, "git_branch": null, "source_type": 9, "source_json": "{\"script\": \"\\n#!/bin/sh\\n\\ngit config --global user.email \\\"hello@packit.dev\\\"\\ngit config --global user.name \\\"Packit\\\"\\nresultdir=$PWD\\npackit -d prepare-sources --result-dir \\\"$resultdir\\\" --ref main --job-config-index 1 --release-suffix '{PACKIT_PROJECT_BRANCH}' --merged-ref '28be037500e2338cca4c9bdd4e9b13327df36df9' --no-create-symlinks https://github.com/teemtee/tmt\\n\\n\", \"chroot\": \"fedora-43-x86_64\", \"builddeps\": \"packit hatch python3-hatch-vcs\", \"resultdir\": null, \"repos\": \"copr://packit/packit-stable\"}", "pkg_name": "tmt", "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": "https://download.copr.fedorainfracloud.org/results/@teemtee/latest/srpm-builds/10018581/tmt-1.66.0.dev2+g28be03750-main.src.rpm", "uses_devel_repo": null, "sandbox": "@teemtee/latest--packit", "results": { "name": "tmt", "epoch": null, "version": "1.66.0.dev2+g28be03750", "release": "main", "exclusivearch": [], "excludearch": [] }, "appstream": false, "allow_user_ssh": null, "ssh_public_keys": null, "storage": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/packit/packit-stable/fedora-43-x86_64/", "id": "copr_packit_packit_stable", "name": "Additional repo copr_packit_packit_stable" } ], "background": false, "destdir": "/var/lib/copr/public_html/results/@teemtee/latest", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@teemtee/latest", "result_dir": "10018581", "built_packages": "", "tags": [ "arch_x86_64" ], "pkg_version": "1.66.0.dev2+g28be03750-main", "id": 10018581, "mockchain_macros": { "copr_username": "@teemtee", "copr_projectname": "latest", "vendor": "Fedora Project COPR (@teemtee/latest)" } } ] } [2026-01-16 08:14:22,496][ INFO][PID:3155064] Sending fedora-messaging bus message in build.end [2026-01-16 08:14:22,606][ INFO][PID:3155064] Compressing /var/lib/copr/public_html/results/@teemtee/latest/srpm-builds/10018581/builder-live.log by gzip [2026-01-16 08:14:22,624][ INFO][PID:3155064] Running command 'gzip /var/lib/copr/public_html/results/@teemtee/latest/srpm-builds/10018581/builder-live.log' as PID 3227544 [2026-01-16 08:14:22,716][ INFO][PID:3155064] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@teemtee/latest/srpm-builds/10018581/builder-live.log) [2026-01-16 08:14:22,717][ INFO][PID:3155064] Compressing /var/lib/copr/public_html/results/@teemtee/latest/srpm-builds/10018581/backend.log by gzip [2026-01-16 08:14:22,723][ INFO][PID:3155064] Running command 'gzip /var/lib/copr/public_html/results/@teemtee/latest/srpm-builds/10018581/backend.log' as PID 3227568