[2025-03-06 14:24:33,295][ INFO][PID:3144768] Marking build as starting
[2025-03-06 14:24:33,451][ INFO][PID:3144768] Trying to allocate VM
[2025-03-06 14:25:11,552][ INFO][PID:3144768] Allocated host ResallocHost, ticket_id=1678110, hostname=54.152.109.111, name=aws_x86_64_spot_prod_01724910_20250306_142335
[2025-03-06 14:25:11,552][ INFO][PID:3144768] Allocating ssh connection to builder
[2025-03-06 14:25:11,553][ INFO][PID:3144768] Checking that builder machine is OK
[2025-03-06 14:25:11,928][ INFO][PID:3144768] Running remote command: copr-builder-ready rhel-9-x86_64
[2025-03-06 14:25:42,155][ INFO][PID:3144768] Checking Red Hat subscription (attempt #1)
Checking Red Hat subscription (attempt #2)
Red Hat subscription active
Builder is ready to be used
[2025-03-06 14:25:42,156][ INFO][PID:3144768] Filling build.info file with builder info
[2025-03-06 14:25:42,158][ INFO][PID:3144768] Sending build state back to frontend: {
"builds": [
{
"timeout": 18000,
"frontend_base_url": "https://copr.fedorainfracloud.org",
"memory_reqs": 2048,
"enable_net": false,
"project_owner": "@yggdrasil",
"project_name": "latest",
"project_dirname": "latest",
"submitter": "packit",
"ended_on": null,
"started_on": 1741271142.1578777,
"submitted_on": null,
"status": 3,
"chroot": "rhel-9-x86_64",
"arch": "x86_64",
"buildroot_pkgs": [],
"task_id": "8736127-rhel-9-x86_64",
"build_id": 8736127,
"package_name": "yggdrasil",
"package_version": "0.4.5-99.20250306git1a5a7fb.fc41",
"git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@yggdrasil/latest/yggdrasil",
"git_hash": "0213f8aec121f1508f78302128a23a5f01023a1a",
"git_branch": null,
"source_type": null,
"source_json": null,
"pkg_name": null,
"pkg_main_version": null,
"pkg_epoch": null,
"pkg_release": null,
"srpm_url": null,
"uses_devel_repo": false,
"sandbox": "@yggdrasil/latest--packit",
"results": null,
"appstream": false,
"allow_user_ssh": false,
"ssh_public_keys": null,
"storage": null,
"background": false,
"fedora_review": false,
"isolation": "default",
"repo_priority": null,
"repos": [
{
"baseurl": "https://download.copr.fedorainfracloud.org/results/@yggdrasil/latest/rhel-9-x86_64/",
"id": "copr_base",
"name": "Copr repository",
"priority": null
}
],
"tags": [
"arch_x86_64"
],
"with_opts": [],
"without_opts": [],
"destdir": "/var/lib/copr/public_html/results/@yggdrasil/latest",
"results_repo_url": "https://download.copr.fedorainfracloud.org/results/@yggdrasil/latest",
"result_dir": "08736127-yggdrasil",
"built_packages": "",
"id": 8736127,
"mockchain_macros": {
"copr_username": "@yggdrasil",
"copr_projectname": "latest",
"vendor": "Fedora Project COPR (@yggdrasil/latest)"
}
}
]
}
[2025-03-06 14:25:42,220][ INFO][PID:3144768] Sending fedora-messaging bus message in build.start
[2025-03-06 14:25:43,032][ INFO][PID:3144768] Sending fedora-messaging bus message in chroot.start
[2025-03-06 14:25:43,054][ INFO][PID:3144768] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8736127-rhel-9-x86_64 --chroot rhel-9-x86_64 --detached
[2025-03-06 14:25:43,470][ INFO][PID:3144768] Downloading the builder-live.log file, attempt 1
[2025-03-06 14:25:43,472][ INFO][PID:3144768] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.152.109.111 copr-rpmbuild-log
[2025-03-06 14:25:48,481][ INFO][PID:3144768] Periodic builder liveness probe: alive
[2025-03-06 14:27:59,323][ ERROR][PID:3144768] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-06 14:28:09,324][WARNING][PID:3144768] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-06 14:28:19,325][ INFO][PID:3144768] Downloading the builder-live.log file, attempt 2
[2025-03-06 14:28:19,327][ INFO][PID:3144768] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.152.109.111 copr-rpmbuild-log
[2025-03-06 14:29:19,396][ ERROR][PID:3144768] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-06 14:29:29,402][WARNING][PID:3144768] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-06 14:29:39,404][ INFO][PID:3144768] Downloading the builder-live.log file, attempt 3
[2025-03-06 14:29:39,406][ INFO][PID:3144768] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.152.109.111 copr-rpmbuild-log
[2025-03-06 14:30:39,475][ ERROR][PID:3144768] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-06 14:30:49,478][WARNING][PID:3144768] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-06 14:30:59,479][ INFO][PID:3144768] Downloading the builder-live.log file, attempt 4
[2025-03-06 14:30:59,481][ INFO][PID:3144768] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.152.109.111 copr-rpmbuild-log
[2025-03-06 14:31:59,550][ ERROR][PID:3144768] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-06 14:32:09,555][ ERROR][PID:3144768] Giving up for unstable SSH, failures: 1741271289.3248913, 1741271369.4026654, 1741271449.478913, 1741271529.555559
[2025-03-06 14:32:09,556][ ERROR][PID:3144768] Re-try request for task on 'ResallocHost, ticket_id=1678110, hostname=54.152.109.111, name=aws_x86_64_spot_prod_01724910_20250306_142335': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-03-06 14:32:09,557][ INFO][PID:3144768] Releasing VM back to pool
[2025-03-06 14:32:09,597][ INFO][PID:3144768] Retry #1 (on other host)
[2025-03-06 14:32:09,598][ INFO][PID:3144768] Trying to allocate VM
[2025-03-06 14:32:12,627][ INFO][PID:3144768] Allocated host ResallocHost, ticket_id=1678187, hostname=2620:52:3:1:dead:beef:cafe:c1d3, name=copr_hv_x86_64_04_prod_01725142_20250306_142953
[2025-03-06 14:32:12,627][ INFO][PID:3144768] Allocating ssh connection to builder
[2025-03-06 14:32:12,628][ INFO][PID:3144768] Checking that builder machine is OK
[2025-03-06 14:32:13,240][ INFO][PID:3144768] Running remote command: copr-builder-ready rhel-9-x86_64
[2025-03-06 14:32:13,598][ INFO][PID:3144768] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used
[2025-03-06 14:32:13,599][ INFO][PID:3144768] Filling build.info file with builder info
[2025-03-06 14:32:13,604][ INFO][PID:3144768] Not re-notifying FE and msg buses for the new host.
[2025-03-06 14:32:13,605][ INFO][PID:3144768] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8736127-rhel-9-x86_64 --chroot rhel-9-x86_64 --detached
[2025-03-06 14:32:14,181][ INFO][PID:3144768] Downloading the builder-live.log file, attempt 1
[2025-03-06 14:32:14,182][ INFO][PID:3144768] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c1d3 copr-rpmbuild-log
[2025-03-06 14:35:47,234][ INFO][PID:3144768] Downloading results from builder
[2025-03-06 14:35:47,234][ INFO][PID:3144768] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c1d3]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil started
[2025-03-06 14:35:47,235][ INFO][PID:3144768] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c1d3]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/ &> /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/build-08736127.rsync.log
[2025-03-06 14:35:47,899][ INFO][PID:3144768] rsyncing finished.
[2025-03-06 14:35:47,900][ INFO][PID:3144768] Releasing VM back to pool
[2025-03-06 14:35:47,916][ INFO][PID:3144768] Searching for 'success' file in resultdir
[2025-03-06 14:35:47,917][ INFO][PID:3144768] Going to sign pkgs from source: 8736127-rhel-9-x86_64 in chroot: /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64
[2025-03-06 14:35:47,918][ INFO][PID:3144768] Calling '/bin/sign -u @yggdrasil#latest@copr.fedorahosted.org -p' (attempt #1)
[2025-03-06 14:35:48,097][ INFO][PID:3144768] Calling '/bin/sign -4 -h sha256 -u @yggdrasil#latest@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/yggdrasil-debuginfo-0.4.5-99.20250306git1a5a7fb.el9.x86_64.rpm' (attempt #1)
[2025-03-06 14:35:48,869][ INFO][PID:3144768] signed rpm: /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/yggdrasil-debuginfo-0.4.5-99.20250306git1a5a7fb.el9.x86_64.rpm
[2025-03-06 14:35:48,869][ INFO][PID:3144768] Calling '/bin/sign -4 -h sha256 -u @yggdrasil#latest@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/yggdrasil-0.4.5-99.20250306git1a5a7fb.el9.src.rpm' (attempt #1)
[2025-03-06 14:35:49,721][ INFO][PID:3144768] signed rpm: /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/yggdrasil-0.4.5-99.20250306git1a5a7fb.el9.src.rpm
[2025-03-06 14:35:49,721][ INFO][PID:3144768] Calling '/bin/sign -4 -h sha256 -u @yggdrasil#latest@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/yggdrasil-debugsource-0.4.5-99.20250306git1a5a7fb.el9.x86_64.rpm' (attempt #1)
[2025-03-06 14:35:50,416][ INFO][PID:3144768] signed rpm: /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/yggdrasil-debugsource-0.4.5-99.20250306git1a5a7fb.el9.x86_64.rpm
[2025-03-06 14:35:50,417][ INFO][PID:3144768] Calling '/bin/sign -4 -h sha256 -u @yggdrasil#latest@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/yggdrasil-devel-0.4.5-99.20250306git1a5a7fb.el9.x86_64.rpm' (attempt #1)
[2025-03-06 14:35:51,333][ INFO][PID:3144768] signed rpm: /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/yggdrasil-devel-0.4.5-99.20250306git1a5a7fb.el9.x86_64.rpm
[2025-03-06 14:35:51,334][ INFO][PID:3144768] Calling '/bin/sign -4 -h sha256 -u @yggdrasil#latest@copr.fedorahosted.org -r /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/yggdrasil-0.4.5-99.20250306git1a5a7fb.el9.x86_64.rpm' (attempt #1)
[2025-03-06 14:35:52,101][ INFO][PID:3144768] signed rpm: /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/yggdrasil-0.4.5-99.20250306git1a5a7fb.el9.x86_64.rpm
[2025-03-06 14:35:52,102][ INFO][PID:3144768] Sign done
[2025-03-06 14:35:52,103][ INFO][PID:3144768] Incremental createrepo run, adding 08736127-yggdrasil into https://download.copr.fedorainfracloud.org/results/@yggdrasil/latest/rhel-9-x86_64, (auto-create-repo=True)
[2025-03-06 14:35:52,104][ INFO][PID:3144768] Running command 'copr-repo --batched /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64 --add 08736127-yggdrasil --no-appstream-metadata' as PID 3248702
[2025-03-06 14:35:53,091][ INFO][PID:3144768] Finished after 0 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64 --add 08736127-yggdrasil --no-appstream-metadata)
[2025-03-06 14:35:53,092][ INFO][PID:3144768] Getting build details
[2025-03-06 14:35:53,093][ INFO][PID:3144768] Listing built binary packages in /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil
[2025-03-06 14:35:53,093][ INFO][PID:3144768] Built packages:
['yggdrasil-devel 0.4.5', 'yggdrasil 0.4.5', 'yggdrasil-debugsource 0.4.5', 'yggdrasil-debuginfo 0.4.5']
[2025-03-06 14:35:53,094][ INFO][PID:3144768] build details: {'built_packages': 'yggdrasil-devel 0.4.5\nyggdrasil 0.4.5\nyggdrasil-debugsource 0.4.5\nyggdrasil-debuginfo 0.4.5'}
[2025-03-06 14:35:53,095][ INFO][PID:3144768] Retrieving pubkey
[2025-03-06 14:35:53,095][ INFO][PID:3144768] Calling '/bin/sign -u @yggdrasil#latest@copr.fedorahosted.org -p' (attempt #1)
[2025-03-06 14:35:53,406][ INFO][PID:3144768] Added pubkey for user @yggdrasil project latest into: /var/lib/copr/public_html/results/@yggdrasil/latest/pubkey.gpg
[2025-03-06 14:35:53,407][ INFO][PID:3144768] Finished build: id=8736127 failed=False timeout=18000 destdir=/var/lib/copr/public_html/results/@yggdrasil/latest chroot=rhel-9-x86_64
[2025-03-06 14:35:53,409][ INFO][PID:3144768] Worker succeeded build, took 219.805082321167
[2025-03-06 14:35:53,410][ INFO][PID:3144768] Sending build state back to frontend: {
"builds": [
{
"timeout": 18000,
"frontend_base_url": "https://copr.fedorainfracloud.org",
"memory_reqs": 2048,
"enable_net": false,
"project_owner": "@yggdrasil",
"project_name": "latest",
"project_dirname": "latest",
"submitter": "packit",
"ended_on": 1741271753.409476,
"started_on": 1741271533.6043937,
"submitted_on": null,
"status": 1,
"chroot": "rhel-9-x86_64",
"arch": "x86_64",
"buildroot_pkgs": [],
"task_id": "8736127-rhel-9-x86_64",
"build_id": 8736127,
"package_name": "yggdrasil",
"package_version": "0.4.5-99.20250306git1a5a7fb.fc41",
"git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@yggdrasil/latest/yggdrasil",
"git_hash": "0213f8aec121f1508f78302128a23a5f01023a1a",
"git_branch": null,
"source_type": null,
"source_json": null,
"pkg_name": null,
"pkg_main_version": null,
"pkg_epoch": null,
"pkg_release": null,
"srpm_url": null,
"uses_devel_repo": false,
"sandbox": "@yggdrasil/latest--packit",
"results": {
"packages": [
{
"name": "yggdrasil-devel",
"epoch": null,
"version": "0.4.5",
"release": "99.20250306git1a5a7fb.el9",
"arch": "x86_64"
},
{
"name": "yggdrasil",
"epoch": null,
"version": "0.4.5",
"release": "99.20250306git1a5a7fb.el9",
"arch": "x86_64"
},
{
"name": "yggdrasil",
"epoch": null,
"version": "0.4.5",
"release": "99.20250306git1a5a7fb.el9",
"arch": "src"
},
{
"name": "yggdrasil-debugsource",
"epoch": null,
"version": "0.4.5",
"release": "99.20250306git1a5a7fb.el9",
"arch": "x86_64"
},
{
"name": "yggdrasil-debuginfo",
"epoch": null,
"version": "0.4.5",
"release": "99.20250306git1a5a7fb.el9",
"arch": "x86_64"
}
]
},
"appstream": false,
"allow_user_ssh": false,
"ssh_public_keys": null,
"storage": null,
"background": false,
"fedora_review": false,
"isolation": "default",
"repo_priority": null,
"repos": [
{
"baseurl": "https://download.copr.fedorainfracloud.org/results/@yggdrasil/latest/rhel-9-x86_64/",
"id": "copr_base",
"name": "Copr repository",
"priority": null
}
],
"tags": [
"arch_x86_64"
],
"with_opts": [],
"without_opts": [],
"destdir": "/var/lib/copr/public_html/results/@yggdrasil/latest",
"results_repo_url": "https://download.copr.fedorainfracloud.org/results/@yggdrasil/latest",
"result_dir": "08736127-yggdrasil",
"built_packages": "yggdrasil-devel 0.4.5\nyggdrasil 0.4.5\nyggdrasil-debugsource 0.4.5\nyggdrasil-debuginfo 0.4.5",
"id": 8736127,
"mockchain_macros": {
"copr_username": "@yggdrasil",
"copr_projectname": "latest",
"vendor": "Fedora Project COPR (@yggdrasil/latest)"
}
}
]
}
[2025-03-06 14:35:53,458][ INFO][PID:3144768] Sending fedora-messaging bus message in build.end
[2025-03-06 14:35:53,478][ INFO][PID:3144768] Compressing /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/builder-live.log by gzip
[2025-03-06 14:35:53,480][ INFO][PID:3144768] Running command 'gzip /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/builder-live.log' as PID 3248950
[2025-03-06 14:35:53,518][ INFO][PID:3144768] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/builder-live.log)
[2025-03-06 14:35:53,519][ INFO][PID:3144768] Compressing /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/backend.log by gzip
[2025-03-06 14:35:53,520][ INFO][PID:3144768] Running command 'gzip /var/lib/copr/public_html/results/@yggdrasil/latest/rhel-9-x86_64/08736127-yggdrasil/backend.log' as PID 3248959