[2025-03-01 06:52:31,053][  INFO][PID:4010807] Marking build as starting
[2025-03-01 06:52:31,109][  INFO][PID:4010807] Trying to allocate VM
[2025-03-01 06:52:37,204][  INFO][PID:4010807] Allocated host ResallocHost, ticket_id=1574205, hostname=2620:52:3:1:dead:beef:cafe:c295, name=copr_p09_01_prod_01572545_20250301_062741
[2025-03-01 06:52:37,209][  INFO][PID:4010807] Allocating ssh connection to builder
[2025-03-01 06:52:37,210][  INFO][PID:4010807] Checking that builder machine is OK
[2025-03-01 06:52:38,181][  INFO][PID:4010807] Running remote command: copr-builder-ready fedora-40-ppc64le
[2025-03-01 06:52:38,737][  INFO][PID:4010807] Red Hat subscription not needed for fedora-40-ppc64le
Builder is ready to be used

[2025-03-01 06:52:38,738][  INFO][PID:4010807] Filling build.info file with builder info
[2025-03-01 06:52:38,743][  INFO][PID:4010807] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 36000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@kernel-vanilla",
            "project_name": "mainline-wo-mergew",
            "project_dirname": "mainline-wo-mergew",
            "submitter": "gitlab.com:knurd42",
            "ended_on": null,
            "started_on": 1740811958.743307,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-40-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8714211-fedora-40-ppc64le",
            "build_id": 8714211,
            "package_name": "mainline-womergew-releases",
            "package_version": "6.14.0-0.rc4.20250301gt03d38806.340.vanilla",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@kernel-vanilla/mainline-wo-mergew/mainline-womergew-releases",
            "git_hash": "9f9263ae219a4bf6a9c7dd7f57ac87d3c40fdd95",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@kernel-vanilla/mainline-wo-mergew--gitlab.com:knurd42",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@kernel-vanilla/mainline-wo-mergew/fedora-40-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@kernel-vanilla/mainline-wo-mergew",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@kernel-vanilla/mainline-wo-mergew",
            "result_dir": "08714211-mainline-womergew-releases",
            "built_packages": "",
            "id": 8714211,
            "mockchain_macros": {
                "copr_username": "@kernel-vanilla",
                "copr_projectname": "mainline-wo-mergew",
                "vendor": "Fedora Project COPR (@kernel-vanilla/mainline-wo-mergew)"
            }
        }
    ]
}
[2025-03-01 06:52:38,791][  INFO][PID:4010807] Sending fedora-messaging bus message in build.start
[2025-03-01 06:52:40,676][  INFO][PID:4010807] Sending fedora-messaging bus message in chroot.start
[2025-03-01 06:52:40,737][  INFO][PID:4010807] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8714211-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2025-03-01 06:52:41,507][  INFO][PID:4010807] Downloading the builder-live.log file, attempt 1
[2025-03-01 06:52:41,516][  INFO][PID:4010807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c295 copr-rpmbuild-log
[2025-03-01 06:52:46,542][  INFO][PID:4010807] Periodic builder liveness probe: alive
[2025-03-01 07:02:46,777][  INFO][PID:4010807] Periodic builder liveness probe: alive
[2025-03-01 07:12:47,169][  INFO][PID:4010807] Periodic builder liveness probe: alive
[2025-03-01 07:22:47,489][  INFO][PID:4010807] Periodic builder liveness probe: alive
[2025-03-01 07:32:47,723][  INFO][PID:4010807] Periodic builder liveness probe: alive
[2025-03-01 07:42:47,888][  INFO][PID:4010807] Periodic builder liveness probe: alive
[2025-03-01 07:52:48,036][  INFO][PID:4010807] Periodic builder liveness probe: alive
[2025-03-01 07:53:40,235][ ERROR][PID:4010807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-01 07:53:50,246][WARNING][PID:4010807] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-01 07:54:00,247][  INFO][PID:4010807] Downloading the builder-live.log file, attempt 2
[2025-03-01 07:54:00,254][  INFO][PID:4010807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c295 copr-rpmbuild-log
[2025-03-01 07:55:00,326][ ERROR][PID:4010807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-01 07:55:10,327][WARNING][PID:4010807] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-01 07:55:20,328][  INFO][PID:4010807] Downloading the builder-live.log file, attempt 3
[2025-03-01 07:55:20,330][  INFO][PID:4010807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c295 copr-rpmbuild-log
[2025-03-01 07:56:20,400][ ERROR][PID:4010807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-01 07:56:30,403][WARNING][PID:4010807] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-01 07:56:40,405][  INFO][PID:4010807] Downloading the builder-live.log file, attempt 4
[2025-03-01 07:56:40,407][  INFO][PID:4010807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c295 copr-rpmbuild-log
[2025-03-01 07:57:40,478][ ERROR][PID:4010807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-01 07:57:50,487][ ERROR][PID:4010807] Giving up for unstable SSH, failures: 1740815630.2460887, 1740815710.3274078, 1740815790.4037433, 1740815870.4872582
[2025-03-01 07:57:50,489][ ERROR][PID:4010807] Re-try request for task on 'ResallocHost, ticket_id=1574205, hostname=2620:52:3:1:dead:beef:cafe:c295, name=copr_p09_01_prod_01572545_20250301_062741': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-03-01 07:57:50,496][  INFO][PID:4010807] Releasing VM back to pool
[2025-03-01 07:57:50,523][  INFO][PID:4010807] Retry #1 (on other host)
[2025-03-01 07:57:50,577][  INFO][PID:4010807] Trying to allocate VM
[2025-03-01 07:58:58,739][  INFO][PID:4010807] Allocated host ResallocHost, ticket_id=1575675, hostname=140.211.168.190, name=copr_osuosl_p09_prod_01573719_20250301_075501
[2025-03-01 07:58:58,739][  INFO][PID:4010807] Allocating ssh connection to builder
[2025-03-01 07:58:58,740][  INFO][PID:4010807] Checking that builder machine is OK
[2025-03-01 07:58:59,939][  INFO][PID:4010807] Running remote command: copr-builder-ready fedora-40-ppc64le
[2025-03-01 07:59:00,916][  INFO][PID:4010807] Red Hat subscription not needed for fedora-40-ppc64le
Builder is ready to be used

[2025-03-01 07:59:00,917][  INFO][PID:4010807] Filling build.info file with builder info
[2025-03-01 07:59:00,924][  INFO][PID:4010807] Not re-notifying FE and msg buses for the new host.
[2025-03-01 07:59:00,925][  INFO][PID:4010807] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8714211-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2025-03-01 07:59:02,235][  INFO][PID:4010807] Downloading the builder-live.log file, attempt 1
[2025-03-01 07:59:02,236][  INFO][PID:4010807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.190 copr-rpmbuild-log
[2025-03-01 08:02:52,310][  INFO][PID:4010807] Periodic builder liveness probe: alive
[2025-03-01 08:12:52,632][  INFO][PID:4010807] Periodic builder liveness probe: alive
[2025-03-01 08:22:53,064][  INFO][PID:4010807] Periodic builder liveness probe: alive
[2025-03-01 08:32:53,315][  INFO][PID:4010807] Periodic builder liveness probe: dead
[2025-03-01 08:33:53,836][ ERROR][PID:4010807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
Connection timed out during banner exchange
Connection to 140.211.168.190 port 22 timed out

[2025-03-01 08:34:52,233][ ERROR][PID:4010807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-01 08:35:02,236][WARNING][PID:4010807] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-01 08:35:03,927][ ERROR][PID:4010807] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.190 port 22: Connection timed out

[2025-03-01 08:35:12,238][  INFO][PID:4010807] Downloading the builder-live.log file, attempt 2
[2025-03-01 08:35:12,246][  INFO][PID:4010807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.190 copr-rpmbuild-log
[2025-03-01 08:36:12,315][ ERROR][PID:4010807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-01 08:36:14,004][ ERROR][PID:4010807] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.190 port 22: Connection timed out

[2025-03-01 08:36:22,456][WARNING][PID:4010807] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-01 08:36:32,457][  INFO][PID:4010807] Downloading the builder-live.log file, attempt 3
[2025-03-01 08:36:32,459][  INFO][PID:4010807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.190 copr-rpmbuild-log
[2025-03-01 08:37:24,073][ ERROR][PID:4010807] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.190 port 22: Connection timed out

[2025-03-01 08:37:32,527][ ERROR][PID:4010807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-01 08:37:34,074][ ERROR][PID:4010807] Can't ssh to cancel build.
[2025-03-01 08:37:42,560][WARNING][PID:4010807] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-01 08:37:52,561][  INFO][PID:4010807] Downloading the builder-live.log file, attempt 4
[2025-03-01 08:37:52,563][  INFO][PID:4010807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.190 copr-rpmbuild-log
[2025-03-01 08:38:52,631][ ERROR][PID:4010807] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-01 08:39:02,704][ ERROR][PID:4010807] Giving up for unstable SSH, failures: 1740818102.2364967, 1740818182.4567952, 1740818262.560632, 1740818342.704278
[2025-03-01 08:39:02,705][ ERROR][PID:4010807] Re-try request for task on 'ResallocHost, ticket_id=1575675, hostname=140.211.168.190, name=copr_osuosl_p09_prod_01573719_20250301_075501': SSH problems when downloading live log: Resalloc ticket FAILED
[2025-03-01 08:39:02,706][  INFO][PID:4010807] Releasing VM back to pool
[2025-03-01 08:39:02,728][  INFO][PID:4010807] Retry #2 (on other host)
[2025-03-01 08:39:02,739][  INFO][PID:4010807] Trying to allocate VM
[2025-03-01 08:39:05,782][  INFO][PID:4010807] Allocated host ResallocHost, ticket_id=1576653, hostname=2620:52:3:1:dead:beef:cafe:c291, name=copr_p09_01_prod_01574202_20250301_082222
[2025-03-01 08:39:05,783][  INFO][PID:4010807] Allocating ssh connection to builder
[2025-03-01 08:39:05,783][  INFO][PID:4010807] Checking that builder machine is OK
[2025-03-01 08:39:06,485][  INFO][PID:4010807] Running remote command: copr-builder-ready fedora-40-ppc64le
[2025-03-01 08:39:07,462][  INFO][PID:4010807] Red Hat subscription not needed for fedora-40-ppc64le
Builder is ready to be used

[2025-03-01 08:39:07,898][  INFO][PID:4010807] Filling build.info file with builder info
[2025-03-01 08:39:08,230][  INFO][PID:4010807] Not re-notifying FE and msg buses for the new host.
[2025-03-01 08:39:08,231][  INFO][PID:4010807] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8714211-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2025-03-01 08:39:09,541][  INFO][PID:4010807] Downloading the builder-live.log file, attempt 1
[2025-03-01 08:39:09,543][  INFO][PID:4010807] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c291 copr-rpmbuild-log
[2025-03-01 08:39:19,374][  INFO][PID:4010807] Cancel request succeeded
out:
success
err:
Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c291' (ED25519) to the list of known hosts.

[2025-03-01 08:39:19,376][  INFO][PID:4010807] Downloading results from builder
[2025-03-01 08:39:19,384][  INFO][PID:4010807] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c291]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@kernel-vanilla/mainline-wo-mergew/fedora-40-ppc64le/08714211-mainline-womergew-releases started
[2025-03-01 08:39:19,400][  INFO][PID:4010807] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c291]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@kernel-vanilla/mainline-wo-mergew/fedora-40-ppc64le/08714211-mainline-womergew-releases/ &> /var/lib/copr/public_html/results/@kernel-vanilla/mainline-wo-mergew/fedora-40-ppc64le/08714211-mainline-womergew-releases/build-08714211.rsync.log
[2025-03-01 08:39:19,883][  INFO][PID:4010807] rsyncing finished.
[2025-03-01 08:39:19,884][  INFO][PID:4010807] Releasing VM back to pool
[2025-03-01 08:39:19,903][  INFO][PID:4010807] Searching for 'success' file in resultdir
[2025-03-01 08:39:19,907][ ERROR][PID:4010807] Build failed: Backend process error: No success file => build failure
[2025-03-01 08:39:20,161][  INFO][PID:4010807] Finished build: id=8714211 failed=True timeout=36000 destdir=/var/lib/copr/public_html/results/@kernel-vanilla/mainline-wo-mergew chroot=fedora-40-ppc64le 
[2025-03-01 08:39:20,281][ ERROR][PID:4010807] Backend process error: No success file => build failure
[2025-03-01 08:39:20,284][  INFO][PID:4010807] Worker failed build, took 12.05443549156189
[2025-03-01 08:39:20,298][  INFO][PID:4010807] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 36000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@kernel-vanilla",
            "project_name": "mainline-wo-mergew",
            "project_dirname": "mainline-wo-mergew",
            "submitter": "gitlab.com:knurd42",
            "ended_on": 1740818360.2849095,
            "started_on": 1740818348.230474,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-40-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8714211-fedora-40-ppc64le",
            "build_id": 8714211,
            "package_name": "mainline-womergew-releases",
            "package_version": "6.14.0-0.rc4.20250301gt03d38806.340.vanilla",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@kernel-vanilla/mainline-wo-mergew/mainline-womergew-releases",
            "git_hash": "9f9263ae219a4bf6a9c7dd7f57ac87d3c40fdd95",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@kernel-vanilla/mainline-wo-mergew--gitlab.com:knurd42",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@kernel-vanilla/mainline-wo-mergew/fedora-40-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@kernel-vanilla/mainline-wo-mergew",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@kernel-vanilla/mainline-wo-mergew",
            "result_dir": "08714211-mainline-womergew-releases",
            "built_packages": "",
            "id": 8714211,
            "mockchain_macros": {
                "copr_username": "@kernel-vanilla",
                "copr_projectname": "mainline-wo-mergew",
                "vendor": "Fedora Project COPR (@kernel-vanilla/mainline-wo-mergew)"
            }
        }
    ]
}
[2025-03-01 08:39:21,718][  INFO][PID:4010807] Sending fedora-messaging bus message in build.end
[2025-03-01 08:39:22,245][  INFO][PID:4010807] Compressing /var/lib/copr/public_html/results/@kernel-vanilla/mainline-wo-mergew/fedora-40-ppc64le/08714211-mainline-womergew-releases/builder-live.log by gzip
[2025-03-01 08:39:22,276][  INFO][PID:4010807] Running command 'gzip /var/lib/copr/public_html/results/@kernel-vanilla/mainline-wo-mergew/fedora-40-ppc64le/08714211-mainline-womergew-releases/builder-live.log' as PID 920673
[2025-03-01 08:39:22,327][  INFO][PID:4010807] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@kernel-vanilla/mainline-wo-mergew/fedora-40-ppc64le/08714211-mainline-womergew-releases/builder-live.log)
[2025-03-01 08:39:22,342][  INFO][PID:4010807] Compressing /var/lib/copr/public_html/results/@kernel-vanilla/mainline-wo-mergew/fedora-40-ppc64le/08714211-mainline-womergew-releases/backend.log by gzip
[2025-03-01 08:39:22,344][  INFO][PID:4010807] Running command 'gzip /var/lib/copr/public_html/results/@kernel-vanilla/mainline-wo-mergew/fedora-40-ppc64le/08714211-mainline-womergew-releases/backend.log' as PID 920679