[2025-04-27 04:27:08,317][  INFO][PID:3201835] Marking build as starting
[2025-04-27 04:27:08,438][  INFO][PID:3201835] Trying to allocate VM: ResallocHost, ticket_id=2679990, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-27 04:27:11,465][  INFO][PID:3201835] Allocated host ResallocHost, ticket_id=2679990, hostname=140.211.168.243, name=copr_osuosl_p09_prod_03404069_20250427_040210, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-27 04:27:11,466][  INFO][PID:3201835] Allocating ssh connection to builder
[2025-04-27 04:27:11,467][  INFO][PID:3201835] Checking that builder machine is OK
[2025-04-27 04:27:12,776][  INFO][PID:3201835] Running remote command: copr-builder-ready fedora-40-ppc64le
[2025-04-27 04:27:13,781][  INFO][PID:3201835] Red Hat subscription not needed for fedora-40-ppc64le
Builder is ready to be used

[2025-04-27 04:27:13,782][  INFO][PID:3201835] Filling build.info file with builder info
[2025-04-27 04:27:13,784][  INFO][PID:3201835] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "luya",
            "project_name": "xournalpp",
            "project_dirname": "xournalpp",
            "submitter": "https://api.github.com/users/github-actions%5Bbot%5D",
            "ended_on": null,
            "started_on": 1745728033.7844942,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-40-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8970227-fedora-40-ppc64le",
            "build_id": 8970227,
            "package_name": "xournalpp",
            "package_version": "1.2.6^400.g28791d76-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/luya/xournalpp/xournalpp",
            "git_hash": "083571d6bea9fca43ac674d17407e4d889df12ae",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "luya/xournalpp--https://api.github.com/users/github-actions%5Bbot%5D",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": true,
            "bootstrap": "on",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/luya/xournalpp/fedora-40-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/luya/xournalpp",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/luya/xournalpp",
            "result_dir": "08970227-xournalpp",
            "built_packages": "",
            "id": 8970227,
            "mockchain_macros": {
                "copr_username": "luya",
                "copr_projectname": "xournalpp",
                "vendor": "Fedora Project COPR (luya/xournalpp)"
            }
        }
    ]
}
[2025-04-27 04:27:13,826][  INFO][PID:3201835] Sending fedora-messaging bus message in build.start
[2025-04-27 04:27:14,582][  INFO][PID:3201835] Sending fedora-messaging bus message in chroot.start
[2025-04-27 04:27:14,600][  INFO][PID:3201835] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8970227-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2025-04-27 04:27:15,962][  INFO][PID:3201835] Downloading the builder-live.log file, attempt 1
[2025-04-27 04:27:15,965][  INFO][PID:3201835] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log
[2025-04-27 04:27:20,970][  INFO][PID:3201835] Periodic builder liveness probe: alive
[2025-04-27 04:37:21,118][  INFO][PID:3201835] Periodic builder liveness probe: alive
[2025-04-27 04:43:02,140][ ERROR][PID:3201835] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-27 04:43:12,141][WARNING][PID:3201835] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-27 04:43:22,142][  INFO][PID:3201835] Downloading the builder-live.log file, attempt 2
[2025-04-27 04:43:22,144][  INFO][PID:3201835] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log
[2025-04-27 04:44:22,213][ ERROR][PID:3201835] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-27 04:44:32,215][WARNING][PID:3201835] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-27 04:44:42,216][  INFO][PID:3201835] Downloading the builder-live.log file, attempt 3
[2025-04-27 04:44:42,221][  INFO][PID:3201835] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log
[2025-04-27 04:45:42,302][ ERROR][PID:3201835] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-27 04:45:52,308][WARNING][PID:3201835] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-27 04:46:02,312][  INFO][PID:3201835] Downloading the builder-live.log file, attempt 4
[2025-04-27 04:46:02,314][  INFO][PID:3201835] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.243 copr-rpmbuild-log
[2025-04-27 04:47:02,382][ ERROR][PID:3201835] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-27 04:47:12,384][ ERROR][PID:3201835] Giving up for unstable SSH, failures: 1745728992.1413362, 1745729072.2150807, 1745729152.3081017, 1745729232.3842337
[2025-04-27 04:47:12,385][ ERROR][PID:3201835] Re-try request for task on 'ResallocHost, ticket_id=2679990, hostname=140.211.168.243, name=copr_osuosl_p09_prod_03404069_20250427_040210, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-04-27 04:47:12,386][  INFO][PID:3201835] Releasing VM back to pool
[2025-04-27 04:47:12,397][  INFO][PID:3201835] Retry #1 (on other host)
[2025-04-27 04:47:12,447][  INFO][PID:3201835] Trying to allocate VM: ResallocHost, ticket_id=2680105, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-27 04:47:15,485][  INFO][PID:3201835] Allocated host ResallocHost, ticket_id=2680105, hostname=140.211.168.171, name=copr_osuosl_p09_prod_03405172_20250427_044044, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-27 04:47:15,485][  INFO][PID:3201835] Allocating ssh connection to builder
[2025-04-27 04:47:15,486][  INFO][PID:3201835] Checking that builder machine is OK
[2025-04-27 04:47:16,770][  INFO][PID:3201835] Running remote command: copr-builder-ready fedora-40-ppc64le
[2025-04-27 04:47:17,763][  INFO][PID:3201835] Red Hat subscription not needed for fedora-40-ppc64le
Builder is ready to be used

[2025-04-27 04:47:17,764][  INFO][PID:3201835] Filling build.info file with builder info
[2025-04-27 04:47:17,768][  INFO][PID:3201835] Not re-notifying FE and msg buses for the new host.
[2025-04-27 04:47:17,769][  INFO][PID:3201835] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8970227-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2025-04-27 04:47:19,064][  INFO][PID:3201835] Downloading the builder-live.log file, attempt 1
[2025-04-27 04:47:19,066][  INFO][PID:3201835] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.171 copr-rpmbuild-log
[2025-04-27 04:47:24,070][  INFO][PID:3201835] Periodic builder liveness probe: alive
[2025-04-27 04:57:24,184][  INFO][PID:3201835] Periodic builder liveness probe: alive
[2025-04-27 05:03:22,139][ ERROR][PID:3201835] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-27 05:03:32,141][WARNING][PID:3201835] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-27 05:03:42,142][  INFO][PID:3201835] Downloading the builder-live.log file, attempt 2
[2025-04-27 05:03:42,144][  INFO][PID:3201835] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.171 copr-rpmbuild-log
[2025-04-27 05:04:42,211][ ERROR][PID:3201835] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-27 05:04:52,216][WARNING][PID:3201835] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-27 05:05:02,217][  INFO][PID:3201835] Downloading the builder-live.log file, attempt 3
[2025-04-27 05:05:02,218][  INFO][PID:3201835] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.171 copr-rpmbuild-log
[2025-04-27 05:06:02,286][ ERROR][PID:3201835] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-27 05:06:12,287][WARNING][PID:3201835] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-27 05:06:22,288][  INFO][PID:3201835] Downloading the builder-live.log file, attempt 4
[2025-04-27 05:06:22,292][  INFO][PID:3201835] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.171 copr-rpmbuild-log
[2025-04-27 05:07:22,359][ ERROR][PID:3201835] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-27 05:07:24,307][  INFO][PID:3201835] Periodic builder liveness probe: dead
[2025-04-27 05:07:32,361][ ERROR][PID:3201835] Giving up for unstable SSH, failures: 1745730212.1413927, 1745730292.2161047, 1745730372.2874072, 1745730452.3612819
[2025-04-27 05:08:24,398][ ERROR][PID:3201835] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.171 port 22: Connection timed out

[2025-04-27 05:09:34,488][ ERROR][PID:3201835] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.171 port 22: Connection timed out

[2025-04-27 05:10:44,570][ ERROR][PID:3201835] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.171 port 22: Connection timed out

[2025-04-27 05:11:54,647][ ERROR][PID:3201835] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.171 port 22: Connection timed out

[2025-04-27 05:12:04,651][ ERROR][PID:3201835] Can't ssh to cancel build.
[2025-04-27 05:12:04,652][ ERROR][PID:3201835] Re-try request for task on 'ResallocHost, ticket_id=2680105, hostname=140.211.168.171, name=copr_osuosl_p09_prod_03405172_20250427_044044, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED
[2025-04-27 05:12:04,653][  INFO][PID:3201835] Releasing VM back to pool
[2025-04-27 05:12:04,664][  INFO][PID:3201835] Retry #2 (on other host)
[2025-04-27 05:12:04,704][  INFO][PID:3201835] Trying to allocate VM: ResallocHost, ticket_id=2680190, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-27 05:12:07,744][  INFO][PID:3201835] Allocated host ResallocHost, ticket_id=2680190, hostname=140.211.168.240, name=copr_osuosl_p09_prod_03405885_20250427_050554, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-04-27 05:12:07,746][  INFO][PID:3201835] Allocating ssh connection to builder
[2025-04-27 05:12:07,746][  INFO][PID:3201835] Checking that builder machine is OK
[2025-04-27 05:12:09,091][  INFO][PID:3201835] Running remote command: copr-builder-ready fedora-40-ppc64le
[2025-04-27 05:12:10,077][  INFO][PID:3201835] Red Hat subscription not needed for fedora-40-ppc64le
Builder is ready to be used

[2025-04-27 05:12:10,078][  INFO][PID:3201835] Filling build.info file with builder info
[2025-04-27 05:12:10,084][  INFO][PID:3201835] Not re-notifying FE and msg buses for the new host.
[2025-04-27 05:12:10,085][  INFO][PID:3201835] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8970227-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached
[2025-04-27 05:12:11,421][  INFO][PID:3201835] Downloading the builder-live.log file, attempt 1
[2025-04-27 05:12:11,423][  INFO][PID:3201835] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.240 copr-rpmbuild-log
[2025-04-27 05:12:20,667][  INFO][PID:3201835] Cancel request succeeded
out:
success
err:
Warning: Permanently added '140.211.168.240' (ED25519) to the list of known hosts.

[2025-04-27 05:12:20,668][  INFO][PID:3201835] Downloading results from builder
[2025-04-27 05:12:20,669][  INFO][PID:3201835] rsyncing of mockbuilder@140.211.168.240:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/luya/xournalpp/fedora-40-ppc64le/08970227-xournalpp started
[2025-04-27 05:12:20,670][  INFO][PID:3201835] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@140.211.168.240:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/luya/xournalpp/fedora-40-ppc64le/08970227-xournalpp/ &> /var/lib/copr/public_html/results/luya/xournalpp/fedora-40-ppc64le/08970227-xournalpp/build-08970227.rsync.log
[2025-04-27 05:12:22,237][  INFO][PID:3201835] rsyncing finished.
[2025-04-27 05:12:22,238][  INFO][PID:3201835] Releasing VM back to pool
[2025-04-27 05:12:22,250][  INFO][PID:3201835] Searching for 'success' file in resultdir
[2025-04-27 05:12:22,251][ ERROR][PID:3201835] Build failed: Backend process error: No success file => build failure
[2025-04-27 05:12:22,252][  INFO][PID:3201835] Finished build: id=8970227 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/luya/xournalpp chroot=fedora-40-ppc64le 
[2025-04-27 05:12:22,256][ ERROR][PID:3201835] Backend process error: No success file => build failure
[2025-04-27 05:12:22,257][  INFO][PID:3201835] Worker failed build, took 12.172767639160156
[2025-04-27 05:12:22,258][  INFO][PID:3201835] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "luya",
            "project_name": "xournalpp",
            "project_dirname": "xournalpp",
            "submitter": "https://api.github.com/users/github-actions%5Bbot%5D",
            "ended_on": 1745730742.25709,
            "started_on": 1745730730.0843225,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-40-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8970227-fedora-40-ppc64le",
            "build_id": 8970227,
            "package_name": "xournalpp",
            "package_version": "1.2.6^400.g28791d76-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/luya/xournalpp/xournalpp",
            "git_hash": "083571d6bea9fca43ac674d17407e4d889df12ae",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "luya/xournalpp--https://api.github.com/users/github-actions%5Bbot%5D",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": true,
            "bootstrap": "on",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/luya/xournalpp/fedora-40-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/luya/xournalpp",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/luya/xournalpp",
            "result_dir": "08970227-xournalpp",
            "built_packages": "",
            "id": 8970227,
            "mockchain_macros": {
                "copr_username": "luya",
                "copr_projectname": "xournalpp",
                "vendor": "Fedora Project COPR (luya/xournalpp)"
            }
        }
    ]
}
[2025-04-27 05:12:22,294][  INFO][PID:3201835] Sending fedora-messaging bus message in build.end
[2025-04-27 05:12:22,318][  INFO][PID:3201835] Compressing /var/lib/copr/public_html/results/luya/xournalpp/fedora-40-ppc64le/08970227-xournalpp/builder-live.log by gzip
[2025-04-27 05:12:22,320][  INFO][PID:3201835] Running command 'gzip /var/lib/copr/public_html/results/luya/xournalpp/fedora-40-ppc64le/08970227-xournalpp/builder-live.log' as PID 3482080
[2025-04-27 05:12:22,322][  INFO][PID:3201835] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/luya/xournalpp/fedora-40-ppc64le/08970227-xournalpp/builder-live.log)
[2025-04-27 05:12:22,323][  INFO][PID:3201835] Compressing /var/lib/copr/public_html/results/luya/xournalpp/fedora-40-ppc64le/08970227-xournalpp/backend.log by gzip
[2025-04-27 05:12:22,324][  INFO][PID:3201835] Running command 'gzip /var/lib/copr/public_html/results/luya/xournalpp/fedora-40-ppc64le/08970227-xournalpp/backend.log' as PID 3482083