[2025-04-28 06:27:36,542][ INFO][PID:2611018] Marking build as starting [2025-04-28 06:27:36,656][ INFO][PID:2611018] Trying to allocate VM: ResallocHost, ticket_id=2689272, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 06:27:39,749][ INFO][PID:2611018] Allocated host ResallocHost, ticket_id=2689272, hostname=140.211.168.104, name=copr_osuosl_p09_prod_03446669_20250428_055723, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 06:27:39,750][ INFO][PID:2611018] Allocating ssh connection to builder [2025-04-28 06:27:39,751][ INFO][PID:2611018] Checking that builder machine is OK [2025-04-28 06:27:41,111][ INFO][PID:2611018] Running remote command: copr-builder-ready fedora-40-ppc64le [2025-04-28 06:27:42,178][ INFO][PID:2611018] Red Hat subscription not needed for fedora-40-ppc64le Builder is ready to be used [2025-04-28 06:27:42,188][ INFO][PID:2611018] Filling build.info file with builder info [2025-04-28 06:27:42,199][ INFO][PID:2611018] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "osloup", "project_name": "postgresql_update_15_2", "project_dirname": "postgresql_update_15_2", "submitter": "https://src.fedoraproject.org/user/jgrulich", "ended_on": null, "started_on": 1745821662.1994498, "submitted_on": null, "status": 3, "chroot": "fedora-40-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "8971985-fedora-40-ppc64le", "build_id": 8971985, "package_name": "qt6-qtbase", "package_version": "6.9.0-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osloup/postgresql_update_15_2/qt6-qtbase", "git_hash": "7b86756d2e09f075c999c756b51d512a97f82859", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "osloup/postgresql_update_15_2--https://src.fedoraproject.org/user/jgrulich", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/osloup/postgresql_update_15_2", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_update_15_2", "result_dir": "08971985-qt6-qtbase", "built_packages": "", "id": 8971985, "mockchain_macros": { "copr_username": "osloup", "copr_projectname": "postgresql_update_15_2", "vendor": "Fedora Project COPR (osloup/postgresql_update_15_2)" } } ] } [2025-04-28 06:27:42,246][ INFO][PID:2611018] Sending fedora-messaging bus message in build.start [2025-04-28 06:27:43,109][ INFO][PID:2611018] Sending fedora-messaging bus message in chroot.start [2025-04-28 06:27:43,130][ INFO][PID:2611018] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971985-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached [2025-04-28 06:27:44,436][ INFO][PID:2611018] Downloading the builder-live.log file, attempt 1 [2025-04-28 06:27:44,438][ INFO][PID:2611018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.104 copr-rpmbuild-log [2025-04-28 06:27:49,443][ INFO][PID:2611018] Periodic builder liveness probe: alive [2025-04-28 06:37:49,597][ INFO][PID:2611018] Periodic builder liveness probe: dead [2025-04-28 06:38:49,737][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: Connection timed out during banner exchange Connection to 140.211.168.104 port 22 timed out [2025-04-28 06:39:59,882][ ERROR][PID:2611018] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke: OUT: ERR: Connection timed out during banner exchange Connection to 140.211.168.104 port 22 timed out [2025-04-28 06:41:09,953][ ERROR][PID:2611018] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.104 port 22: Connection timed out [2025-04-28 06:41:54,139][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 06:42:04,141][WARNING][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 06:42:14,142][ INFO][PID:2611018] Downloading the builder-live.log file, attempt 2 [2025-04-28 06:42:14,144][ INFO][PID:2611018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.104 copr-rpmbuild-log [2025-04-28 06:42:20,023][ ERROR][PID:2611018] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.104 port 22: Connection timed out [2025-04-28 06:42:30,025][ ERROR][PID:2611018] Can't ssh to cancel build. [2025-04-28 06:43:14,211][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 06:43:24,212][WARNING][PID:2611018] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 06:43:34,213][ INFO][PID:2611018] Downloading the builder-live.log file, attempt 3 [2025-04-28 06:43:34,217][ INFO][PID:2611018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.104 copr-rpmbuild-log [2025-04-28 06:44:34,285][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 06:44:44,291][WARNING][PID:2611018] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 06:44:54,292][ INFO][PID:2611018] Downloading the builder-live.log file, attempt 4 [2025-04-28 06:44:54,294][ INFO][PID:2611018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.104 copr-rpmbuild-log [2025-04-28 06:45:54,362][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 06:46:04,364][ ERROR][PID:2611018] Giving up for unstable SSH, failures: 1745822524.1410775, 1745822604.2124646, 1745822684.2911923, 1745822764.364382 [2025-04-28 06:46:04,365][ ERROR][PID:2611018] Re-try request for task on 'ResallocHost, ticket_id=2689272, hostname=140.211.168.104, name=copr_osuosl_p09_prod_03446669_20250428_055723, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED [2025-04-28 06:46:04,366][ INFO][PID:2611018] Releasing VM back to pool [2025-04-28 06:46:04,393][ INFO][PID:2611018] Retry #1 (on other host) [2025-04-28 06:46:04,437][ INFO][PID:2611018] Trying to allocate VM: ResallocHost, ticket_id=2689419, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 07:06:22,687][ INFO][PID:2611018] Allocated host ResallocHost, ticket_id=2689419, hostname=140.211.168.171, name=copr_osuosl_p09_prod_03448210_20250428_070150, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 07:06:22,701][ INFO][PID:2611018] Allocating ssh connection to builder [2025-04-28 07:06:22,702][ INFO][PID:2611018] Checking that builder machine is OK [2025-04-28 07:06:23,965][ INFO][PID:2611018] Running remote command: copr-builder-ready fedora-40-ppc64le [2025-04-28 07:06:24,915][ INFO][PID:2611018] Red Hat subscription not needed for fedora-40-ppc64le Builder is ready to be used [2025-04-28 07:06:24,916][ INFO][PID:2611018] Filling build.info file with builder info [2025-04-28 07:06:24,922][ INFO][PID:2611018] Not re-notifying FE and msg buses for the new host. [2025-04-28 07:06:24,923][ INFO][PID:2611018] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971985-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached [2025-04-28 07:06:26,231][ INFO][PID:2611018] Downloading the builder-live.log file, attempt 1 [2025-04-28 07:06:26,232][ INFO][PID:2611018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.171 copr-rpmbuild-log [2025-04-28 07:06:31,242][ INFO][PID:2611018] Periodic builder liveness probe: alive [2025-04-28 07:16:31,555][ INFO][PID:2611018] Periodic builder liveness probe: alive [2025-04-28 07:26:31,707][ INFO][PID:2611018] Periodic builder liveness probe: dead [2025-04-28 07:27:31,780][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.171 port 22: Connection timed out [2025-04-28 07:27:42,661][ ERROR][PID:2611018] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '140.211.168.171' (ED25519) to the list of known hosts. Permission denied, please try again. Permission denied, please try again. mockbuilder@140.211.168.171: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2025-04-28 07:27:52,778][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 07:27:53,524][ ERROR][PID:2611018] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '140.211.168.171' (ED25519) to the list of known hosts. Permission denied, please try again. Permission denied, please try again. mockbuilder@140.211.168.171: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2025-04-28 07:28:02,779][WARNING][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 07:28:04,378][ ERROR][PID:2611018] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '140.211.168.171' (ED25519) to the list of known hosts. Permission denied, please try again. Permission denied, please try again. mockbuilder@140.211.168.171: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2025-04-28 07:28:12,781][ INFO][PID:2611018] Downloading the builder-live.log file, attempt 2 [2025-04-28 07:28:12,783][ INFO][PID:2611018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.171 copr-rpmbuild-log [2025-04-28 07:28:13,681][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 07:28:14,379][ ERROR][PID:2611018] Can't ssh to cancel build. [2025-04-28 07:28:23,686][WARNING][PID:2611018] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 07:28:33,688][ INFO][PID:2611018] Downloading the builder-live.log file, attempt 3 [2025-04-28 07:28:33,689][ INFO][PID:2611018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.171 copr-rpmbuild-log [2025-04-28 07:28:34,571][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 07:28:44,572][WARNING][PID:2611018] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 07:28:54,573][ INFO][PID:2611018] Downloading the builder-live.log file, attempt 4 [2025-04-28 07:28:54,575][ INFO][PID:2611018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.171 copr-rpmbuild-log [2025-04-28 07:28:55,471][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 07:29:05,473][ ERROR][PID:2611018] Giving up for unstable SSH, failures: 1745825282.779766, 1745825303.6867175, 1745825324.572654, 1745825345.47263 [2025-04-28 07:29:05,478][ ERROR][PID:2611018] Re-try request for task on 'ResallocHost, ticket_id=2689419, hostname=140.211.168.171, name=copr_osuosl_p09_prod_03448210_20250428_070150, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED [2025-04-28 07:29:05,481][ INFO][PID:2611018] Releasing VM back to pool [2025-04-28 07:29:05,523][ INFO][PID:2611018] Retry #2 (on other host) [2025-04-28 07:29:05,577][ INFO][PID:2611018] Trying to allocate VM: ResallocHost, ticket_id=2689806, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 08:02:24,061][ INFO][PID:2611018] Allocated host ResallocHost, ticket_id=2689806, hostname=140.211.168.199, name=copr_osuosl_p09_prod_03449801_20250428_075800, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 08:02:24,063][ INFO][PID:2611018] Allocating ssh connection to builder [2025-04-28 08:02:24,065][ INFO][PID:2611018] Checking that builder machine is OK [2025-04-28 08:02:25,325][ INFO][PID:2611018] Running remote command: copr-builder-ready fedora-40-ppc64le [2025-04-28 08:02:26,371][ INFO][PID:2611018] Red Hat subscription not needed for fedora-40-ppc64le Builder is ready to be used [2025-04-28 08:02:26,374][ INFO][PID:2611018] Filling build.info file with builder info [2025-04-28 08:02:26,391][ INFO][PID:2611018] Not re-notifying FE and msg buses for the new host. [2025-04-28 08:02:26,394][ INFO][PID:2611018] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971985-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached [2025-04-28 08:02:27,735][ INFO][PID:2611018] Downloading the builder-live.log file, attempt 1 [2025-04-28 08:02:27,739][ INFO][PID:2611018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.199 copr-rpmbuild-log [2025-04-28 08:02:32,771][ INFO][PID:2611018] Periodic builder liveness probe: alive [2025-04-28 08:12:33,159][ INFO][PID:2611018] Periodic builder liveness probe: alive [2025-04-28 08:22:33,332][ INFO][PID:2611018] Periodic builder liveness probe: alive [2025-04-28 08:32:33,528][ INFO][PID:2611018] Periodic builder liveness probe: dead [2025-04-28 08:33:12,139][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 08:33:22,140][WARNING][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 08:33:32,141][ INFO][PID:2611018] Downloading the builder-live.log file, attempt 2 [2025-04-28 08:33:32,143][ INFO][PID:2611018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.199 copr-rpmbuild-log [2025-04-28 08:33:32,220][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 08:33:33,597][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.199 port 22: Connection timed out [2025-04-28 08:33:42,225][WARNING][PID:2611018] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 08:33:44,490][ ERROR][PID:2611018] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '140.211.168.199' (ED25519) to the list of known hosts. Permission denied, please try again. Permission denied, please try again. mockbuilder@140.211.168.199: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2025-04-28 08:33:52,226][ INFO][PID:2611018] Downloading the builder-live.log file, attempt 3 [2025-04-28 08:33:52,230][ INFO][PID:2611018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.199 copr-rpmbuild-log [2025-04-28 08:33:53,137][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 08:33:55,356][ ERROR][PID:2611018] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '140.211.168.199' (ED25519) to the list of known hosts. Permission denied, please try again. Permission denied, please try again. mockbuilder@140.211.168.199: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2025-04-28 08:34:03,138][WARNING][PID:2611018] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 08:34:06,278][ ERROR][PID:2611018] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '140.211.168.199' (ED25519) to the list of known hosts. Permission denied, please try again. Permission denied, please try again. mockbuilder@140.211.168.199: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2025-04-28 08:34:13,140][ INFO][PID:2611018] Downloading the builder-live.log file, attempt 4 [2025-04-28 08:34:13,142][ INFO][PID:2611018] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.199 copr-rpmbuild-log [2025-04-28 08:34:14,015][ ERROR][PID:2611018] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 08:34:16,279][ ERROR][PID:2611018] Can't ssh to cancel build. [2025-04-28 08:34:24,017][ ERROR][PID:2611018] Giving up for unstable SSH, failures: 1745829202.140607, 1745829222.2251885, 1745829243.13878, 1745829264.016968 [2025-04-28 08:34:24,018][ ERROR][PID:2611018] Re-try request for task on 'ResallocHost, ticket_id=2689806, hostname=140.211.168.199, name=copr_osuosl_p09_prod_03449801_20250428_075800, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED [2025-04-28 08:34:24,019][ INFO][PID:2611018] Releasing VM back to pool [2025-04-28 08:34:24,034][ ERROR][PID:2611018] Backend process error: Three host tried without success: {'140.211.168.199', '140.211.168.171', '140.211.168.104'} [2025-04-28 08:34:24,035][WARNING][PID:2611018] Switching not-finished job state to 'failed' [2025-04-28 08:34:24,036][ INFO][PID:2611018] Worker failed build, took 1917.644152879715 [2025-04-28 08:34:24,037][ INFO][PID:2611018] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "osloup", "project_name": "postgresql_update_15_2", "project_dirname": "postgresql_update_15_2", "submitter": "https://src.fedoraproject.org/user/jgrulich", "ended_on": 1745829264.0354977, "started_on": 1745827346.3913448, "submitted_on": null, "status": 0, "chroot": "fedora-40-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "8971985-fedora-40-ppc64le", "build_id": 8971985, "package_name": "qt6-qtbase", "package_version": "6.9.0-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osloup/postgresql_update_15_2/qt6-qtbase", "git_hash": "7b86756d2e09f075c999c756b51d512a97f82859", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "osloup/postgresql_update_15_2--https://src.fedoraproject.org/user/jgrulich", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/osloup/postgresql_update_15_2", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_update_15_2", "result_dir": "08971985-qt6-qtbase", "built_packages": "", "id": 8971985, "mockchain_macros": { "copr_username": "osloup", "copr_projectname": "postgresql_update_15_2", "vendor": "Fedora Project COPR (osloup/postgresql_update_15_2)" } } ] } [2025-04-28 08:34:24,072][ INFO][PID:2611018] Sending fedora-messaging bus message in build.end [2025-04-28 08:34:24,093][ INFO][PID:2611018] Compressing /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/08971985-qt6-qtbase/builder-live.log by gzip [2025-04-28 08:34:24,095][ INFO][PID:2611018] Running command 'gzip /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/08971985-qt6-qtbase/builder-live.log' as PID 3454401 [2025-04-28 08:34:24,096][ INFO][PID:2611018] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/08971985-qt6-qtbase/builder-live.log) [2025-04-28 08:34:24,098][ INFO][PID:2611018] Compressing /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/08971985-qt6-qtbase/backend.log by gzip [2025-04-28 08:34:24,100][ INFO][PID:2611018] Running command 'gzip /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/08971985-qt6-qtbase/backend.log' as PID 3454402