[2025-04-28 06:28:40,246][ INFO][PID:2618168] Marking build as starting [2025-04-28 06:28:40,371][ INFO][PID:2618168] Trying to allocate VM: ResallocHost, ticket_id=2689311, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 06:44:58,629][ INFO][PID:2618168] Allocated host ResallocHost, ticket_id=2689311, hostname=140.211.168.61, name=copr_osuosl_p09_prod_03447690_20250428_064007, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 06:44:58,629][ INFO][PID:2618168] Allocating ssh connection to builder [2025-04-28 06:44:58,630][ INFO][PID:2618168] Checking that builder machine is OK [2025-04-28 06:44:59,894][ INFO][PID:2618168] Running remote command: copr-builder-ready fedora-40-ppc64le [2025-04-28 06:45:00,881][ INFO][PID:2618168] Red Hat subscription not needed for fedora-40-ppc64le Builder is ready to be used [2025-04-28 06:45:00,883][ INFO][PID:2618168] Filling build.info file with builder info [2025-04-28 06:45:00,886][ INFO][PID:2618168] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "osloup", "project_name": "postgresql_update_15_2", "project_dirname": "postgresql_update_15_2", "submitter": "https://src.fedoraproject.org/user/jgrulich", "ended_on": null, "started_on": 1745822700.885749, "submitted_on": null, "status": 3, "chroot": "fedora-40-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "8971989-fedora-40-ppc64le", "build_id": 8971989, "package_name": "qt6-qtbase", "package_version": "6.9.0-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osloup/postgresql_update_15_2/qt6-qtbase", "git_hash": "7b86756d2e09f075c999c756b51d512a97f82859", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "osloup/postgresql_update_15_2--https://src.fedoraproject.org/user/jgrulich", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/osloup/postgresql_update_15_2", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_update_15_2", "result_dir": "08971989-qt6-qtbase", "built_packages": "", "id": 8971989, "mockchain_macros": { "copr_username": "osloup", "copr_projectname": "postgresql_update_15_2", "vendor": "Fedora Project COPR (osloup/postgresql_update_15_2)" } } ] } [2025-04-28 06:45:00,936][ INFO][PID:2618168] Sending fedora-messaging bus message in build.start [2025-04-28 06:45:01,559][ INFO][PID:2618168] Sending fedora-messaging bus message in chroot.start [2025-04-28 06:45:01,576][ INFO][PID:2618168] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971989-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached [2025-04-28 06:45:02,949][ INFO][PID:2618168] Downloading the builder-live.log file, attempt 1 [2025-04-28 06:45:02,952][ INFO][PID:2618168] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.61 copr-rpmbuild-log [2025-04-28 06:45:07,956][ INFO][PID:2618168] Periodic builder liveness probe: alive [2025-04-28 06:55:08,112][ INFO][PID:2618168] Periodic builder liveness probe: alive [2025-04-28 07:05:08,254][ INFO][PID:2618168] Periodic builder liveness probe: dead [2025-04-28 07:05:25,138][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 07:05:35,140][WARNING][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 07:05:45,146][ INFO][PID:2618168] Downloading the builder-live.log file, attempt 2 [2025-04-28 07:05:45,148][ INFO][PID:2618168] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.61 copr-rpmbuild-log [2025-04-28 07:06:08,329][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.61 port 22: Connection timed out [2025-04-28 07:06:45,217][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 07:06:55,228][WARNING][PID:2618168] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 07:07:05,229][ INFO][PID:2618168] Downloading the builder-live.log file, attempt 3 [2025-04-28 07:07:05,233][ INFO][PID:2618168] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.61 copr-rpmbuild-log [2025-04-28 07:07:18,408][ ERROR][PID:2618168] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.61 port 22: Connection timed out [2025-04-28 07:07:41,392][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 07:07:49,076][ ERROR][PID:2618168] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '140.211.168.61' (ED25519) to the list of known hosts. Permission denied, please try again. Permission denied, please try again. mockbuilder@140.211.168.61: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2025-04-28 07:07:51,405][WARNING][PID:2618168] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 07:08:00,110][ ERROR][PID:2618168] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '140.211.168.61' (ED25519) to the list of known hosts. Permission denied, please try again. Permission denied, please try again. mockbuilder@140.211.168.61: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2025-04-28 07:08:01,407][ INFO][PID:2618168] Downloading the builder-live.log file, attempt 4 [2025-04-28 07:08:01,421][ INFO][PID:2618168] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.61 copr-rpmbuild-log [2025-04-28 07:08:02,424][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 07:08:10,116][ ERROR][PID:2618168] Can't ssh to cancel build. [2025-04-28 07:08:12,429][ ERROR][PID:2618168] Giving up for unstable SSH, failures: 1745823935.1402855, 1745824015.2280793, 1745824071.4058335, 1745824092.429384 [2025-04-28 07:08:12,434][ ERROR][PID:2618168] Re-try request for task on 'ResallocHost, ticket_id=2689311, hostname=140.211.168.61, name=copr_osuosl_p09_prod_03447690_20250428_064007, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED [2025-04-28 07:08:12,439][ INFO][PID:2618168] Releasing VM back to pool [2025-04-28 07:08:12,515][ INFO][PID:2618168] Retry #1 (on other host) [2025-04-28 07:08:12,586][ INFO][PID:2618168] Trying to allocate VM: ResallocHost, ticket_id=2689635, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 07:45:31,325][ INFO][PID:2618168] Allocated host ResallocHost, ticket_id=2689635, hostname=140.211.168.83, name=copr_osuosl_p09_prod_03449209_20250428_074038, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 07:45:31,326][ INFO][PID:2618168] Allocating ssh connection to builder [2025-04-28 07:45:31,327][ INFO][PID:2618168] Checking that builder machine is OK [2025-04-28 07:45:32,563][ INFO][PID:2618168] Running remote command: copr-builder-ready fedora-40-ppc64le [2025-04-28 07:45:33,554][ INFO][PID:2618168] Red Hat subscription not needed for fedora-40-ppc64le Builder is ready to be used [2025-04-28 07:45:33,555][ INFO][PID:2618168] Filling build.info file with builder info [2025-04-28 07:45:33,557][ INFO][PID:2618168] Not re-notifying FE and msg buses for the new host. [2025-04-28 07:45:33,558][ INFO][PID:2618168] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971989-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached [2025-04-28 07:45:34,885][ INFO][PID:2618168] Downloading the builder-live.log file, attempt 1 [2025-04-28 07:45:34,887][ INFO][PID:2618168] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.83 copr-rpmbuild-log [2025-04-28 07:45:39,896][ INFO][PID:2618168] Periodic builder liveness probe: alive [2025-04-28 07:55:40,099][ INFO][PID:2618168] Periodic builder liveness probe: alive [2025-04-28 08:05:40,362][ INFO][PID:2618168] Periodic builder liveness probe: alive [2025-04-28 08:15:40,709][ INFO][PID:2618168] Periodic builder liveness probe: dead [2025-04-28 08:16:40,850][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: Connection timed out during banner exchange Connection to 140.211.168.83 port 22 timed out [2025-04-28 08:17:50,989][ ERROR][PID:2618168] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke: OUT: ERR: Connection timed out during banner exchange Connection to 140.211.168.83 port 22 timed out [2025-04-28 08:18:52,139][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 08:19:01,058][ ERROR][PID:2618168] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.83 port 22: Connection timed out [2025-04-28 08:19:02,140][WARNING][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 08:19:12,141][ INFO][PID:2618168] Downloading the builder-live.log file, attempt 2 [2025-04-28 08:19:12,144][ INFO][PID:2618168] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.83 copr-rpmbuild-log [2025-04-28 08:20:11,132][ ERROR][PID:2618168] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.83 port 22: Connection timed out [2025-04-28 08:20:12,212][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 08:20:21,133][ ERROR][PID:2618168] Can't ssh to cancel build. [2025-04-28 08:20:22,220][WARNING][PID:2618168] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 08:20:32,221][ INFO][PID:2618168] Downloading the builder-live.log file, attempt 3 [2025-04-28 08:20:32,222][ INFO][PID:2618168] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.83 copr-rpmbuild-log [2025-04-28 08:21:32,286][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 08:21:42,288][WARNING][PID:2618168] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 08:21:52,289][ INFO][PID:2618168] Downloading the builder-live.log file, attempt 4 [2025-04-28 08:21:52,291][ INFO][PID:2618168] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.83 copr-rpmbuild-log [2025-04-28 08:22:52,360][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 08:23:02,365][ ERROR][PID:2618168] Giving up for unstable SSH, failures: 1745828342.1403654, 1745828422.2199655, 1745828502.2886755, 1745828582.365309 [2025-04-28 08:23:02,366][ ERROR][PID:2618168] Re-try request for task on 'ResallocHost, ticket_id=2689635, hostname=140.211.168.83, name=copr_osuosl_p09_prod_03449209_20250428_074038, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED [2025-04-28 08:23:02,367][ INFO][PID:2618168] Releasing VM back to pool [2025-04-28 08:23:02,399][ INFO][PID:2618168] Retry #2 (on other host) [2025-04-28 08:23:02,414][ INFO][PID:2618168] Trying to allocate VM: ResallocHost, ticket_id=2690621, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 09:48:22,639][ INFO][PID:2618168] Allocated host ResallocHost, ticket_id=2690621, hostname=140.211.168.5, name=copr_osuosl_p09_prod_03454002_20250428_094335, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 09:48:22,652][ INFO][PID:2618168] Allocating ssh connection to builder [2025-04-28 09:48:22,658][ INFO][PID:2618168] Checking that builder machine is OK [2025-04-28 09:48:23,996][ INFO][PID:2618168] Running remote command: copr-builder-ready fedora-40-ppc64le [2025-04-28 09:48:25,022][ INFO][PID:2618168] Red Hat subscription not needed for fedora-40-ppc64le Builder is ready to be used [2025-04-28 09:48:25,024][ INFO][PID:2618168] Filling build.info file with builder info [2025-04-28 09:48:25,065][ INFO][PID:2618168] Not re-notifying FE and msg buses for the new host. [2025-04-28 09:48:25,067][ INFO][PID:2618168] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971989-fedora-40-ppc64le --chroot fedora-40-ppc64le --detached [2025-04-28 09:48:26,449][ INFO][PID:2618168] Downloading the builder-live.log file, attempt 1 [2025-04-28 09:48:26,452][ INFO][PID:2618168] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.5 copr-rpmbuild-log [2025-04-28 09:48:31,469][ INFO][PID:2618168] Periodic builder liveness probe: alive [2025-04-28 09:58:31,838][ INFO][PID:2618168] Periodic builder liveness probe: alive [2025-04-28 10:08:32,156][ INFO][PID:2618168] Periodic builder liveness probe: alive [2025-04-28 10:18:32,678][ INFO][PID:2618168] Periodic builder liveness probe: alive [2025-04-28 10:23:56,141][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 10:24:06,152][WARNING][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 10:24:16,154][ INFO][PID:2618168] Downloading the builder-live.log file, attempt 2 [2025-04-28 10:24:16,158][ INFO][PID:2618168] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.5 copr-rpmbuild-log [2025-04-28 10:25:16,239][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 10:25:26,241][WARNING][PID:2618168] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 10:25:36,247][ INFO][PID:2618168] Downloading the builder-live.log file, attempt 3 [2025-04-28 10:25:36,256][ INFO][PID:2618168] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.5 copr-rpmbuild-log [2025-04-28 10:26:36,346][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 10:26:46,361][WARNING][PID:2618168] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 10:26:56,367][ INFO][PID:2618168] Downloading the builder-live.log file, attempt 4 [2025-04-28 10:26:56,374][ INFO][PID:2618168] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.5 copr-rpmbuild-log [2025-04-28 10:27:56,476][ ERROR][PID:2618168] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 10:28:06,481][ ERROR][PID:2618168] Giving up for unstable SSH, failures: 1745835846.1525893, 1745835926.241206, 1745836006.3612385, 1745836086.481222 [2025-04-28 10:28:06,486][ ERROR][PID:2618168] Re-try request for task on 'ResallocHost, ticket_id=2690621, hostname=140.211.168.5, name=copr_osuosl_p09_prod_03454002_20250428_094335, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-04-28 10:28:06,490][ INFO][PID:2618168] Releasing VM back to pool [2025-04-28 10:28:06,571][ ERROR][PID:2618168] Backend process error: Three host tried without success: {'140.211.168.83', '140.211.168.5', '140.211.168.61'} [2025-04-28 10:28:06,576][WARNING][PID:2618168] Switching not-finished job state to 'failed' [2025-04-28 10:28:06,580][ INFO][PID:2618168] Worker failed build, took 2381.5111401081085 [2025-04-28 10:28:06,583][ INFO][PID:2618168] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "osloup", "project_name": "postgresql_update_15_2", "project_dirname": "postgresql_update_15_2", "submitter": "https://src.fedoraproject.org/user/jgrulich", "ended_on": 1745836086.5764275, "started_on": 1745833705.0652874, "submitted_on": null, "status": 0, "chroot": "fedora-40-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "8971989-fedora-40-ppc64le", "build_id": 8971989, "package_name": "qt6-qtbase", "package_version": "6.9.0-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osloup/postgresql_update_15_2/qt6-qtbase", "git_hash": "7b86756d2e09f075c999c756b51d512a97f82859", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "osloup/postgresql_update_15_2--https://src.fedoraproject.org/user/jgrulich", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/osloup/postgresql_update_15_2", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_update_15_2", "result_dir": "08971989-qt6-qtbase", "built_packages": "", "id": 8971989, "mockchain_macros": { "copr_username": "osloup", "copr_projectname": "postgresql_update_15_2", "vendor": "Fedora Project COPR (osloup/postgresql_update_15_2)" } } ] } [2025-04-28 10:28:06,662][ INFO][PID:2618168] Sending fedora-messaging bus message in build.end [2025-04-28 10:28:06,715][ INFO][PID:2618168] Compressing /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/08971989-qt6-qtbase/builder-live.log by gzip [2025-04-28 10:28:06,722][ INFO][PID:2618168] Running command 'gzip /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/08971989-qt6-qtbase/builder-live.log' as PID 389882 [2025-04-28 10:28:06,731][ INFO][PID:2618168] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/08971989-qt6-qtbase/builder-live.log) [2025-04-28 10:28:06,739][ INFO][PID:2618168] Compressing /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/08971989-qt6-qtbase/backend.log by gzip [2025-04-28 10:28:06,741][ INFO][PID:2618168] Running command 'gzip /var/lib/copr/public_html/results/osloup/postgresql_update_15_2/fedora-40-ppc64le/08971989-qt6-qtbase/backend.log' as PID 389887