[2025-04-28 06:27:36,759][ INFO][PID:2611073] Marking build as starting [2025-04-28 06:27:36,977][ INFO][PID:2611073] Trying to allocate VM: ResallocHost, ticket_id=2689277, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 06:27:40,005][ INFO][PID:2611073] Allocated host ResallocHost, ticket_id=2689277, hostname=140.211.168.240, name=copr_osuosl_p09_prod_03446740_20250428_060047, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 06:27:40,008][ INFO][PID:2611073] Allocating ssh connection to builder [2025-04-28 06:27:40,011][ INFO][PID:2611073] Checking that builder machine is OK [2025-04-28 06:27:41,330][ INFO][PID:2611073] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-04-28 06:27:42,503][ INFO][PID:2611073] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-04-28 06:27:42,504][ INFO][PID:2611073] Filling build.info file with builder info [2025-04-28 06:27:42,509][ INFO][PID:2611073] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "osloup", "project_name": "postgresql_demodularization", "project_dirname": "postgresql_demodularization", "submitter": "https://src.fedoraproject.org/user/jgrulich", "ended_on": null, "started_on": 1745821662.5095305, "submitted_on": null, "status": 3, "chroot": "fedora-41-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "8971984-fedora-41-ppc64le", "build_id": 8971984, "package_name": "qt6-qtbase", "package_version": "6.9.0-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osloup/postgresql_demodularization/qt6-qtbase", "git_hash": "6b2d2e9bdeb0a68756ae301f49b13e509e1bd682", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "osloup/postgresql_demodularization--https://src.fedoraproject.org/user/jgrulich", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_demodularization/fedora-41-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/osloup/postgresql_demodularization", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_demodularization", "result_dir": "08971984-qt6-qtbase", "built_packages": "", "id": 8971984, "mockchain_macros": { "copr_username": "osloup", "copr_projectname": "postgresql_demodularization", "vendor": "Fedora Project COPR (osloup/postgresql_demodularization)" } } ] } [2025-04-28 06:27:42,571][ INFO][PID:2611073] Sending fedora-messaging bus message in build.start [2025-04-28 06:27:43,313][ INFO][PID:2611073] Sending fedora-messaging bus message in chroot.start [2025-04-28 06:27:43,336][ INFO][PID:2611073] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971984-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-04-28 06:27:44,659][ INFO][PID:2611073] Downloading the builder-live.log file, attempt 1 [2025-04-28 06:27:44,663][ INFO][PID:2611073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.240 copr-rpmbuild-log [2025-04-28 06:27:49,666][ INFO][PID:2611073] Periodic builder liveness probe: alive [2025-04-28 06:37:49,818][ INFO][PID:2611073] Periodic builder liveness probe: alive [2025-04-28 06:43:46,142][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 06:43:56,146][WARNING][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 06:44:06,148][ INFO][PID:2611073] Downloading the builder-live.log file, attempt 2 [2025-04-28 06:44:06,150][ INFO][PID:2611073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.240 copr-rpmbuild-log [2025-04-28 06:45:06,219][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 06:45:16,220][WARNING][PID:2611073] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 06:45:26,222][ INFO][PID:2611073] Downloading the builder-live.log file, attempt 3 [2025-04-28 06:45:26,223][ INFO][PID:2611073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.240 copr-rpmbuild-log [2025-04-28 06:46:26,291][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 06:46:36,292][WARNING][PID:2611073] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 06:46:46,293][ INFO][PID:2611073] Downloading the builder-live.log file, attempt 4 [2025-04-28 06:46:46,295][ INFO][PID:2611073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.240 copr-rpmbuild-log [2025-04-28 06:47:46,353][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 06:47:49,965][ INFO][PID:2611073] Periodic builder liveness probe: dead [2025-04-28 06:47:56,354][ ERROR][PID:2611073] Giving up for unstable SSH, failures: 1745822636.14668, 1745822716.220891, 1745822796.2927735, 1745822876.3545327 [2025-04-28 06:48:50,038][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.240 port 22: Connection timed out [2025-04-28 06:50:00,115][ ERROR][PID:2611073] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.240 port 22: Connection timed out [2025-04-28 06:51:10,199][ ERROR][PID:2611073] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.240 port 22: Connection timed out [2025-04-28 06:52:20,271][ ERROR][PID:2611073] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.240 port 22: Connection timed out [2025-04-28 06:52:30,272][ ERROR][PID:2611073] Can't ssh to cancel build. [2025-04-28 06:52:30,273][ ERROR][PID:2611073] Re-try request for task on 'ResallocHost, ticket_id=2689277, hostname=140.211.168.240, name=copr_osuosl_p09_prod_03446740_20250428_060047, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED [2025-04-28 06:52:30,274][ INFO][PID:2611073] Releasing VM back to pool [2025-04-28 06:52:30,284][ INFO][PID:2611073] Retry #1 (on other host) [2025-04-28 06:52:30,320][ INFO][PID:2611073] Trying to allocate VM: ResallocHost, ticket_id=2689474, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 07:11:48,714][ INFO][PID:2611073] Allocated host ResallocHost, ticket_id=2689474, hostname=140.211.168.29, name=copr_osuosl_p09_prod_03448326_20250428_070606, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 07:11:48,735][ INFO][PID:2611073] Allocating ssh connection to builder [2025-04-28 07:11:48,737][ INFO][PID:2611073] Checking that builder machine is OK [2025-04-28 07:11:50,066][ INFO][PID:2611073] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-04-28 07:11:51,094][ INFO][PID:2611073] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-04-28 07:11:51,096][ INFO][PID:2611073] Filling build.info file with builder info [2025-04-28 07:11:51,098][ INFO][PID:2611073] Not re-notifying FE and msg buses for the new host. [2025-04-28 07:11:51,100][ INFO][PID:2611073] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971984-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-04-28 07:11:52,426][ INFO][PID:2611073] Downloading the builder-live.log file, attempt 1 [2025-04-28 07:11:52,435][ INFO][PID:2611073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.29 copr-rpmbuild-log [2025-04-28 07:11:57,450][ INFO][PID:2611073] Periodic builder liveness probe: alive [2025-04-28 07:21:57,603][ INFO][PID:2611073] Periodic builder liveness probe: alive [2025-04-28 07:31:57,779][ INFO][PID:2611073] Periodic builder liveness probe: dead [2025-04-28 07:31:58,643][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '140.211.168.29' (ED25519) to the list of known hosts. Permission denied, please try again. Permission denied, please try again. mockbuilder@140.211.168.29: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2025-04-28 07:32:01,140][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 07:32:09,532][ ERROR][PID:2611073] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '140.211.168.29' (ED25519) to the list of known hosts. Permission denied, please try again. Permission denied, please try again. mockbuilder@140.211.168.29: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2025-04-28 07:32:11,141][WARNING][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 07:32:20,406][ ERROR][PID:2611073] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '140.211.168.29' (ED25519) to the list of known hosts. Permission denied, please try again. Permission denied, please try again. mockbuilder@140.211.168.29: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2025-04-28 07:32:21,142][ INFO][PID:2611073] Downloading the builder-live.log file, attempt 2 [2025-04-28 07:32:21,144][ INFO][PID:2611073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.29 copr-rpmbuild-log [2025-04-28 07:32:22,021][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 07:32:31,300][ ERROR][PID:2611073] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke: OUT: ERR: Warning: Permanently added '140.211.168.29' (ED25519) to the list of known hosts. Permission denied, please try again. Permission denied, please try again. mockbuilder@140.211.168.29: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [2025-04-28 07:32:32,022][WARNING][PID:2611073] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 07:32:41,301][ ERROR][PID:2611073] Can't ssh to cancel build. [2025-04-28 07:32:42,023][ INFO][PID:2611073] Downloading the builder-live.log file, attempt 3 [2025-04-28 07:32:42,025][ INFO][PID:2611073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.29 copr-rpmbuild-log [2025-04-28 07:32:42,920][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 07:32:52,922][WARNING][PID:2611073] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 07:33:02,924][ INFO][PID:2611073] Downloading the builder-live.log file, attempt 4 [2025-04-28 07:33:02,926][ INFO][PID:2611073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.29 copr-rpmbuild-log [2025-04-28 07:33:03,791][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 07:33:13,792][ ERROR][PID:2611073] Giving up for unstable SSH, failures: 1745825531.1412127, 1745825552.022291, 1745825572.9227598, 1745825593.7925558 [2025-04-28 07:33:13,794][ ERROR][PID:2611073] Re-try request for task on 'ResallocHost, ticket_id=2689474, hostname=140.211.168.29, name=copr_osuosl_p09_prod_03448326_20250428_070606, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED [2025-04-28 07:33:13,795][ INFO][PID:2611073] Releasing VM back to pool [2025-04-28 07:33:13,808][ INFO][PID:2611073] Retry #2 (on other host) [2025-04-28 07:33:13,821][ INFO][PID:2611073] Trying to allocate VM: ResallocHost, ticket_id=2689834, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 08:22:32,551][ INFO][PID:2611073] Allocated host ResallocHost, ticket_id=2689834, hostname=140.211.168.150, name=copr_osuosl_p09_prod_03450639_20250428_081745, requested_tags=['copr_builder', 'arch_ppc64le'] [2025-04-28 08:22:32,552][ INFO][PID:2611073] Allocating ssh connection to builder [2025-04-28 08:22:32,553][ INFO][PID:2611073] Checking that builder machine is OK [2025-04-28 08:22:33,824][ INFO][PID:2611073] Running remote command: copr-builder-ready fedora-41-ppc64le [2025-04-28 08:22:34,794][ INFO][PID:2611073] Red Hat subscription not needed for fedora-41-ppc64le Builder is ready to be used [2025-04-28 08:22:34,795][ INFO][PID:2611073] Filling build.info file with builder info [2025-04-28 08:22:34,801][ INFO][PID:2611073] Not re-notifying FE and msg buses for the new host. [2025-04-28 08:22:34,802][ INFO][PID:2611073] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8971984-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached [2025-04-28 08:22:36,091][ INFO][PID:2611073] Downloading the builder-live.log file, attempt 1 [2025-04-28 08:22:36,092][ INFO][PID:2611073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.150 copr-rpmbuild-log [2025-04-28 08:22:41,100][ INFO][PID:2611073] Periodic builder liveness probe: alive [2025-04-28 08:32:41,329][ INFO][PID:2611073] Periodic builder liveness probe: alive [2025-04-28 08:42:41,456][ INFO][PID:2611073] Periodic builder liveness probe: alive [2025-04-28 08:50:33,139][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 08:50:43,141][WARNING][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 08:50:53,142][ INFO][PID:2611073] Downloading the builder-live.log file, attempt 2 [2025-04-28 08:50:53,144][ INFO][PID:2611073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.150 copr-rpmbuild-log [2025-04-28 08:51:53,213][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 08:52:03,218][WARNING][PID:2611073] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 08:52:13,219][ INFO][PID:2611073] Downloading the builder-live.log file, attempt 3 [2025-04-28 08:52:13,221][ INFO][PID:2611073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.150 copr-rpmbuild-log [2025-04-28 08:52:41,673][ INFO][PID:2611073] Periodic builder liveness probe: dead [2025-04-28 08:53:13,303][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 08:53:23,311][WARNING][PID:2611073] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-04-28 08:53:33,313][ INFO][PID:2611073] Downloading the builder-live.log file, attempt 4 [2025-04-28 08:53:33,316][ INFO][PID:2611073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.150 copr-rpmbuild-log [2025-04-28 08:53:41,755][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.150 port 22: Connection timed out [2025-04-28 08:54:33,386][ ERROR][PID:2611073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-04-28 08:54:43,391][ ERROR][PID:2611073] Giving up for unstable SSH, failures: 1745830243.1410997, 1745830323.2181635, 1745830403.3118572, 1745830483.3914404 [2025-04-28 08:54:51,834][ ERROR][PID:2611073] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.150 port 22: Connection timed out [2025-04-28 08:56:01,971][ ERROR][PID:2611073] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.150 port 22: Connection timed out [2025-04-28 08:57:12,042][ ERROR][PID:2611073] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke: OUT: ERR: ssh: connect to host 140.211.168.150 port 22: Connection timed out [2025-04-28 08:57:22,043][ ERROR][PID:2611073] Can't ssh to cancel build. [2025-04-28 08:57:22,044][ ERROR][PID:2611073] Re-try request for task on 'ResallocHost, ticket_id=2689834, hostname=140.211.168.150, name=copr_osuosl_p09_prod_03450639_20250428_081745, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Resalloc ticket FAILED [2025-04-28 08:57:22,045][ INFO][PID:2611073] Releasing VM back to pool [2025-04-28 08:57:22,058][ ERROR][PID:2611073] Backend process error: Three host tried without success: {'140.211.168.240', '140.211.168.29', '140.211.168.150'} [2025-04-28 08:57:22,059][WARNING][PID:2611073] Switching not-finished job state to 'failed' [2025-04-28 08:57:22,060][ INFO][PID:2611073] Worker failed build, took 2087.258069753647 [2025-04-28 08:57:22,061][ INFO][PID:2611073] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "osloup", "project_name": "postgresql_demodularization", "project_dirname": "postgresql_demodularization", "submitter": "https://src.fedoraproject.org/user/jgrulich", "ended_on": 1745830642.0593162, "started_on": 1745828554.8012464, "submitted_on": null, "status": 0, "chroot": "fedora-41-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "8971984-fedora-41-ppc64le", "build_id": 8971984, "package_name": "qt6-qtbase", "package_version": "6.9.0-2", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/osloup/postgresql_demodularization/qt6-qtbase", "git_hash": "6b2d2e9bdeb0a68756ae301f49b13e509e1bd682", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "osloup/postgresql_demodularization--https://src.fedoraproject.org/user/jgrulich", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_demodularization/fedora-41-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/osloup/postgresql_demodularization", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/osloup/postgresql_demodularization", "result_dir": "08971984-qt6-qtbase", "built_packages": "", "id": 8971984, "mockchain_macros": { "copr_username": "osloup", "copr_projectname": "postgresql_demodularization", "vendor": "Fedora Project COPR (osloup/postgresql_demodularization)" } } ] } [2025-04-28 08:57:22,102][ INFO][PID:2611073] Sending fedora-messaging bus message in build.end [2025-04-28 08:57:22,118][ INFO][PID:2611073] Compressing /var/lib/copr/public_html/results/osloup/postgresql_demodularization/fedora-41-ppc64le/08971984-qt6-qtbase/builder-live.log by gzip [2025-04-28 08:57:22,119][ INFO][PID:2611073] Running command 'gzip /var/lib/copr/public_html/results/osloup/postgresql_demodularization/fedora-41-ppc64le/08971984-qt6-qtbase/builder-live.log' as PID 3613164 [2025-04-28 08:57:22,121][ INFO][PID:2611073] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/osloup/postgresql_demodularization/fedora-41-ppc64le/08971984-qt6-qtbase/builder-live.log) [2025-04-28 08:57:22,122][ INFO][PID:2611073] Compressing /var/lib/copr/public_html/results/osloup/postgresql_demodularization/fedora-41-ppc64le/08971984-qt6-qtbase/backend.log by gzip [2025-04-28 08:57:22,123][ INFO][PID:2611073] Running command 'gzip /var/lib/copr/public_html/results/osloup/postgresql_demodularization/fedora-41-ppc64le/08971984-qt6-qtbase/backend.log' as PID 3613165