[2025-01-16 18:19:49,816][ INFO][PID:584118] Marking build as starting [2025-01-16 18:19:49,898][ INFO][PID:584118] Trying to allocate VM [2025-01-16 18:20:18,112][ INFO][PID:584118] Allocated host ResallocHost, ticket_id=717100, hostname=140.211.168.239, name=copr_osuosl_p09_prod_00601660_20250116_181532 [2025-01-16 18:20:18,115][ INFO][PID:584118] Allocating ssh connection to builder [2025-01-16 18:20:18,118][ INFO][PID:584118] Checking that builder machine is OK [2025-01-16 18:20:19,453][ INFO][PID:584118] Running remote command: copr-builder-ready epel-8-ppc64le [2025-01-16 18:20:20,520][ INFO][PID:584118] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-01-16 18:20:20,521][ INFO][PID:584118] Filling build.info file with builder info [2025-01-16 18:20:20,526][ INFO][PID:584118] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "dchen", "project_name": "el-pkgs", "project_dirname": "el-pkgs", "submitter": "https://src.fedoraproject.org/user/releng", "ended_on": null, "started_on": 1737051620.5260534, "submitted_on": null, "status": 3, "chroot": "epel-8-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "8522438-epel-8-ppc64le", "build_id": 8522438, "package_name": "firebird", "package_version": "4.0.4.3010-6", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/dchen/el-pkgs/firebird", "git_hash": "86cbd42b4e69a267bb4a2b7ed3d88660624e1b70", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "dchen/el-pkgs--https://src.fedoraproject.org/user/releng", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/dchen/el-pkgs/epel-8-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/dchen/el-pkgs", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/dchen/el-pkgs", "result_dir": "08522438-firebird", "built_packages": "", "id": 8522438, "mockchain_macros": { "copr_username": "dchen", "copr_projectname": "el-pkgs", "vendor": "Fedora Project COPR (dchen/el-pkgs)" } } ] } [2025-01-16 18:20:20,602][ INFO][PID:584118] Sending fedora-messaging bus message in build.start [2025-01-16 18:20:22,356][ INFO][PID:584118] Sending fedora-messaging bus message in chroot.start [2025-01-16 18:20:22,385][ INFO][PID:584118] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8522438-epel-8-ppc64le --chroot epel-8-ppc64le --detached [2025-01-16 18:20:23,754][ INFO][PID:584118] Downloading the builder-live.log file, attempt 1 [2025-01-16 18:20:23,760][ INFO][PID:584118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.239 copr-rpmbuild-log [2025-01-16 18:20:28,770][ INFO][PID:584118] Periodic builder liveness probe: alive [2025-01-16 18:30:28,996][ INFO][PID:584118] Periodic builder liveness probe: alive [2025-01-16 18:40:29,195][ INFO][PID:584118] Periodic builder liveness probe: alive [2025-01-16 18:47:58,232][ ERROR][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-16 18:48:08,234][WARNING][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-16 18:48:18,235][ INFO][PID:584118] Downloading the builder-live.log file, attempt 2 [2025-01-16 18:48:18,240][ INFO][PID:584118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.239 copr-rpmbuild-log [2025-01-16 18:49:18,310][ ERROR][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-16 18:49:28,314][WARNING][PID:584118] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-16 18:49:38,319][ INFO][PID:584118] Downloading the builder-live.log file, attempt 3 [2025-01-16 18:49:38,327][ INFO][PID:584118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.239 copr-rpmbuild-log [2025-01-16 18:50:29,380][ INFO][PID:584118] Periodic builder liveness probe: alive [2025-01-16 18:50:38,430][ ERROR][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-16 18:50:48,431][WARNING][PID:584118] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-16 18:50:58,438][ INFO][PID:584118] Downloading the builder-live.log file, attempt 4 [2025-01-16 18:50:58,445][ INFO][PID:584118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.239 copr-rpmbuild-log [2025-01-16 18:51:58,541][ ERROR][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-16 18:52:08,545][ ERROR][PID:584118] Giving up for unstable SSH, failures: 1737053288.234267, 1737053368.3140242, 1737053448.4313126, 1737053528.5448763 [2025-01-16 18:52:08,546][ ERROR][PID:584118] Re-try request for task on 'ResallocHost, ticket_id=717100, hostname=140.211.168.239, name=copr_osuosl_p09_prod_00601660_20250116_181532': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-01-16 18:52:08,546][ INFO][PID:584118] Releasing VM back to pool [2025-01-16 18:52:08,603][ INFO][PID:584118] Retry #1 (on other host) [2025-01-16 18:52:08,604][ INFO][PID:584118] Trying to allocate VM [2025-01-16 18:52:11,668][ INFO][PID:584118] Allocated host ResallocHost, ticket_id=717929, hostname=2620:52:3:1:dead:beef:cafe:c29f, name=copr_p09_01_prod_00602504_20250116_184817 [2025-01-16 18:52:11,668][ INFO][PID:584118] Allocating ssh connection to builder [2025-01-16 18:52:11,669][ INFO][PID:584118] Checking that builder machine is OK [2025-01-16 18:52:12,715][ INFO][PID:584118] Running remote command: copr-builder-ready epel-8-ppc64le [2025-01-16 18:52:13,174][ INFO][PID:584118] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-01-16 18:52:13,174][ INFO][PID:584118] Filling build.info file with builder info [2025-01-16 18:52:13,176][ INFO][PID:584118] Not re-notifying FE and msg buses for the new host. [2025-01-16 18:52:13,176][ INFO][PID:584118] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8522438-epel-8-ppc64le --chroot epel-8-ppc64le --detached [2025-01-16 18:52:14,259][ INFO][PID:584118] Downloading the builder-live.log file, attempt 1 [2025-01-16 18:52:14,261][ INFO][PID:584118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29f copr-rpmbuild-log [2025-01-16 19:00:29,435][ INFO][PID:584118] Periodic builder liveness probe: alive [2025-01-16 19:03:58,230][ ERROR][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-16 19:04:08,234][WARNING][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-16 19:04:18,235][ INFO][PID:584118] Downloading the builder-live.log file, attempt 2 [2025-01-16 19:04:18,250][ INFO][PID:584118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29f copr-rpmbuild-log [2025-01-16 19:05:18,337][ ERROR][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-16 19:05:28,344][WARNING][PID:584118] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-16 19:05:38,347][ INFO][PID:584118] Downloading the builder-live.log file, attempt 3 [2025-01-16 19:05:38,349][ INFO][PID:584118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29f copr-rpmbuild-log [2025-01-16 19:06:38,418][ ERROR][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-16 19:06:48,422][WARNING][PID:584118] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-16 19:06:58,426][ INFO][PID:584118] Downloading the builder-live.log file, attempt 4 [2025-01-16 19:06:58,432][ INFO][PID:584118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c29f copr-rpmbuild-log [2025-01-16 19:07:58,510][ ERROR][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-16 19:08:08,520][ ERROR][PID:584118] Giving up for unstable SSH, failures: 1737054248.2342627, 1737054328.3446696, 1737054408.4220629, 1737054488.5200074 [2025-01-16 19:08:08,523][ ERROR][PID:584118] Re-try request for task on 'ResallocHost, ticket_id=717929, hostname=2620:52:3:1:dead:beef:cafe:c29f, name=copr_p09_01_prod_00602504_20250116_184817': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-01-16 19:08:08,529][ INFO][PID:584118] Releasing VM back to pool [2025-01-16 19:08:08,570][ INFO][PID:584118] Retry #2 (on other host) [2025-01-16 19:08:08,577][ INFO][PID:584118] Trying to allocate VM [2025-01-16 19:11:26,907][ INFO][PID:584118] Allocated host ResallocHost, ticket_id=718476, hostname=140.211.168.14, name=copr_osuosl_p09_prod_00602966_20250116_190651 [2025-01-16 19:11:26,914][ INFO][PID:584118] Allocating ssh connection to builder [2025-01-16 19:11:26,917][ INFO][PID:584118] Checking that builder machine is OK [2025-01-16 19:11:28,362][ INFO][PID:584118] Running remote command: copr-builder-ready epel-8-ppc64le [2025-01-16 19:11:29,517][ INFO][PID:584118] Checking Red Hat subscription (attempt #1) Red Hat subscription active Builder is ready to be used [2025-01-16 19:11:29,518][ INFO][PID:584118] Filling build.info file with builder info [2025-01-16 19:11:29,527][ INFO][PID:584118] Not re-notifying FE and msg buses for the new host. [2025-01-16 19:11:29,530][ INFO][PID:584118] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8522438-epel-8-ppc64le --chroot epel-8-ppc64le --detached [2025-01-16 19:11:31,042][ INFO][PID:584118] Downloading the builder-live.log file, attempt 1 [2025-01-16 19:11:31,057][ INFO][PID:584118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log [2025-01-16 19:11:36,061][ INFO][PID:584118] Periodic builder liveness probe: alive [2025-01-16 19:21:36,376][ INFO][PID:584118] Periodic builder liveness probe: alive [2025-01-16 19:31:36,474][ INFO][PID:584118] Periodic builder liveness probe: alive [2025-01-16 19:38:13,229][ ERROR][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-16 19:38:23,230][WARNING][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-16 19:38:33,231][ INFO][PID:584118] Downloading the builder-live.log file, attempt 2 [2025-01-16 19:38:33,235][ INFO][PID:584118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log [2025-01-16 19:39:33,304][ ERROR][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-16 19:39:43,306][WARNING][PID:584118] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-16 19:39:53,307][ INFO][PID:584118] Downloading the builder-live.log file, attempt 3 [2025-01-16 19:39:53,309][ INFO][PID:584118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log [2025-01-16 19:40:53,377][ ERROR][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-16 19:41:03,381][WARNING][PID:584118] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts [2025-01-16 19:41:13,382][ INFO][PID:584118] Downloading the builder-live.log file, attempt 4 [2025-01-16 19:41:13,383][ INFO][PID:584118] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log [2025-01-16 19:41:36,589][ INFO][PID:584118] Periodic builder liveness probe: alive [2025-01-16 19:42:13,452][ ERROR][PID:584118] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke. [2025-01-16 19:42:23,455][ ERROR][PID:584118] Giving up for unstable SSH, failures: 1737056303.2304058, 1737056383.3069093, 1737056463.3812335, 1737056543.4555805 [2025-01-16 19:42:23,456][ ERROR][PID:584118] Re-try request for task on 'ResallocHost, ticket_id=718476, hostname=140.211.168.14, name=copr_osuosl_p09_prod_00602966_20250116_190651': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts [2025-01-16 19:42:23,457][ INFO][PID:584118] Releasing VM back to pool [2025-01-16 19:42:23,466][ ERROR][PID:584118] Backend process error: Three host tried without success: {'140.211.168.14', '140.211.168.239', '2620:52:3:1:dead:beef:cafe:c29f'} [2025-01-16 19:42:23,466][WARNING][PID:584118] Switching not-finished job state to 'failed' [2025-01-16 19:42:23,467][ INFO][PID:584118] Worker failed build, took 1853.939121723175 [2025-01-16 19:42:23,467][ INFO][PID:584118] Sending build state back to frontend: { "builds": [ { "timeout": 18000, "frontend_base_url": "https://copr.fedorainfracloud.org", "memory_reqs": 2048, "enable_net": false, "project_owner": "dchen", "project_name": "el-pkgs", "project_dirname": "el-pkgs", "submitter": "https://src.fedoraproject.org/user/releng", "ended_on": 1737056543.4666817, "started_on": 1737054689.52756, "submitted_on": null, "status": 0, "chroot": "epel-8-ppc64le", "arch": "ppc64le", "buildroot_pkgs": [], "task_id": "8522438-epel-8-ppc64le", "build_id": 8522438, "package_name": "firebird", "package_version": "4.0.4.3010-6", "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/dchen/el-pkgs/firebird", "git_hash": "86cbd42b4e69a267bb4a2b7ed3d88660624e1b70", "git_branch": null, "source_type": null, "source_json": null, "pkg_name": null, "pkg_main_version": null, "pkg_epoch": null, "pkg_release": null, "srpm_url": null, "uses_devel_repo": false, "sandbox": "dchen/el-pkgs--https://src.fedoraproject.org/user/releng", "results": null, "appstream": false, "allow_user_ssh": false, "ssh_public_keys": null, "storage": null, "background": true, "bootstrap": "off", "fedora_review": false, "isolation": "default", "repo_priority": null, "repos": [ { "baseurl": "https://download.copr.fedorainfracloud.org/results/dchen/el-pkgs/epel-8-ppc64le/", "id": "copr_base", "name": "Copr repository", "priority": null } ], "tags": [ "arch_ppc64le" ], "with_opts": [], "without_opts": [], "destdir": "/var/lib/copr/public_html/results/dchen/el-pkgs", "results_repo_url": "https://download.copr.fedorainfracloud.org/results/dchen/el-pkgs", "result_dir": "08522438-firebird", "built_packages": "", "id": 8522438, "mockchain_macros": { "copr_username": "dchen", "copr_projectname": "el-pkgs", "vendor": "Fedora Project COPR (dchen/el-pkgs)" } } ] } [2025-01-16 19:42:23,508][ INFO][PID:584118] Sending fedora-messaging bus message in build.end [2025-01-16 19:42:23,527][ INFO][PID:584118] Compressing /var/lib/copr/public_html/results/dchen/el-pkgs/epel-8-ppc64le/08522438-firebird/builder-live.log by gzip [2025-01-16 19:42:23,528][ INFO][PID:584118] Running command 'gzip /var/lib/copr/public_html/results/dchen/el-pkgs/epel-8-ppc64le/08522438-firebird/builder-live.log' as PID 1341041 [2025-01-16 19:42:23,531][ INFO][PID:584118] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/dchen/el-pkgs/epel-8-ppc64le/08522438-firebird/builder-live.log) [2025-01-16 19:42:23,532][ INFO][PID:584118] Compressing /var/lib/copr/public_html/results/dchen/el-pkgs/epel-8-ppc64le/08522438-firebird/backend.log by gzip [2025-01-16 19:42:23,533][ INFO][PID:584118] Running command 'gzip /var/lib/copr/public_html/results/dchen/el-pkgs/epel-8-ppc64le/08522438-firebird/backend.log' as PID 1341044