[2026-01-08 21:25:38,701][  INFO][PID:1935386] Marking build as starting
[2026-01-08 21:25:38,850][  INFO][PID:1935386] Checking that /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2026-01-08 21:25:38,851][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:25:38,852][  INFO][PID:1935386] VM allocation process starts
[2026-01-08 21:25:38,863][  INFO][PID:1935386] Trying to allocate VM: ResallocHost, ticket_id=722169, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-08 21:25:41,883][  INFO][PID:1935386] Allocated host ResallocHost, ticket_id=722169, hostname=2620:52:6:1161:dead:beef:cafe:c18c, name=vmhost_p09_04_prod_01038580_20260108_210901, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-08 21:25:41,883][  INFO][PID:1935386] Allocating ssh connection to builder
[2026-01-08 21:25:41,884][  INFO][PID:1935386] Checking that builder machine is OK
[2026-01-08 21:26:41,954][ ERROR][PID:1935386] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c18c port 22: Connection timed out

[2026-01-08 21:26:51,955][ ERROR][PID:1935386] Re-try request for task on 'ResallocHost, ticket_id=722169, hostname=2620:52:6:1161:dead:beef:cafe:c18c, name=vmhost_p09_04_prod_01038580_20260108_210901, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2026-01-08 21:26:51,956][  INFO][PID:1935386] Releasing VM back to pool
[2026-01-08 21:26:51,967][  INFO][PID:1935386] Retry #1 (on other host)
[2026-01-08 21:26:51,967][  INFO][PID:1935386] Checking that /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2026-01-08 21:26:51,968][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:26:51,970][  INFO][PID:1935386] VM allocation process starts
[2026-01-08 21:26:51,982][  INFO][PID:1935386] Trying to allocate VM: ResallocHost, ticket_id=722197, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-08 21:26:54,995][  INFO][PID:1935386] Allocated host ResallocHost, ticket_id=722197, hostname=2620:52:6:1161:dead:beef:cafe:c150, name=vmhost_p09_03_prod_01038870_20260108_211529, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-08 21:26:54,996][  INFO][PID:1935386] Allocating ssh connection to builder
[2026-01-08 21:26:54,996][  INFO][PID:1935386] Checking that builder machine is OK
[2026-01-08 21:26:56,087][ ERROR][PID:1935386] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c150 port 22: Permission denied

[2026-01-08 21:27:06,088][ ERROR][PID:1935386] Re-try request for task on 'ResallocHost, ticket_id=722197, hostname=2620:52:6:1161:dead:beef:cafe:c150, name=vmhost_p09_03_prod_01038870_20260108_211529, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2026-01-08 21:27:06,089][  INFO][PID:1935386] Releasing VM back to pool
[2026-01-08 21:27:06,099][  INFO][PID:1935386] Retry #2 (on other host)
[2026-01-08 21:27:06,100][  INFO][PID:1935386] Checking that /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/repodata/repomd.xml exists
[2026-01-08 21:27:06,101][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:27:06,101][  INFO][PID:1935386] VM allocation process starts
[2026-01-08 21:27:06,112][  INFO][PID:1935386] Trying to allocate VM: ResallocHost, ticket_id=722199, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-08 21:27:09,123][  INFO][PID:1935386] Allocated host ResallocHost, ticket_id=722199, hostname=2620:52:6:1161:dead:beef:cafe:c14b, name=vmhost_p09_03_prod_01039011_20260108_211957, requested_tags=['copr_builder', 'arch_ppc64le']
[2026-01-08 21:27:09,124][  INFO][PID:1935386] Allocating ssh connection to builder
[2026-01-08 21:27:09,124][  INFO][PID:1935386] Checking that builder machine is OK
[2026-01-08 21:27:10,033][  INFO][PID:1935386] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2026-01-08 21:27:10,490][  INFO][PID:1935386] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2026-01-08 21:27:10,490][  INFO][PID:1935386] Filling build.info file with builder info
[2026-01-08 21:27:10,491][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:27:10,492][  INFO][PID:1935386] Not re-notifying FE and msg buses for the new host.
[2026-01-08 21:27:10,493][  INFO][PID:1935386] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9988207-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2026-01-08 21:27:11,386][  INFO][PID:1935386] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c14b' (ED25519) to the list of known hosts.


[2026-01-08 21:27:11,387][  INFO][PID:1935386] Downloading the builder-live.log file, attempt 1
[2026-01-08 21:27:11,388][  INFO][PID:1935386] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14b copr-rpmbuild-log
[2026-01-08 21:27:16,393][  INFO][PID:1935386] Periodic builder liveness probe: alive
[2026-01-08 21:27:16,394][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:27:21,394][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:27:26,395][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:27:31,396][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:27:36,397][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:27:41,398][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:27:46,399][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:27:51,400][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:27:56,401][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:28:01,402][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:28:06,403][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:28:11,404][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:28:16,405][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:28:21,406][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:28:26,407][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:28:31,408][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:28:36,409][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:28:41,410][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:28:46,411][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:28:51,412][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:28:56,413][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:29:01,414][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:29:06,415][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:29:11,416][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:29:16,417][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:29:21,418][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:29:26,419][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:29:31,420][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:29:36,421][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:29:41,422][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:29:46,423][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:29:51,424][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:29:56,425][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:30:01,426][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:30:06,427][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:30:11,428][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:30:16,429][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:30:21,430][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:30:26,431][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:30:31,432][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:30:36,433][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:30:41,434][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:30:46,435][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:30:51,436][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:30:56,437][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:31:01,438][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:31:06,439][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:31:11,440][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:31:16,442][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:31:21,443][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:31:26,444][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:31:31,445][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:31:36,446][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:31:41,447][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:31:46,448][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:31:51,449][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:31:56,450][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:32:01,451][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:32:06,452][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:32:11,453][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:32:16,454][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:32:21,455][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:32:26,456][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:32:31,457][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:32:36,458][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:32:41,459][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:32:46,460][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:32:51,461][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:32:56,462][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:33:01,463][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:33:06,464][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:33:11,465][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:33:16,466][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:33:21,467][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:33:26,467][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:33:31,468][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:33:36,469][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:33:41,470][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:33:46,471][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:33:51,472][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:33:56,473][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:34:01,474][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:34:06,475][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:34:11,476][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:34:16,477][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:34:21,478][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:34:26,479][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:34:31,480][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:34:36,481][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:34:41,482][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:34:46,483][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:34:51,484][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:34:56,485][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:35:01,486][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:35:06,487][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:35:11,488][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:35:16,489][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:35:21,490][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:35:26,491][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:35:31,492][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:35:36,492][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:35:41,493][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:35:46,494][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:35:51,495][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:35:56,496][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:36:01,497][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:36:06,498][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:36:11,499][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:36:16,500][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:36:21,501][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:36:26,628][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:36:31,629][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:36:36,630][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:36:41,693][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:36:46,694][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:36:51,695][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:36:56,696][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:37:01,697][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:37:06,698][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:37:11,699][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:37:16,839][  INFO][PID:1935386] Periodic builder liveness probe: alive
[2026-01-08 21:37:16,840][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:37:21,840][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:37:26,974][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:37:32,233][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:37:37,233][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:37:42,234][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:37:47,235][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:37:52,291][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:37:57,292][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:38:02,632][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:38:07,633][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:38:12,634][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:38:17,634][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:38:22,635][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:38:27,636][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:38:32,637][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:38:37,638][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:38:42,639][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:38:47,640][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:38:52,780][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:38:57,781][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:39:02,782][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:39:07,873][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:39:12,875][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:39:17,903][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:39:22,954][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:39:28,196][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:39:33,198][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:39:38,199][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:39:43,200][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:39:48,445][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:39:53,447][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:39:58,448][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:40:03,640][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:40:08,641][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:40:13,643][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:40:18,644][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:40:23,723][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:40:28,724][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:40:33,725][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:40:38,726][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:40:43,875][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:40:48,876][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:40:53,877][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:40:58,878][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:41:03,962][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:41:09,021][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:41:14,318][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:41:19,319][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:41:24,320][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:41:29,321][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:41:34,441][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:41:39,442][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:41:44,444][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:41:49,449][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:41:54,450][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:41:59,534][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:42:04,535][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:42:09,748][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:42:14,748][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:42:19,749][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:42:24,750][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:42:29,751][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:42:34,774][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:42:39,821][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:42:44,822][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:42:49,823][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:42:54,824][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:42:59,870][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:43:04,907][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:43:09,948][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:43:15,111][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:43:20,189][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:43:25,198][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:43:30,201][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:43:35,246][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:43:40,280][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:43:45,283][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:43:50,286][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:43:55,288][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:44:00,355][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:44:05,375][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:44:10,397][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:44:15,426][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:44:20,441][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:44:25,468][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:44:30,487][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:44:35,513][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:44:40,527][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:44:45,548][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:44:50,552][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:44:55,566][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:45:00,593][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:45:05,594][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:45:10,620][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:45:15,644][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:45:20,646][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:45:25,688][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:45:30,693][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:45:35,720][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:45:41,071][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:45:46,089][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:45:51,090][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:45:56,091][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:46:01,092][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:46:06,093][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:46:11,094][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:46:16,095][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:46:21,097][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:46:26,098][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:46:31,099][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:46:36,100][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:46:41,266][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:46:46,267][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:46:51,268][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:46:56,269][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:47:01,270][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:47:06,271][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:47:11,272][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:47:16,273][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:47:21,279][  INFO][PID:1935386] Periodic builder liveness probe: alive
[2026-01-08 21:47:21,280][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:47:26,281][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:47:31,282][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:47:36,283][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:47:41,284][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:47:46,285][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:47:51,286][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:47:56,287][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:48:01,287][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:48:06,323][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:48:11,325][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:48:16,326][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:48:21,384][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:48:26,400][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:48:31,529][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:48:36,540][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:48:41,675][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:48:46,676][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:48:51,677][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:48:56,680][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:49:01,681][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:49:06,682][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:49:11,683][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:49:16,684][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:49:21,685][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:49:26,686][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:49:31,687][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:49:36,688][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:49:41,689][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:49:46,690][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:49:51,702][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:49:56,704][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:50:01,734][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:50:06,739][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:50:11,740][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:50:16,755][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:50:21,773][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:50:26,778][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:50:31,806][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:50:36,824][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:50:41,895][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:50:46,921][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:50:52,051][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:50:57,082][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:51:02,083][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:51:07,084][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:51:12,095][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:51:17,505][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:51:22,506][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:51:27,507][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:51:32,508][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:51:37,509][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:51:42,509][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:51:47,510][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:51:52,512][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:51:57,513][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:52:02,513][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:52:07,514][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:52:12,516][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:52:17,516][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:52:22,517][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:52:27,518][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:52:32,520][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:52:37,521][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:52:42,522][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:52:47,523][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:52:52,524][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:52:57,525][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:53:02,526][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:53:07,527][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:53:12,528][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:53:17,529][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:53:22,530][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:53:27,531][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:53:32,559][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:53:37,560][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:53:42,606][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:53:47,608][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:53:52,609][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:53:57,813][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:54:02,814][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:54:07,815][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:54:12,816][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:54:17,817][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:54:19,370][ ERROR][PID:1935386] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-08 21:54:22,818][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:54:27,819][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:54:29,371][WARNING][PID:1935386] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-08 21:54:32,819][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:54:37,905][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:54:39,372][  INFO][PID:1935386] Downloading the builder-live.log file, attempt 2
[2026-01-08 21:54:39,424][  INFO][PID:1935386] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14b copr-rpmbuild-log
[2026-01-08 21:54:42,906][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:54:47,907][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:54:52,908][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:54:57,910][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:02,912][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:07,913][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:12,914][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:17,916][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:22,917][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:28,044][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:33,045][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:38,047][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:39,460][ ERROR][PID:1935386] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-08 21:55:43,048][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:48,150][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:49,471][WARNING][PID:1935386] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-08 21:55:53,152][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:58,154][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:55:59,472][  INFO][PID:1935386] Downloading the builder-live.log file, attempt 3
[2026-01-08 21:55:59,473][  INFO][PID:1935386] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14b copr-rpmbuild-log
[2026-01-08 21:56:03,155][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:56:08,156][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:56:13,157][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:56:18,158][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:56:23,160][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:56:28,162][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:56:33,166][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:56:38,167][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:56:43,169][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:56:48,170][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:56:53,171][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:56:58,172][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:56:59,540][ ERROR][PID:1935386] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-08 21:57:03,402][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:57:08,404][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:57:09,541][WARNING][PID:1935386] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-01-08 21:57:13,406][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:57:18,534][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:57:19,542][  INFO][PID:1935386] Downloading the builder-live.log file, attempt 4
[2026-01-08 21:57:19,543][  INFO][PID:1935386] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c14b copr-rpmbuild-log
[2026-01-08 21:57:23,542][  INFO][PID:1935386] Periodic builder liveness probe: alive
[2026-01-08 21:57:23,618][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:57:28,619][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:57:33,623][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:57:38,624][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:57:43,663][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:57:48,677][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:57:53,703][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:57:58,707][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:58:03,708][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:58:08,709][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:58:13,710][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:58:18,792][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:58:19,611][ ERROR][PID:1935386] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-01-08 21:58:23,794][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:58:28,795][  INFO][PID:1935386] Checking for cancel request
[2026-01-08 21:58:29,612][ ERROR][PID:1935386] Giving up for unstable SSH, failures: 1767909269.371753, 1767909349.4715366, 1767909429.5415761, 1767909509.612397
[2026-01-08 21:58:29,613][ ERROR][PID:1935386] Re-try request for task on 'ResallocHost, ticket_id=722199, hostname=2620:52:6:1161:dead:beef:cafe:c14b, name=vmhost_p09_03_prod_01039011_20260108_211957, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-01-08 21:58:29,614][  INFO][PID:1935386] Releasing VM back to pool
[2026-01-08 21:58:29,622][ ERROR][PID:1935386] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c14b', '2620:52:6:1161:dead:beef:cafe:c150', '2620:52:6:1161:dead:beef:cafe:c18c'}
[2026-01-08 21:58:29,622][WARNING][PID:1935386] Switching not-finished job state to 'failed'
[2026-01-08 21:58:29,623][  INFO][PID:1935386] Worker failed build, took 1879.12983751297
[2026-01-08 21:58:29,623][  INFO][PID:1935386] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "mochaa",
            "project_name": "protobuf",
            "project_dirname": "protobuf",
            "submitter": "https://src.fedoraproject.org/user/kkeithle",
            "ended_on": 1767909509.6226468,
            "started_on": 1767907630.4928093,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9988207-fedora-rawhide-ppc64le",
            "build_id": 9988207,
            "package_name": "libarrow",
            "package_version": "18.0.0-7",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/mochaa/protobuf/libarrow",
            "git_hash": "8c8dd099aaeae4dbf5b60b710b818ef507e2ca34",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "mochaa/protobuf--https://src.fedoraproject.org/user/kkeithle",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/mochaa/protobuf/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/mochaa/protobuf",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/mochaa/protobuf",
            "result_dir": "09988207-libarrow",
            "built_packages": "",
            "id": 9988207,
            "mockchain_macros": {
                "copr_username": "mochaa",
                "copr_projectname": "protobuf",
                "vendor": "Fedora Project COPR (mochaa/protobuf)"
            }
        }
    ]
}
[2026-01-08 21:58:29,682][  INFO][PID:1935386] Sending fedora-messaging bus message in build.end
[2026-01-08 21:58:30,219][  INFO][PID:1935386] Compressing /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/09988207-libarrow/builder-live.log by gzip
[2026-01-08 21:58:30,220][  INFO][PID:1935386] Running command 'gzip /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/09988207-libarrow/builder-live.log' as PID 2044885
[2026-01-08 21:58:30,221][  INFO][PID:1935386] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/09988207-libarrow/builder-live.log)
[2026-01-08 21:58:30,222][  INFO][PID:1935386] Compressing /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/09988207-libarrow/backend.log by gzip
[2026-01-08 21:58:30,223][  INFO][PID:1935386] Running command 'gzip /var/lib/copr/public_html/results/mochaa/protobuf/fedora-rawhide-ppc64le/09988207-libarrow/backend.log' as PID 2044886