[2025-12-22 02:25:55,858][  INFO][PID:494351] Marking build as starting
[2025-12-22 02:25:55,935][  INFO][PID:494351] Checking that /var/lib/copr/public_html/results/amatej/dnf-nightly-sanitizers/fedora-42-ppc64le/repodata/repomd.xml exists
[2025-12-22 02:25:56,037][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:25:56,038][  INFO][PID:494351] VM allocation process starts
[2025-12-22 02:25:56,049][  INFO][PID:494351] Trying to allocate VM: ResallocHost, ticket_id=467675, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-22 02:25:59,062][  INFO][PID:494351] Allocated host ResallocHost, ticket_id=467675, hostname=2620:52:6:1161:dead:beef:cafe:c154, name=vmhost_p09_03_prod_00726547_20251222_021733, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-22 02:25:59,063][  INFO][PID:494351] Allocating ssh connection to builder
[2025-12-22 02:25:59,063][  INFO][PID:494351] Checking that builder machine is OK
[2025-12-22 02:25:59,921][  INFO][PID:494351] Running remote command: copr-builder-ready fedora-42-ppc64le
[2025-12-22 02:26:00,426][  INFO][PID:494351] Red Hat subscription not needed for fedora-42-ppc64le
Builder is ready to be used

[2025-12-22 02:26:00,427][  INFO][PID:494351] Filling build.info file with builder info
[2025-12-22 02:26:00,427][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:26:00,428][  INFO][PID:494351] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "amatej",
            "project_name": "dnf-nightly-sanitizers",
            "project_dirname": "dnf-nightly-sanitizers",
            "submitter": "amatej",
            "ended_on": null,
            "started_on": 1766370360.4285564,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-42-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9940265-fedora-42-ppc64le",
            "build_id": 9940265,
            "package_name": "microdnf",
            "package_version": "3.10.1-1.20251222021150159262.master.3.g0c194e8",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/amatej/dnf-nightly-sanitizers/microdnf",
            "git_hash": "dea5bd3c8bfd90a146f7a5f034b95b633d4e893e",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "amatej/dnf-nightly-sanitizers--amatej",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/amatej/dnf-nightly-sanitizers/fedora-42-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/amatej/dnf-nightly-sanitizers",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/amatej/dnf-nightly-sanitizers",
            "result_dir": "09940265-microdnf",
            "built_packages": "",
            "id": 9940265,
            "mockchain_macros": {
                "copr_username": "amatej",
                "copr_projectname": "dnf-nightly-sanitizers",
                "vendor": "Fedora Project COPR (amatej/dnf-nightly-sanitizers)"
            }
        }
    ]
}
[2025-12-22 02:26:00,509][  INFO][PID:494351] Sending fedora-messaging bus message in build.start
[2025-12-22 02:26:01,036][  INFO][PID:494351] Sending fedora-messaging bus message in chroot.start
[2025-12-22 02:26:01,063][  INFO][PID:494351] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9940265-fedora-42-ppc64le --chroot fedora-42-ppc64le --detached
[2025-12-22 02:26:02,019][  INFO][PID:494351] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '2620:52:6:1161:dead:beef:cafe:c154' (ED25519) to the list of known hosts.


[2025-12-22 02:26:02,020][  INFO][PID:494351] Downloading the builder-live.log file, attempt 1
[2025-12-22 02:26:02,021][  INFO][PID:494351] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c154 copr-rpmbuild-log
[2025-12-22 02:26:07,025][  INFO][PID:494351] Periodic builder liveness probe: alive
[2025-12-22 02:26:07,026][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:26:12,026][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:26:17,027][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:26:22,029][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:26:27,030][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:26:32,031][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:26:37,032][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:26:42,033][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:26:47,034][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:26:52,035][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:26:57,036][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:27:02,037][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:27:07,038][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:27:12,039][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:27:17,040][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:27:22,041][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:27:27,042][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:27:32,043][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:27:37,044][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:27:42,045][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:27:47,046][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:27:52,047][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:27:57,049][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:28:02,050][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:28:07,051][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:28:12,052][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:28:17,053][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:28:22,054][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:28:27,055][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:28:32,056][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:28:32,387][ ERROR][PID:494351] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-22 02:28:37,082][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:28:42,083][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:28:42,388][WARNING][PID:494351] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-22 02:28:47,084][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:28:52,085][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:28:52,389][  INFO][PID:494351] Downloading the builder-live.log file, attempt 2
[2025-12-22 02:28:52,390][  INFO][PID:494351] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c154 copr-rpmbuild-log
[2025-12-22 02:28:57,086][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:29:02,087][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:29:07,088][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:29:12,089][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:29:17,090][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:29:22,091][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:29:27,092][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:29:32,093][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:29:37,094][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:29:42,095][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:29:47,096][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:29:52,097][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:29:52,458][ ERROR][PID:494351] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-22 02:29:57,098][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:30:02,099][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:30:02,521][WARNING][PID:494351] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-22 02:30:07,100][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:30:12,101][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:30:12,521][  INFO][PID:494351] Downloading the builder-live.log file, attempt 3
[2025-12-22 02:30:12,523][  INFO][PID:494351] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c154 copr-rpmbuild-log
[2025-12-22 02:30:17,102][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:30:22,103][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:30:27,104][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:30:32,105][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:30:37,106][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:30:42,107][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:30:47,108][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:30:52,109][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:30:57,110][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:31:02,111][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:31:07,112][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:31:12,113][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:31:12,540][ ERROR][PID:494351] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-22 02:31:17,125][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:31:22,126][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:31:22,589][WARNING][PID:494351] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-12-22 02:31:27,127][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:31:32,128][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:31:32,590][  INFO][PID:494351] Downloading the builder-live.log file, attempt 4
[2025-12-22 02:31:32,591][  INFO][PID:494351] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:6:1161:dead:beef:cafe:c154 copr-rpmbuild-log
[2025-12-22 02:31:37,129][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:31:42,130][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:31:47,131][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:31:52,133][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:31:57,134][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:32:02,135][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:32:07,136][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:32:12,136][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:32:17,137][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:32:22,138][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:32:27,139][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:32:32,141][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:32:32,657][ ERROR][PID:494351] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-12-22 02:32:37,142][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:32:42,142][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:32:42,675][ ERROR][PID:494351] Giving up for unstable SSH, failures: 1766370522.3883512, 1766370602.5210238, 1766370682.589366, 1766370762.6753101
[2025-12-22 02:32:42,676][ ERROR][PID:494351] Re-try request for task on 'ResallocHost, ticket_id=467675, hostname=2620:52:6:1161:dead:beef:cafe:c154, name=vmhost_p09_03_prod_00726547_20251222_021733, requested_tags=['copr_builder', 'arch_ppc64le']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-12-22 02:32:42,676][  INFO][PID:494351] Releasing VM back to pool
[2025-12-22 02:32:42,684][  INFO][PID:494351] Retry #1 (on other host)
[2025-12-22 02:32:42,685][  INFO][PID:494351] Checking that /var/lib/copr/public_html/results/amatej/dnf-nightly-sanitizers/fedora-42-ppc64le/repodata/repomd.xml exists
[2025-12-22 02:32:42,685][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:32:42,686][  INFO][PID:494351] VM allocation process starts
[2025-12-22 02:32:42,727][  INFO][PID:494351] Trying to allocate VM: ResallocHost, ticket_id=467752, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-22 02:32:45,756][  INFO][PID:494351] Allocated host ResallocHost, ticket_id=467752, hostname=2620:52:6:1161:dead:beef:cafe:c188, name=vmhost_p09_04_prod_00726564_20251222_021858, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-22 02:32:45,757][  INFO][PID:494351] Allocating ssh connection to builder
[2025-12-22 02:32:45,757][  INFO][PID:494351] Checking that builder machine is OK
[2025-12-22 02:33:45,825][ ERROR][PID:494351] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c188 port 22: Connection timed out

[2025-12-22 02:33:55,833][ ERROR][PID:494351] Re-try request for task on 'ResallocHost, ticket_id=467752, hostname=2620:52:6:1161:dead:beef:cafe:c188, name=vmhost_p09_04_prod_00726564_20251222_021858, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-22 02:33:55,834][  INFO][PID:494351] Releasing VM back to pool
[2025-12-22 02:33:55,843][  INFO][PID:494351] Retry #2 (on other host)
[2025-12-22 02:33:55,843][  INFO][PID:494351] Checking that /var/lib/copr/public_html/results/amatej/dnf-nightly-sanitizers/fedora-42-ppc64le/repodata/repomd.xml exists
[2025-12-22 02:33:55,844][  INFO][PID:494351] Checking for cancel request
[2025-12-22 02:33:55,844][  INFO][PID:494351] VM allocation process starts
[2025-12-22 02:33:55,910][  INFO][PID:494351] Trying to allocate VM: ResallocHost, ticket_id=467763, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-22 02:33:58,948][  INFO][PID:494351] Allocated host ResallocHost, ticket_id=467763, hostname=2620:52:6:1161:dead:beef:cafe:c150, name=vmhost_p09_03_prod_00726618_20251222_022532, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-12-22 02:33:58,948][  INFO][PID:494351] Allocating ssh connection to builder
[2025-12-22 02:33:58,949][  INFO][PID:494351] Checking that builder machine is OK
[2025-12-22 02:34:59,018][ ERROR][PID:494351] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 2620:52:6:1161:dead:beef:cafe:c150 port 22: Connection timed out

[2025-12-22 02:35:09,040][ ERROR][PID:494351] Re-try request for task on 'ResallocHost, ticket_id=467763, hostname=2620:52:6:1161:dead:beef:cafe:c150, name=vmhost_p09_03_prod_00726618_20251222_022532, requested_tags=['copr_builder', 'arch_ppc64le']': Unable to finish after 1 SSH attempts
[2025-12-22 02:35:09,040][  INFO][PID:494351] Releasing VM back to pool
[2025-12-22 02:35:09,049][ ERROR][PID:494351] Backend process error: Three host tried without success: {'2620:52:6:1161:dead:beef:cafe:c154', '2620:52:6:1161:dead:beef:cafe:c188', '2620:52:6:1161:dead:beef:cafe:c150'}
[2025-12-22 02:35:09,049][WARNING][PID:494351] Switching not-finished job state to 'failed'
[2025-12-22 02:35:09,050][  INFO][PID:494351] Worker failed build, took 548.6211941242218
[2025-12-22 02:35:09,050][  INFO][PID:494351] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "amatej",
            "project_name": "dnf-nightly-sanitizers",
            "project_dirname": "dnf-nightly-sanitizers",
            "submitter": "amatej",
            "ended_on": 1766370909.0497506,
            "started_on": 1766370360.4285564,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-42-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9940265-fedora-42-ppc64le",
            "build_id": 9940265,
            "package_name": "microdnf",
            "package_version": "3.10.1-1.20251222021150159262.master.3.g0c194e8",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/amatej/dnf-nightly-sanitizers/microdnf",
            "git_hash": "dea5bd3c8bfd90a146f7a5f034b95b633d4e893e",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "amatej/dnf-nightly-sanitizers--amatej",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/amatej/dnf-nightly-sanitizers/fedora-42-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/amatej/dnf-nightly-sanitizers",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/amatej/dnf-nightly-sanitizers",
            "result_dir": "09940265-microdnf",
            "built_packages": "",
            "id": 9940265,
            "mockchain_macros": {
                "copr_username": "amatej",
                "copr_projectname": "dnf-nightly-sanitizers",
                "vendor": "Fedora Project COPR (amatej/dnf-nightly-sanitizers)"
            }
        }
    ]
}
[2025-12-22 02:35:09,151][  INFO][PID:494351] Sending fedora-messaging bus message in build.end
[2025-12-22 02:35:09,185][  INFO][PID:494351] Compressing /var/lib/copr/public_html/results/amatej/dnf-nightly-sanitizers/fedora-42-ppc64le/09940265-microdnf/builder-live.log by gzip
[2025-12-22 02:35:09,186][  INFO][PID:494351] Running command 'gzip /var/lib/copr/public_html/results/amatej/dnf-nightly-sanitizers/fedora-42-ppc64le/09940265-microdnf/builder-live.log' as PID 533628
[2025-12-22 02:35:09,307][  INFO][PID:494351] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/amatej/dnf-nightly-sanitizers/fedora-42-ppc64le/09940265-microdnf/builder-live.log)
[2025-12-22 02:35:09,307][  INFO][PID:494351] Compressing /var/lib/copr/public_html/results/amatej/dnf-nightly-sanitizers/fedora-42-ppc64le/09940265-microdnf/backend.log by gzip
[2025-12-22 02:35:09,308][  INFO][PID:494351] Running command 'gzip /var/lib/copr/public_html/results/amatej/dnf-nightly-sanitizers/fedora-42-ppc64le/09940265-microdnf/backend.log' as PID 533646