[2026-04-06 19:42:26,068][  INFO][PID:3366101] Marking build as starting
[2026-04-06 19:42:26,132][  INFO][PID:3366101] Checking that https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine/fedora-42-x86_64//repodata/repomd.xml exists
[2026-04-06 19:42:26,324][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:42:26,325][  INFO][PID:3366101] VM allocation process starts
[2026-04-06 19:42:26,335][  INFO][PID:3366101] Trying to allocate VM: ResallocHost, ticket_id=2194230, requested_tags=['copr_builder', 'arch_x86_64']
[2026-04-06 19:42:29,370][  INFO][PID:3366101] Allocated host ResallocHost, ticket_id=2194230, hostname=54.158.63.193, name=aws_x86_64_spot_prod_05879553_20260406_193809, requested_tags=['copr_builder', 'arch_x86_64']
[2026-04-06 19:42:29,370][  INFO][PID:3366101] Allocating ssh connection to builder
[2026-04-06 19:42:29,371][  INFO][PID:3366101] Checking that builder machine is OK
[2026-04-06 19:42:29,696][  INFO][PID:3366101] Running remote command: copr-builder-ready fedora-42-x86_64
[2026-04-06 19:42:29,932][  INFO][PID:3366101] Red Hat subscription not needed for fedora-42-x86_64
Builder is ready to be used

[2026-04-06 19:42:29,933][  INFO][PID:3366101] Filling build.info file with builder info
[2026-04-06 19:42:29,933][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:42:29,934][  INFO][PID:3366101] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "pvermeer",
            "project_name": "sunshine",
            "project_dirname": "sunshine",
            "submitter": null,
            "ended_on": null,
            "started_on": 1775504549.9344988,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-42-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10295700-fedora-42-x86_64",
            "build_id": 10295700,
            "package_name": "sunshine-beta",
            "package_version": "2026.403.124357-2",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/pvermeer/sunshine/sunshine-beta",
            "git_hash": "36a00f27d9f5ff92b998d90d0a2bd4739dea431a",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "pvermeer/sunshine--6e5b440d-dfad-471f-9ba6-84c12db822fa",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 1,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine/fedora-42-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/pvermeer/sunshine",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine",
            "result_dir": "10295700-sunshine-beta",
            "built_packages": "",
            "id": 10295700,
            "mockchain_macros": {
                "copr_username": "pvermeer",
                "copr_projectname": "sunshine",
                "vendor": "Fedora Project COPR (pvermeer/sunshine)"
            }
        }
    ]
}
[2026-04-06 19:42:29,976][  INFO][PID:3366101] Sending fedora-messaging bus message in build.start
[2026-04-06 19:42:30,609][  INFO][PID:3366101] Sending fedora-messaging bus message in chroot.start
[2026-04-06 19:42:30,636][  INFO][PID:3366101] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10295700-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached
[2026-04-06 19:42:30,972][  INFO][PID:3366101] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '54.158.63.193' (ED25519) to the list of known hosts.


[2026-04-06 19:42:30,972][  INFO][PID:3366101] Downloading the builder-live.log file, attempt 1
[2026-04-06 19:42:30,974][  INFO][PID:3366101] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.158.63.193 copr-rpmbuild-log
[2026-04-06 19:42:35,982][  INFO][PID:3366101] Periodic builder liveness probe: alive
[2026-04-06 19:42:35,983][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:42:40,983][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:42:45,984][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:42:50,986][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:42:55,987][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:43:00,988][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:43:05,989][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:43:10,989][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:43:15,990][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:43:20,992][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:43:25,992][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:43:30,994][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:43:35,995][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:43:40,996][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:43:45,999][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:43:51,000][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:43:56,001][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:44:01,002][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:44:06,003][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:44:11,004][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:44:16,005][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:44:21,006][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:44:26,007][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:44:31,008][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:44:36,009][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:44:41,010][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:44:46,011][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:44:51,012][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:44:56,012][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:45:01,013][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:45:06,014][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:45:11,015][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:45:16,016][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:45:21,017][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:45:26,018][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:45:31,019][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:45:36,020][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:45:41,021][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:45:46,022][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:45:51,023][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:45:56,024][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:46:01,025][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:46:06,025][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:46:11,026][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:46:16,027][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:46:21,028][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:46:26,029][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:46:31,030][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:46:36,031][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:46:41,032][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:46:46,033][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:46:51,034][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:46:56,035][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:47:01,036][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:47:06,037][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:47:11,038][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:47:16,039][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:47:21,040][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:47:26,041][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:47:31,042][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:47:36,043][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:47:41,044][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:47:46,045][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:47:51,046][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:47:56,047][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:48:01,048][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:48:06,049][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:48:11,050][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:48:16,051][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:48:21,052][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:48:26,053][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:48:31,055][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:48:36,056][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:48:41,057][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:48:46,058][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:48:51,059][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:48:56,060][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:49:01,061][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:49:06,062][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:49:11,063][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:49:16,064][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:49:21,065][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:49:26,066][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:49:31,067][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:49:36,068][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:49:41,069][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:49:46,070][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:49:51,071][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:49:56,072][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:50:01,073][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:50:06,074][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:50:11,075][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:50:16,076][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:50:21,077][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:50:26,078][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:50:31,079][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:50:36,080][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:50:41,081][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:50:45,481][ ERROR][PID:3366101] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-04-06 19:50:46,082][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:50:51,083][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:50:55,481][WARNING][PID:3366101] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-04-06 19:50:56,084][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:51:01,085][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:51:05,482][  INFO][PID:3366101] Downloading the builder-live.log file, attempt 2
[2026-04-06 19:51:05,549][  INFO][PID:3366101] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.158.63.193 copr-rpmbuild-log
[2026-04-06 19:51:06,086][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:51:11,088][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:51:16,089][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:51:21,090][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:51:26,091][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:51:31,092][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:51:36,093][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:51:41,094][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:51:46,095][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:51:51,096][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:51:56,098][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:52:01,099][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:52:05,810][ ERROR][PID:3366101] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-04-06 19:52:06,100][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:52:11,101][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:52:15,828][WARNING][PID:3366101] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-04-06 19:52:16,102][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:52:21,103][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:52:25,829][  INFO][PID:3366101] Downloading the builder-live.log file, attempt 3
[2026-04-06 19:52:25,831][  INFO][PID:3366101] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.158.63.193 copr-rpmbuild-log
[2026-04-06 19:52:26,104][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:52:31,105][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:52:36,113][  INFO][PID:3366101] Periodic builder liveness probe: alive
[2026-04-06 19:52:36,114][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:52:41,115][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:52:46,116][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:52:51,118][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:52:56,119][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:53:01,120][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:53:06,176][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:53:11,223][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:53:16,312][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:53:21,323][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:53:25,899][ ERROR][PID:3366101] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-04-06 19:53:26,395][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:53:31,567][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:53:35,902][WARNING][PID:3366101] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-04-06 19:53:36,569][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:53:41,572][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:53:45,902][  INFO][PID:3366101] Downloading the builder-live.log file, attempt 4
[2026-04-06 19:53:45,904][  INFO][PID:3366101] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.158.63.193 copr-rpmbuild-log
[2026-04-06 19:53:46,574][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:53:51,877][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:53:56,879][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:01,883][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:06,885][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:11,886][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:16,965][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:21,967][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:27,062][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:32,065][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:37,172][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:42,195][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:45,970][ ERROR][PID:3366101] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-04-06 19:54:47,209][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:52,210][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:55,971][ ERROR][PID:3366101] Giving up for unstable SSH, failures: 1775505055.481845, 1775505135.8287609, 1775505215.902091, 1775505295.9714074
[2026-04-06 19:54:55,972][ ERROR][PID:3366101] Re-try request for task on 'ResallocHost, ticket_id=2194230, hostname=54.158.63.193, name=aws_x86_64_spot_prod_05879553_20260406_193809, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-04-06 19:54:55,973][  INFO][PID:3366101] Releasing VM back to pool
[2026-04-06 19:54:55,985][  INFO][PID:3366101] Retry #1 (on other host)
[2026-04-06 19:54:55,986][  INFO][PID:3366101] Checking that https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine/fedora-42-x86_64//repodata/repomd.xml exists
[2026-04-06 19:54:56,316][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:54:56,318][  INFO][PID:3366101] VM allocation process starts
[2026-04-06 19:54:56,333][  INFO][PID:3366101] Trying to allocate VM: ResallocHost, ticket_id=2194319, requested_tags=['copr_builder', 'arch_x86_64']
[2026-04-06 19:54:59,349][  INFO][PID:3366101] Allocated host ResallocHost, ticket_id=2194319, hostname=34.229.229.69, name=aws_x86_64_spot_prod_05879755_20260406_195057, requested_tags=['copr_builder', 'arch_x86_64']
[2026-04-06 19:54:59,349][  INFO][PID:3366101] Allocating ssh connection to builder
[2026-04-06 19:54:59,350][  INFO][PID:3366101] Checking that builder machine is OK
[2026-04-06 19:55:59,410][ ERROR][PID:3366101] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 34.229.229.69 port 22: Connection timed out

[2026-04-06 19:56:09,426][ ERROR][PID:3366101] Re-try request for task on 'ResallocHost, ticket_id=2194319, hostname=34.229.229.69, name=aws_x86_64_spot_prod_05879755_20260406_195057, requested_tags=['copr_builder', 'arch_x86_64']': Unable to finish after 1 SSH attempts
[2026-04-06 19:56:09,429][  INFO][PID:3366101] Releasing VM back to pool
[2026-04-06 19:56:09,447][  INFO][PID:3366101] Retry #2 (on other host)
[2026-04-06 19:56:09,448][  INFO][PID:3366101] Checking that https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine/fedora-42-x86_64//repodata/repomd.xml exists
[2026-04-06 19:56:09,679][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:56:09,681][  INFO][PID:3366101] VM allocation process starts
[2026-04-06 19:56:09,694][  INFO][PID:3366101] Trying to allocate VM: ResallocHost, ticket_id=2194341, requested_tags=['copr_builder', 'arch_x86_64']
[2026-04-06 19:56:12,710][  INFO][PID:3366101] Allocated host ResallocHost, ticket_id=2194341, hostname=44.192.76.61, name=aws_x86_64_spot_prod_05879840_20260406_195300, requested_tags=['copr_builder', 'arch_x86_64']
[2026-04-06 19:56:12,711][  INFO][PID:3366101] Allocating ssh connection to builder
[2026-04-06 19:56:12,711][  INFO][PID:3366101] Checking that builder machine is OK
[2026-04-06 19:56:13,045][  INFO][PID:3366101] Running remote command: copr-builder-ready fedora-42-x86_64
[2026-04-06 19:56:13,268][  INFO][PID:3366101] Red Hat subscription not needed for fedora-42-x86_64
Builder is ready to be used

[2026-04-06 19:56:13,269][  INFO][PID:3366101] Filling build.info file with builder info
[2026-04-06 19:56:13,282][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:56:13,287][  INFO][PID:3366101] Not re-notifying FE and msg buses for the new host.
[2026-04-06 19:56:13,289][  INFO][PID:3366101] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10295700-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached
[2026-04-06 19:56:13,635][  INFO][PID:3366101] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '44.192.76.61' (ED25519) to the list of known hosts.


[2026-04-06 19:56:13,637][  INFO][PID:3366101] Downloading the builder-live.log file, attempt 1
[2026-04-06 19:56:13,639][  INFO][PID:3366101] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.192.76.61 copr-rpmbuild-log
[2026-04-06 19:56:18,637][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:56:23,651][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:56:28,668][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:56:33,694][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:56:38,695][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:56:43,696][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:56:48,697][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:56:53,698][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:56:58,699][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:57:03,700][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:57:08,718][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:57:13,719][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:57:18,720][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:57:23,722][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:57:28,723][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:57:33,724][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:57:38,725][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:57:43,726][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:57:48,727][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:57:53,728][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:57:58,814][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:58:03,924][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:58:08,929][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:58:13,938][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:58:18,952][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:58:23,985][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:58:29,172][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:58:34,173][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:58:39,179][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:58:44,181][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:58:49,182][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:58:54,183][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:58:59,210][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:59:04,213][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:59:09,217][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:59:14,219][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:59:19,220][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:59:24,221][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:59:29,222][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:59:34,293][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:59:39,530][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:59:44,532][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:59:49,632][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:59:54,897][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 19:59:59,898][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:00:04,899][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:00:09,956][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:00:15,083][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:00:20,089][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:00:25,091][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:00:30,096][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:00:35,098][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:00:40,101][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:00:45,102][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:00:50,102][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:00:55,168][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:01:00,177][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:01:03,664][ ERROR][PID:3366101] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-04-06 20:01:05,183][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:01:10,186][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:01:13,665][WARNING][PID:3366101] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-04-06 20:01:15,189][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:01:20,194][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:01:23,667][  INFO][PID:3366101] Downloading the builder-live.log file, attempt 2
[2026-04-06 20:01:23,669][  INFO][PID:3366101] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.192.76.61 copr-rpmbuild-log
[2026-04-06 20:01:25,198][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:01:30,199][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:01:35,204][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:01:40,205][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:01:45,208][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:01:50,209][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:01:55,214][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:02:00,216][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:02:05,217][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:02:10,219][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:02:15,229][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:02:20,237][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:02:23,738][ ERROR][PID:3366101] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-04-06 20:02:25,238][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:02:30,240][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:02:33,753][WARNING][PID:3366101] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-04-06 20:02:35,247][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:02:40,254][  INFO][PID:3366101] Periodic builder liveness probe: alive
[2026-04-06 20:02:40,255][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:02:43,754][  INFO][PID:3366101] Downloading the builder-live.log file, attempt 3
[2026-04-06 20:02:43,755][  INFO][PID:3366101] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.192.76.61 copr-rpmbuild-log
[2026-04-06 20:02:45,255][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:02:50,256][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:02:55,259][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:03:00,260][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:03:05,261][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:03:10,282][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:03:15,283][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:03:20,284][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:03:25,285][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:03:30,286][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:03:35,287][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:03:40,288][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:03:43,823][ ERROR][PID:3366101] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-04-06 20:03:45,289][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:03:50,290][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:03:53,839][WARNING][PID:3366101] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-04-06 20:03:55,291][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:04:00,292][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:04:03,840][  INFO][PID:3366101] Downloading the builder-live.log file, attempt 4
[2026-04-06 20:04:03,841][  INFO][PID:3366101] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.192.76.61 copr-rpmbuild-log
[2026-04-06 20:04:05,293][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:04:10,293][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:04:15,294][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:04:20,514][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:04:25,515][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:04:30,516][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:04:35,517][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:04:40,518][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:04:45,520][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:04:50,521][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:04:55,522][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:05:00,522][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:05:03,907][ ERROR][PID:3366101] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-04-06 20:05:05,523][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:05:10,525][  INFO][PID:3366101] Checking for cancel request
[2026-04-06 20:05:13,911][ ERROR][PID:3366101] Giving up for unstable SSH, failures: 1775505673.664944, 1775505753.753093, 1775505833.8394654, 1775505913.9116888
[2026-04-06 20:05:13,912][ ERROR][PID:3366101] Re-try request for task on 'ResallocHost, ticket_id=2194341, hostname=44.192.76.61, name=aws_x86_64_spot_prod_05879840_20260406_195300, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-04-06 20:05:13,913][  INFO][PID:3366101] Releasing VM back to pool
[2026-04-06 20:05:13,922][ ERROR][PID:3366101] Backend process error: Three host tried without success: {'44.192.76.61', '54.158.63.193', '34.229.229.69'}
[2026-04-06 20:05:13,922][WARNING][PID:3366101] Switching not-finished job state to 'failed'
[2026-04-06 20:05:13,923][  INFO][PID:3366101] Worker failed build, took 540.6358008384705
[2026-04-06 20:05:13,923][  INFO][PID:3366101] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "pvermeer",
            "project_name": "sunshine",
            "project_dirname": "sunshine",
            "submitter": null,
            "ended_on": 1775505913.9228919,
            "started_on": 1775505373.287091,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-42-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10295700-fedora-42-x86_64",
            "build_id": 10295700,
            "package_name": "sunshine-beta",
            "package_version": "2026.403.124357-2",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/pvermeer/sunshine/sunshine-beta",
            "git_hash": "36a00f27d9f5ff92b998d90d0a2bd4739dea431a",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "pvermeer/sunshine--6e5b440d-dfad-471f-9ba6-84c12db822fa",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 1,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine/fedora-42-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/pvermeer/sunshine",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/pvermeer/sunshine",
            "result_dir": "10295700-sunshine-beta",
            "built_packages": "",
            "id": 10295700,
            "mockchain_macros": {
                "copr_username": "pvermeer",
                "copr_projectname": "sunshine",
                "vendor": "Fedora Project COPR (pvermeer/sunshine)"
            }
        }
    ]
}
[2026-04-06 20:05:13,966][  INFO][PID:3366101] Sending fedora-messaging bus message in build.end
[2026-04-06 20:05:13,997][  INFO][PID:3366101] Compressing /var/lib/copr/public_html/results/pvermeer/sunshine/fedora-42-x86_64/10295700-sunshine-beta/builder-live.log by gzip
[2026-04-06 20:05:13,998][  INFO][PID:3366101] Running command 'gzip /var/lib/copr/public_html/results/pvermeer/sunshine/fedora-42-x86_64/10295700-sunshine-beta/builder-live.log' as PID 3485469
[2026-04-06 20:05:13,999][  INFO][PID:3366101] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/pvermeer/sunshine/fedora-42-x86_64/10295700-sunshine-beta/builder-live.log)
[2026-04-06 20:05:13,999][  INFO][PID:3366101] Compressing /var/lib/copr/public_html/results/pvermeer/sunshine/fedora-42-x86_64/10295700-sunshine-beta/backend.log by gzip
[2026-04-06 20:05:14,000][  INFO][PID:3366101] Running command 'gzip /var/lib/copr/public_html/results/pvermeer/sunshine/fedora-42-x86_64/10295700-sunshine-beta/backend.log' as PID 3485470