[2026-03-30 21:47:25,068][  INFO][PID:2756301] Marking build as starting
[2026-03-30 21:47:25,178][  INFO][PID:2756301] Checking that /var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS/openeuler-24.03-x86_64/repodata/repomd.xml exists
[2026-03-30 21:47:25,186][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:47:25,187][  INFO][PID:2756301] VM allocation process starts
[2026-03-30 21:47:25,200][  INFO][PID:2756301] Trying to allocate VM: ResallocHost, ticket_id=2100109, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-30 21:47:30,201][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:47:35,203][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:47:40,205][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:47:45,207][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:47:50,208][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:47:55,209][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:00,210][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:05,211][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:10,212][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:15,214][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:20,215][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:25,218][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:30,219][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:35,220][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:40,221][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:45,222][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:50,223][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:53,273][  INFO][PID:2756301] Allocated host ResallocHost, ticket_id=2100109, hostname=35.170.242.92, name=aws_x86_64_spot_prod_04400259_20260330_214255, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-30 21:48:53,274][  INFO][PID:2756301] Allocating ssh connection to builder
[2026-03-30 21:48:53,274][  INFO][PID:2756301] Checking that builder machine is OK
[2026-03-30 21:48:53,610][  INFO][PID:2756301] Running remote command: copr-builder-ready openeuler-24.03-x86_64
[2026-03-30 21:48:53,816][  INFO][PID:2756301] Red Hat subscription not needed for openeuler-24.03-x86_64
Builder is ready to be used

[2026-03-30 21:48:53,817][  INFO][PID:2756301] Filling build.info file with builder info
[2026-03-30 21:48:53,817][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:48:53,818][  INFO][PID:2756301] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "victoriametrics",
            "project_name": "VictoriaMetrics-LTS",
            "project_dirname": "VictoriaMetrics-LTS",
            "submitter": "https://api.github.com/users/denisgolius",
            "ended_on": null,
            "started_on": 1774907333.8186924,
            "submitted_on": null,
            "status": 3,
            "chroot": "openeuler-24.03-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10274286-openeuler-24.03-x86_64",
            "build_id": 10274286,
            "package_name": "vminsert",
            "package_version": "1.139.0-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/victoriametrics/VictoriaMetrics-LTS/vminsert",
            "git_hash": "437a6d12e27ce0eb856b25cd1d4dd50da8e15a8e",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "victoriametrics/VictoriaMetrics-LTS--https://api.github.com/users/denisgolius",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/victoriametrics/VictoriaMetrics-LTS/openeuler-24.03-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": 1
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": 1,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/victoriametrics/VictoriaMetrics-LTS",
            "result_dir": "10274286-vminsert",
            "built_packages": "",
            "id": 10274286,
            "mockchain_macros": {
                "copr_username": "victoriametrics",
                "copr_projectname": "VictoriaMetrics-LTS",
                "vendor": "Fedora Project COPR (victoriametrics/VictoriaMetrics-LTS)"
            }
        }
    ]
}
[2026-03-30 21:48:53,927][  INFO][PID:2756301] Sending fedora-messaging bus message in build.start
[2026-03-30 21:48:54,621][  INFO][PID:2756301] Sending fedora-messaging bus message in chroot.start
[2026-03-30 21:48:54,703][  INFO][PID:2756301] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10274286-openeuler-24.03-x86_64 --chroot openeuler-24.03-x86_64 --detached
[2026-03-30 21:48:55,018][  INFO][PID:2756301] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '35.170.242.92' (ED25519) to the list of known hosts.


[2026-03-30 21:48:55,019][  INFO][PID:2756301] Downloading the builder-live.log file, attempt 1
[2026-03-30 21:48:55,021][  INFO][PID:2756301] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.170.242.92 copr-rpmbuild-log
[2026-03-30 21:49:00,029][  INFO][PID:2756301] Periodic builder liveness probe: alive
[2026-03-30 21:49:00,030][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:49:05,032][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:49:10,033][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:49:15,035][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:49:20,036][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:49:25,037][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:49:30,038][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:49:32,942][ ERROR][PID:2756301] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-30 21:49:35,040][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:49:40,041][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:49:42,942][WARNING][PID:2756301] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-30 21:49:45,042][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:49:50,043][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:49:52,944][  INFO][PID:2756301] Downloading the builder-live.log file, attempt 2
[2026-03-30 21:49:52,948][  INFO][PID:2756301] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.170.242.92 copr-rpmbuild-log
[2026-03-30 21:49:55,046][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:50:00,047][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:50:05,048][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:50:10,049][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:50:15,050][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:50:20,051][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:50:25,053][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:50:30,054][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:50:35,055][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:50:40,075][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:50:45,077][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:50:50,079][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:50:53,023][ ERROR][PID:2756301] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-30 21:50:55,081][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:51:00,082][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:51:03,041][WARNING][PID:2756301] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-30 21:51:05,083][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:51:10,085][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:51:13,042][  INFO][PID:2756301] Downloading the builder-live.log file, attempt 3
[2026-03-30 21:51:13,044][  INFO][PID:2756301] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.170.242.92 copr-rpmbuild-log
[2026-03-30 21:51:15,087][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:51:20,088][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:51:25,090][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:51:30,091][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:51:35,092][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:51:40,094][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:51:45,095][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:51:50,096][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:51:55,098][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:52:00,100][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:52:05,104][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:52:10,105][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:52:13,111][ ERROR][PID:2756301] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-30 21:52:15,108][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:52:20,109][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:52:23,405][WARNING][PID:2756301] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2026-03-30 21:52:25,142][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:52:30,143][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:52:33,406][  INFO][PID:2756301] Downloading the builder-live.log file, attempt 4
[2026-03-30 21:52:33,738][  INFO][PID:2756301] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.170.242.92 copr-rpmbuild-log
[2026-03-30 21:52:35,145][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:52:40,146][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:52:45,148][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:52:50,151][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:52:55,192][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:00,193][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:05,210][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:10,262][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:15,263][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:20,280][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:25,467][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:30,581][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:33,807][ ERROR][PID:2756301] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2026-03-30 21:53:35,896][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:40,906][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:43,858][ ERROR][PID:2756301] Giving up for unstable SSH, failures: 1774907382.9429092, 1774907463.0418546, 1774907543.4050784, 1774907623.8579211
[2026-03-30 21:53:43,859][ ERROR][PID:2756301] Re-try request for task on 'ResallocHost, ticket_id=2100109, hostname=35.170.242.92, name=aws_x86_64_spot_prod_04400259_20260330_214255, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2026-03-30 21:53:43,860][  INFO][PID:2756301] Releasing VM back to pool
[2026-03-30 21:53:43,872][  INFO][PID:2756301] Retry #1 (on other host)
[2026-03-30 21:53:43,873][  INFO][PID:2756301] Checking that /var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS/openeuler-24.03-x86_64/repodata/repomd.xml exists
[2026-03-30 21:53:43,907][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:43,908][  INFO][PID:2756301] VM allocation process starts
[2026-03-30 21:53:43,926][  INFO][PID:2756301] Trying to allocate VM: ResallocHost, ticket_id=2100239, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-30 21:53:48,927][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:49,986][  INFO][PID:2756301] Allocated host ResallocHost, ticket_id=2100239, hostname=44.222.179.60, name=aws_x86_64_normalreserved_prod_04400286_20260330_214400, requested_tags=['copr_builder', 'arch_x86_64']
[2026-03-30 21:53:49,987][  INFO][PID:2756301] Allocating ssh connection to builder
[2026-03-30 21:53:49,988][  INFO][PID:2756301] Checking that builder machine is OK
[2026-03-30 21:53:50,366][  INFO][PID:2756301] Running remote command: copr-builder-ready openeuler-24.03-x86_64
[2026-03-30 21:53:50,622][  INFO][PID:2756301] Red Hat subscription not needed for openeuler-24.03-x86_64
Builder is ready to be used

[2026-03-30 21:53:50,624][  INFO][PID:2756301] Filling build.info file with builder info
[2026-03-30 21:53:50,663][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:53:50,664][  INFO][PID:2756301] Not re-notifying FE and msg buses for the new host.
[2026-03-30 21:53:50,665][  INFO][PID:2756301] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/10274286-openeuler-24.03-x86_64 --chroot openeuler-24.03-x86_64 --detached
[2026-03-30 21:53:51,029][  INFO][PID:2756301] The copr-rpmbuild seems started, per:
stdout: 
stderr: Warning: Permanently added '44.222.179.60' (ED25519) to the list of known hosts.


[2026-03-30 21:53:51,030][  INFO][PID:2756301] Downloading the builder-live.log file, attempt 1
[2026-03-30 21:53:51,032][  INFO][PID:2756301] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.222.179.60 copr-rpmbuild-log
[2026-03-30 21:53:56,030][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:54:01,053][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:54:06,060][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:54:11,061][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:54:16,062][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:54:21,131][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:54:26,132][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:54:31,133][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:54:36,136][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:54:41,138][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:54:46,140][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:54:51,141][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:54:56,142][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:55:01,143][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:55:06,163][  INFO][PID:2756301] Checking for cancel request
[2026-03-30 21:55:08,191][  INFO][PID:2756301] Downloading results from builder
[2026-03-30 21:55:08,192][  INFO][PID:2756301] rsyncing of mockbuilder@44.222.179.60:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS/openeuler-24.03-x86_64/10274286-vminsert started
[2026-03-30 21:55:08,193][  INFO][PID:2756301] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@44.222.179.60:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS/openeuler-24.03-x86_64/10274286-vminsert/ &> /var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS/openeuler-24.03-x86_64/10274286-vminsert/build-10274286.rsync.log
[2026-03-30 21:55:08,474][  INFO][PID:2756301] rsyncing finished.
[2026-03-30 21:55:08,475][  INFO][PID:2756301] Releasing VM back to pool
[2026-03-30 21:55:08,485][  INFO][PID:2756301] Searching for 'success' file in resultdir
[2026-03-30 21:55:08,486][ ERROR][PID:2756301] Build failed: Backend process error: No success file => build failure
[2026-03-30 21:55:08,486][  INFO][PID:2756301] Finished build: id=10274286 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS chroot=openeuler-24.03-x86_64 
[2026-03-30 21:55:08,488][ ERROR][PID:2756301] Backend process error: No success file => build failure
[2026-03-30 21:55:08,489][  INFO][PID:2756301] Worker failed build, took 77.82450795173645
[2026-03-30 21:55:08,489][  INFO][PID:2756301] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "victoriametrics",
            "project_name": "VictoriaMetrics-LTS",
            "project_dirname": "VictoriaMetrics-LTS",
            "submitter": "https://api.github.com/users/denisgolius",
            "ended_on": 1774907708.489007,
            "started_on": 1774907630.664499,
            "submitted_on": null,
            "status": 0,
            "chroot": "openeuler-24.03-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "10274286-openeuler-24.03-x86_64",
            "build_id": 10274286,
            "package_name": "vminsert",
            "package_version": "1.139.0-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/victoriametrics/VictoriaMetrics-LTS/vminsert",
            "git_hash": "437a6d12e27ce0eb856b25cd1d4dd50da8e15a8e",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "victoriametrics/VictoriaMetrics-LTS--https://api.github.com/users/denisgolius",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/victoriametrics/VictoriaMetrics-LTS/openeuler-24.03-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": 1
                }
            ],
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": 1,
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/victoriametrics/VictoriaMetrics-LTS",
            "result_dir": "10274286-vminsert",
            "built_packages": "",
            "id": 10274286,
            "mockchain_macros": {
                "copr_username": "victoriametrics",
                "copr_projectname": "VictoriaMetrics-LTS",
                "vendor": "Fedora Project COPR (victoriametrics/VictoriaMetrics-LTS)"
            }
        }
    ]
}
[2026-03-30 21:55:08,575][  INFO][PID:2756301] Sending fedora-messaging bus message in build.end
[2026-03-30 21:55:08,605][  INFO][PID:2756301] Compressing /var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS/openeuler-24.03-x86_64/10274286-vminsert/builder-live.log by gzip
[2026-03-30 21:55:08,607][  INFO][PID:2756301] Running command 'gzip /var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS/openeuler-24.03-x86_64/10274286-vminsert/builder-live.log' as PID 2806403
[2026-03-30 21:55:08,638][  INFO][PID:2756301] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS/openeuler-24.03-x86_64/10274286-vminsert/builder-live.log)
[2026-03-30 21:55:08,638][  INFO][PID:2756301] Compressing /var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS/openeuler-24.03-x86_64/10274286-vminsert/backend.log by gzip
[2026-03-30 21:55:08,640][  INFO][PID:2756301] Running command 'gzip /var/lib/copr/public_html/results/victoriametrics/VictoriaMetrics-LTS/openeuler-24.03-x86_64/10274286-vminsert/backend.log' as PID 2806426