[2025-06-03 17:03:04,003][  INFO][PID:4192467] Marking build as starting
[2025-06-03 17:03:05,043][  INFO][PID:4192467] Trying to allocate VM: ResallocHost, ticket_id=3331437, requested_tags=['copr_builder', 'arch_noarch']
[2025-06-03 17:07:24,105][  INFO][PID:4192467] Allocated host ResallocHost, ticket_id=3331437, hostname=140.211.168.199, name=copr_osuosl_p09_prod_04562480_20250603_165953, requested_tags=['copr_builder', 'arch_noarch']
[2025-06-03 17:07:24,107][  INFO][PID:4192467] Allocating ssh connection to builder
[2025-06-03 17:07:24,111][  INFO][PID:4192467] Checking that builder machine is OK
[2025-06-03 17:07:25,933][  INFO][PID:4192467] Running remote command: copr-builder-ready srpm-builds
[2025-06-03 17:07:27,276][  INFO][PID:4192467] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-06-03 17:07:27,288][  INFO][PID:4192467] Filling build.info file with builder info
[2025-06-03 17:07:27,295][  INFO][PID:4192467] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "@kicad",
            "project_name": "kicad",
            "project_dirname": "kicad",
            "submitter": "stevenfalco",
            "ended_on": null,
            "started_on": 1748970447.2954803,
            "submitted_on": null,
            "status": 3,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "9128219",
            "build_id": 9128219,
            "package_name": null,
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 2,
            "source_json": "{\"url\": \"https://copr.fedorainfracloud.org/tmp/tmpkg7xoqqo/kicad-nightly.spec\", \"pkg\": \"kicad-nightly.spec\", \"tmp\": \"tmpkg7xoqqo\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "@kicad/kicad--stevenfalco",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "background": false,
            "repos": [],
            "destdir": "/var/lib/copr/public_html/results/@kicad/kicad",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@kicad/kicad",
            "result_dir": "09128219",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "id": 9128219,
            "mockchain_macros": {
                "copr_username": "@kicad",
                "copr_projectname": "kicad",
                "vendor": "Fedora Project COPR (@kicad/kicad)"
            }
        }
    ]
}
[2025-06-03 17:07:27,362][  INFO][PID:4192467] Sending fedora-messaging bus message in build.start
[2025-06-03 17:07:29,580][  INFO][PID:4192467] Sending fedora-messaging bus message in chroot.start
[2025-06-03 17:07:29,618][  INFO][PID:4192467] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9128219 --detached
[2025-06-03 17:07:31,387][  INFO][PID:4192467] Downloading the builder-live.log file, attempt 1
[2025-06-03 17:07:31,398][  INFO][PID:4192467] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.199 copr-rpmbuild-log
[2025-06-03 17:07:36,459][  INFO][PID:4192467] Periodic builder liveness probe: alive
[2025-06-03 17:17:37,345][  INFO][PID:4192467] Periodic builder liveness probe: alive
[2025-06-03 17:27:38,358][  INFO][PID:4192467] Periodic builder liveness probe: alive
[2025-06-03 17:37:39,687][  INFO][PID:4192467] Periodic builder liveness probe: alive
[2025-06-03 17:47:40,763][  INFO][PID:4192467] Periodic builder liveness probe: alive
[2025-06-03 17:57:41,753][  INFO][PID:4192467] Periodic builder liveness probe: dead
[2025-06-03 17:57:53,147][ ERROR][PID:4192467] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-06-03 17:58:03,162][WARNING][PID:4192467] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-06-03 17:58:13,169][  INFO][PID:4192467] Downloading the builder-live.log file, attempt 2
[2025-06-03 17:58:13,176][  INFO][PID:4192467] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.199 copr-rpmbuild-log
[2025-06-03 17:58:41,853][ ERROR][PID:4192467] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.199 port 22: Connection timed out

[2025-06-03 17:59:13,329][ ERROR][PID:4192467] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-06-03 17:59:23,332][WARNING][PID:4192467] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-06-03 17:59:33,367][  INFO][PID:4192467] Downloading the builder-live.log file, attempt 3
[2025-06-03 17:59:33,425][  INFO][PID:4192467] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.199 copr-rpmbuild-log
[2025-06-03 17:59:52,076][ ERROR][PID:4192467] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.199 port 22: Connection timed out

[2025-06-03 18:00:33,562][ ERROR][PID:4192467] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-06-03 18:00:43,570][WARNING][PID:4192467] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-06-03 18:00:53,589][  INFO][PID:4192467] Downloading the builder-live.log file, attempt 4
[2025-06-03 18:00:53,597][  INFO][PID:4192467] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.199 copr-rpmbuild-log
[2025-06-03 18:01:02,379][ ERROR][PID:4192467] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.199 port 22: Connection timed out

[2025-06-03 18:01:53,721][ ERROR][PID:4192467] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-06-03 18:02:03,742][ ERROR][PID:4192467] Giving up for unstable SSH, failures: 1748973483.1626472, 1748973563.3320198, 1748973643.5709221, 1748973723.742151
[2025-06-03 18:02:12,577][ ERROR][PID:4192467] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.199 port 22: Connection timed out

[2025-06-03 18:02:22,587][ ERROR][PID:4192467] Can't ssh to cancel build.
[2025-06-03 18:02:22,590][ ERROR][PID:4192467] Re-try request for task on 'ResallocHost, ticket_id=3331437, hostname=140.211.168.199, name=copr_osuosl_p09_prod_04562480_20250603_165953, requested_tags=['copr_builder', 'arch_noarch']': SSH problems when downloading live log: Resalloc ticket FAILED
[2025-06-03 18:02:22,591][  INFO][PID:4192467] Releasing VM back to pool
[2025-06-03 18:02:22,637][  INFO][PID:4192467] Retry #1 (on other host)
[2025-06-03 18:02:22,680][  INFO][PID:4192467] Trying to allocate VM: ResallocHost, ticket_id=3332850, requested_tags=['copr_builder', 'arch_noarch']
[2025-06-03 18:03:00,971][  INFO][PID:4192467] Allocated host ResallocHost, ticket_id=3332850, hostname=2620:52:3:1:dead:beef:cafe:c2a2, name=copr_p09_01_prod_04565364_20250603_175601, requested_tags=['copr_builder', 'arch_noarch']
[2025-06-03 18:03:00,982][  INFO][PID:4192467] Allocating ssh connection to builder
[2025-06-03 18:03:00,999][  INFO][PID:4192467] Checking that builder machine is OK
[2025-06-03 18:03:01,985][  INFO][PID:4192467] Running remote command: copr-builder-ready srpm-builds
[2025-06-03 18:03:02,689][  INFO][PID:4192467] Red Hat subscription not needed for srpm-builds
Builder is ready to be used

[2025-06-03 18:03:02,691][  INFO][PID:4192467] Filling build.info file with builder info
[2025-06-03 18:03:02,735][  INFO][PID:4192467] Not re-notifying FE and msg buses for the new host.
[2025-06-03 18:03:02,738][  INFO][PID:4192467] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --srpm --task-url https://copr.fedorainfracloud.org/backend/get-srpm-build-task/9128219 --detached
[2025-06-03 18:03:03,895][  INFO][PID:4192467] Downloading the builder-live.log file, attempt 1
[2025-06-03 18:03:03,915][  INFO][PID:4192467] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c2a2 copr-rpmbuild-log
[2025-06-03 18:03:13,013][  INFO][PID:4192467] Cancel request succeeded
out:
success
err:
Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c2a2' (ED25519) to the list of known hosts.

[2025-06-03 18:03:13,015][  INFO][PID:4192467] Downloading results from builder
[2025-06-03 18:03:13,016][  INFO][PID:4192467] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a2]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@kicad/kicad/srpm-builds/09128219 started
[2025-06-03 18:03:13,018][  INFO][PID:4192467] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a2]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@kicad/kicad/srpm-builds/09128219/ &> /var/lib/copr/public_html/results/@kicad/kicad/srpm-builds/09128219/build-09128219.rsync.log
[2025-06-03 18:03:14,002][  INFO][PID:4192467] rsyncing finished.
[2025-06-03 18:03:14,005][  INFO][PID:4192467] Releasing VM back to pool
[2025-06-03 18:03:14,062][  INFO][PID:4192467] Searching for 'success' file in resultdir
[2025-06-03 18:03:14,072][ ERROR][PID:4192467] Build failed: Backend process error: No success file => build failure
[2025-06-03 18:03:14,076][  INFO][PID:4192467] Finished build: id=9128219 failed=True timeout=108000 destdir=/var/lib/copr/public_html/results/@kicad/kicad chroot=srpm-builds 
[2025-06-03 18:03:14,142][ ERROR][PID:4192467] Backend process error: No success file => build failure
[2025-06-03 18:03:14,149][  INFO][PID:4192467] Worker failed build, took 11.414435625076294
[2025-06-03 18:03:14,157][  INFO][PID:4192467] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": null,
            "enable_net": true,
            "project_owner": "@kicad",
            "project_name": "kicad",
            "project_dirname": "kicad",
            "submitter": "stevenfalco",
            "ended_on": 1748973794.149457,
            "started_on": 1748973782.7350214,
            "submitted_on": null,
            "status": 0,
            "chroot": "srpm-builds",
            "arch": null,
            "buildroot_pkgs": null,
            "task_id": "9128219",
            "build_id": 9128219,
            "package_name": null,
            "package_version": null,
            "git_repo": null,
            "git_hash": null,
            "git_branch": null,
            "source_type": 2,
            "source_json": "{\"url\": \"https://copr.fedorainfracloud.org/tmp/tmpkg7xoqqo/kicad-nightly.spec\", \"pkg\": \"kicad-nightly.spec\", \"tmp\": \"tmpkg7xoqqo\"}",
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": null,
            "sandbox": "@kicad/kicad--stevenfalco",
            "results": null,
            "appstream": false,
            "allow_user_ssh": null,
            "ssh_public_keys": null,
            "storage": null,
            "background": false,
            "repos": [],
            "destdir": "/var/lib/copr/public_html/results/@kicad/kicad",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@kicad/kicad",
            "result_dir": "09128219",
            "built_packages": "",
            "tags": [
                "arch_noarch"
            ],
            "id": 9128219,
            "mockchain_macros": {
                "copr_username": "@kicad",
                "copr_projectname": "kicad",
                "vendor": "Fedora Project COPR (@kicad/kicad)"
            }
        }
    ]
}
[2025-06-03 18:03:14,266][  INFO][PID:4192467] Sending fedora-messaging bus message in build.end
[2025-06-03 18:03:14,348][  INFO][PID:4192467] Compressing /var/lib/copr/public_html/results/@kicad/kicad/srpm-builds/09128219/builder-live.log by gzip
[2025-06-03 18:03:14,362][  INFO][PID:4192467] Running command 'gzip /var/lib/copr/public_html/results/@kicad/kicad/srpm-builds/09128219/builder-live.log' as PID 470664
[2025-06-03 18:03:14,401][  INFO][PID:4192467] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@kicad/kicad/srpm-builds/09128219/builder-live.log)
[2025-06-03 18:03:14,404][  INFO][PID:4192467] Compressing /var/lib/copr/public_html/results/@kicad/kicad/srpm-builds/09128219/backend.log by gzip
[2025-06-03 18:03:14,422][  INFO][PID:4192467] Running command 'gzip /var/lib/copr/public_html/results/@kicad/kicad/srpm-builds/09128219/backend.log' as PID 470671