[2025-06-03 15:41:48,240][  INFO][PID:3528073] Marking build as starting
[2025-06-03 15:41:48,382][  INFO][PID:3528073] Trying to allocate VM: ResallocHost, ticket_id=3329533, requested_tags=['copr_builder', 'arch_x86_64']
[2025-06-03 15:50:07,350][  INFO][PID:3528073] Allocated host ResallocHost, ticket_id=3329533, hostname=34.201.129.88, name=aws_x86_64_spot_prod_04557635_20250603_154300, requested_tags=['copr_builder', 'arch_x86_64']
[2025-06-03 15:50:07,352][  INFO][PID:3528073] Allocating ssh connection to builder
[2025-06-03 15:50:07,353][  INFO][PID:3528073] Checking that builder machine is OK
[2025-06-03 15:50:07,730][  INFO][PID:3528073] Running remote command: copr-builder-ready fedora-42-x86_64
[2025-06-03 15:50:07,984][  INFO][PID:3528073] Red Hat subscription not needed for fedora-42-x86_64
Builder is ready to be used

[2025-06-03 15:50:07,986][  INFO][PID:3528073] Filling build.info file with builder info
[2025-06-03 15:50:07,998][  INFO][PID:3528073] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "xxmitsu",
            "project_name": "mesa-and-llvm-git",
            "project_dirname": "mesa-and-llvm-git",
            "submitter": "https://src.fedoraproject.org/user/mcrha",
            "ended_on": null,
            "started_on": 1748965807.9966173,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-42-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [
                "git",
                "libxml2"
            ],
            "task_id": "9127755-fedora-42-x86_64",
            "build_id": 9127755,
            "package_name": "cairo",
            "package_version": "1.18.4-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xxmitsu/mesa-and-llvm-git/cairo",
            "git_hash": "b7b590365ac14b3af36790b1c79f163696add8ef",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "xxmitsu/mesa-and-llvm-git--https://src.fedoraproject.org/user/mcrha",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/mesa-and-llvm-git/fedora-42-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm/fedora-42-x86_64/",
                    "id": "copr_xxmitsu_llvm_priority_10",
                    "name": "Additional repo copr_xxmitsu_llvm_priority_10",
                    "priority": 10
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/mesa-and-llvm-git/fedora-42-x86_64/",
                    "id": "copr_xxmitsu_mesa_and_llvm_git_priority_10",
                    "name": "Additional repo copr_xxmitsu_mesa_and_llvm_git_priority_10",
                    "priority": 10
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/xxmitsu/mesa-and-llvm-git",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xxmitsu/mesa-and-llvm-git",
            "result_dir": "09127755-cairo",
            "built_packages": "",
            "id": 9127755,
            "mockchain_macros": {
                "copr_username": "xxmitsu",
                "copr_projectname": "mesa-and-llvm-git",
                "vendor": "Fedora Project COPR (xxmitsu/mesa-and-llvm-git)"
            }
        }
    ]
}
[2025-06-03 15:50:08,034][  INFO][PID:3528073] Sending fedora-messaging bus message in build.start
[2025-06-03 15:50:08,838][  INFO][PID:3528073] Sending fedora-messaging bus message in chroot.start
[2025-06-03 15:50:08,858][  INFO][PID:3528073] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9127755-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached
[2025-06-03 15:50:09,231][  INFO][PID:3528073] Downloading the builder-live.log file, attempt 1
[2025-06-03 15:50:09,241][  INFO][PID:3528073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.201.129.88 copr-rpmbuild-log
[2025-06-03 15:50:14,256][  INFO][PID:3528073] Periodic builder liveness probe: alive
[2025-06-03 15:50:28,720][ ERROR][PID:3528073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-06-03 15:50:38,722][WARNING][PID:3528073] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-06-03 15:50:48,727][  INFO][PID:3528073] Downloading the builder-live.log file, attempt 2
[2025-06-03 15:50:48,734][  INFO][PID:3528073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.201.129.88 copr-rpmbuild-log
[2025-06-03 15:51:48,830][ ERROR][PID:3528073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-06-03 15:51:58,843][WARNING][PID:3528073] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-06-03 15:52:08,848][  INFO][PID:3528073] Downloading the builder-live.log file, attempt 3
[2025-06-03 15:52:08,858][  INFO][PID:3528073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.201.129.88 copr-rpmbuild-log
[2025-06-03 15:53:08,994][ ERROR][PID:3528073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-06-03 15:53:18,997][WARNING][PID:3528073] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-06-03 15:53:29,005][  INFO][PID:3528073] Downloading the builder-live.log file, attempt 4
[2025-06-03 15:53:29,009][  INFO][PID:3528073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.201.129.88 copr-rpmbuild-log
[2025-06-03 15:54:29,149][ ERROR][PID:3528073] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-06-03 15:54:39,153][ ERROR][PID:3528073] Giving up for unstable SSH, failures: 1748965838.722511, 1748965918.8435771, 1748965998.9974868, 1748966079.153204
[2025-06-03 15:54:39,159][ ERROR][PID:3528073] Re-try request for task on 'ResallocHost, ticket_id=3329533, hostname=34.201.129.88, name=aws_x86_64_spot_prod_04557635_20250603_154300, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-06-03 15:54:39,162][  INFO][PID:3528073] Releasing VM back to pool
[2025-06-03 15:54:39,237][  INFO][PID:3528073] Retry #1 (on other host)
[2025-06-03 15:54:39,368][  INFO][PID:3528073] Trying to allocate VM: ResallocHost, ticket_id=3329816, requested_tags=['copr_builder', 'arch_x86_64']
[2025-06-03 16:03:58,367][  INFO][PID:3528073] Allocated host ResallocHost, ticket_id=3329816, hostname=35.173.255.176, name=aws_x86_64_spot_prod_04558862_20250603_160052, requested_tags=['copr_builder', 'arch_x86_64']
[2025-06-03 16:03:58,382][  INFO][PID:3528073] Allocating ssh connection to builder
[2025-06-03 16:03:58,383][  INFO][PID:3528073] Checking that builder machine is OK
[2025-06-03 16:03:58,963][  INFO][PID:3528073] Running remote command: copr-builder-ready fedora-42-x86_64
[2025-06-03 16:03:59,388][  INFO][PID:3528073] Red Hat subscription not needed for fedora-42-x86_64
Builder is ready to be used

[2025-06-03 16:03:59,390][  INFO][PID:3528073] Filling build.info file with builder info
[2025-06-03 16:03:59,404][  INFO][PID:3528073] Not re-notifying FE and msg buses for the new host.
[2025-06-03 16:03:59,412][  INFO][PID:3528073] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9127755-fedora-42-x86_64 --chroot fedora-42-x86_64 --detached
[2025-06-03 16:04:00,160][  INFO][PID:3528073] Downloading the builder-live.log file, attempt 1
[2025-06-03 16:04:00,172][  INFO][PID:3528073] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.173.255.176 copr-rpmbuild-log
[2025-06-03 16:04:05,220][  INFO][PID:3528073] Periodic builder liveness probe: alive
[2025-06-03 16:04:30,804][  INFO][PID:3528073] Downloading results from builder
[2025-06-03 16:04:30,810][  INFO][PID:3528073] rsyncing of mockbuilder@35.173.255.176:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/xxmitsu/mesa-and-llvm-git/fedora-42-x86_64/09127755-cairo started
[2025-06-03 16:04:30,827][  INFO][PID:3528073] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@35.173.255.176:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/xxmitsu/mesa-and-llvm-git/fedora-42-x86_64/09127755-cairo/ &> /var/lib/copr/public_html/results/xxmitsu/mesa-and-llvm-git/fedora-42-x86_64/09127755-cairo/build-09127755.rsync.log
[2025-06-03 16:04:31,291][  INFO][PID:3528073] rsyncing finished.
[2025-06-03 16:04:31,301][  INFO][PID:3528073] Releasing VM back to pool
[2025-06-03 16:04:31,354][  INFO][PID:3528073] Searching for 'success' file in resultdir
[2025-06-03 16:04:31,357][ ERROR][PID:3528073] Build failed: Backend process error: No success file => build failure
[2025-06-03 16:04:31,362][  INFO][PID:3528073] Finished build: id=9127755 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/xxmitsu/mesa-and-llvm-git chroot=fedora-42-x86_64 
[2025-06-03 16:04:31,373][ ERROR][PID:3528073] Backend process error: No success file => build failure
[2025-06-03 16:04:31,380][  INFO][PID:3528073] Worker failed build, took 31.97573709487915
[2025-06-03 16:04:31,383][  INFO][PID:3528073] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "xxmitsu",
            "project_name": "mesa-and-llvm-git",
            "project_dirname": "mesa-and-llvm-git",
            "submitter": "https://src.fedoraproject.org/user/mcrha",
            "ended_on": 1748966671.37995,
            "started_on": 1748966639.404213,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-42-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [
                "git",
                "libxml2"
            ],
            "task_id": "9127755-fedora-42-x86_64",
            "build_id": 9127755,
            "package_name": "cairo",
            "package_version": "1.18.4-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/xxmitsu/mesa-and-llvm-git/cairo",
            "git_hash": "b7b590365ac14b3af36790b1c79f163696add8ef",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "xxmitsu/mesa-and-llvm-git--https://src.fedoraproject.org/user/mcrha",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": true,
            "bootstrap": "off",
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/mesa-and-llvm-git/fedora-42-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/llvm/fedora-42-x86_64/",
                    "id": "copr_xxmitsu_llvm_priority_10",
                    "name": "Additional repo copr_xxmitsu_llvm_priority_10",
                    "priority": 10
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/xxmitsu/mesa-and-llvm-git/fedora-42-x86_64/",
                    "id": "copr_xxmitsu_mesa_and_llvm_git_priority_10",
                    "name": "Additional repo copr_xxmitsu_mesa_and_llvm_git_priority_10",
                    "priority": 10
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/xxmitsu/mesa-and-llvm-git",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/xxmitsu/mesa-and-llvm-git",
            "result_dir": "09127755-cairo",
            "built_packages": "",
            "id": 9127755,
            "mockchain_macros": {
                "copr_username": "xxmitsu",
                "copr_projectname": "mesa-and-llvm-git",
                "vendor": "Fedora Project COPR (xxmitsu/mesa-and-llvm-git)"
            }
        }
    ]
}
[2025-06-03 16:04:31,422][  INFO][PID:3528073] Sending fedora-messaging bus message in build.end
[2025-06-03 16:04:31,453][  INFO][PID:3528073] Compressing /var/lib/copr/public_html/results/xxmitsu/mesa-and-llvm-git/fedora-42-x86_64/09127755-cairo/builder-live.log by gzip
[2025-06-03 16:04:31,488][  INFO][PID:3528073] Running command 'gzip /var/lib/copr/public_html/results/xxmitsu/mesa-and-llvm-git/fedora-42-x86_64/09127755-cairo/builder-live.log' as PID 3719467
[2025-06-03 16:04:31,491][  INFO][PID:3528073] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/xxmitsu/mesa-and-llvm-git/fedora-42-x86_64/09127755-cairo/builder-live.log)
[2025-06-03 16:04:31,498][  INFO][PID:3528073] Compressing /var/lib/copr/public_html/results/xxmitsu/mesa-and-llvm-git/fedora-42-x86_64/09127755-cairo/backend.log by gzip
[2025-06-03 16:04:31,515][  INFO][PID:3528073] Running command 'gzip /var/lib/copr/public_html/results/xxmitsu/mesa-and-llvm-git/fedora-42-x86_64/09127755-cairo/backend.log' as PID 3719476