[2025-04-25 07:14:27,672][  INFO][PID:872607] Marking build as starting
[2025-04-25 07:14:27,749][  INFO][PID:872607] Trying to allocate VM: ResallocHost, ticket_id=2657787, requested_tags=['copr_builder', 'arch_x86_64']
[2025-04-25 07:14:30,816][  INFO][PID:872607] Allocated host ResallocHost, ticket_id=2657787, hostname=18.232.153.163, name=aws_x86_64_spot_prod_03317081_20250425_070628, requested_tags=['copr_builder', 'arch_x86_64']
[2025-04-25 07:14:30,816][  INFO][PID:872607] Allocating ssh connection to builder
[2025-04-25 07:14:30,818][  INFO][PID:872607] Checking that builder machine is OK
[2025-04-25 07:14:31,213][  INFO][PID:872607] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2025-04-25 07:14:31,470][  INFO][PID:872607] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2025-04-25 07:14:31,471][  INFO][PID:872607] Filling build.info file with builder info
[2025-04-25 07:14:31,472][  INFO][PID:872607] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "thrnciar",
            "project_name": "setuptools-78.1.1-control",
            "project_dirname": "setuptools-78.1.1-control",
            "submitter": "thrnciar",
            "ended_on": null,
            "started_on": 1745565271.4724016,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8966030-fedora-rawhide-x86_64",
            "build_id": 8966030,
            "package_name": "python-torch",
            "package_version": "2.5.1-19",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/thrnciar/setuptools-78.1.1-control/python-torch",
            "git_hash": "ef637f77198d8f7095165215e6d30b3218938b95",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "thrnciar/setuptools-78.1.1-control--thrnciar",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/",
                    "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch",
                    "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch"
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/thrnciar/setuptools-78.1.1-control",
            "result_dir": "08966030-python-torch",
            "built_packages": "",
            "id": 8966030,
            "mockchain_macros": {
                "copr_username": "thrnciar",
                "copr_projectname": "setuptools-78.1.1-control",
                "vendor": "Fedora Project COPR (thrnciar/setuptools-78.1.1-control)"
            }
        }
    ]
}
[2025-04-25 07:14:31,513][  INFO][PID:872607] Sending fedora-messaging bus message in build.start
[2025-04-25 07:14:32,833][  INFO][PID:872607] Sending fedora-messaging bus message in chroot.start
[2025-04-25 07:14:32,854][  INFO][PID:872607] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8966030-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2025-04-25 07:14:33,221][  INFO][PID:872607] Downloading the builder-live.log file, attempt 1
[2025-04-25 07:14:33,223][  INFO][PID:872607] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.232.153.163 copr-rpmbuild-log
[2025-04-25 07:14:38,229][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 07:24:38,356][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 07:34:38,507][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 07:44:38,651][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 07:54:38,789][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 08:04:38,917][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 08:14:39,063][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 08:24:39,186][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 08:27:37,058][ ERROR][PID:872607] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-25 08:27:47,059][WARNING][PID:872607] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-25 08:27:57,060][  INFO][PID:872607] Downloading the builder-live.log file, attempt 2
[2025-04-25 08:27:57,062][  INFO][PID:872607] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.232.153.163 copr-rpmbuild-log
[2025-04-25 08:28:57,130][ ERROR][PID:872607] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-25 08:29:07,134][WARNING][PID:872607] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-25 08:29:17,135][  INFO][PID:872607] Downloading the builder-live.log file, attempt 3
[2025-04-25 08:29:17,136][  INFO][PID:872607] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.232.153.163 copr-rpmbuild-log
[2025-04-25 08:30:17,204][ ERROR][PID:872607] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-25 08:30:27,208][WARNING][PID:872607] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-25 08:30:37,209][  INFO][PID:872607] Downloading the builder-live.log file, attempt 4
[2025-04-25 08:30:37,210][  INFO][PID:872607] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@18.232.153.163 copr-rpmbuild-log
[2025-04-25 08:31:37,278][ ERROR][PID:872607] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-25 08:31:47,283][ ERROR][PID:872607] Giving up for unstable SSH, failures: 1745569667.0593936, 1745569747.1345584, 1745569827.2084184, 1745569907.2834816
[2025-04-25 08:31:47,284][ ERROR][PID:872607] Re-try request for task on 'ResallocHost, ticket_id=2657787, hostname=18.232.153.163, name=aws_x86_64_spot_prod_03317081_20250425_070628, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-04-25 08:31:47,285][  INFO][PID:872607] Releasing VM back to pool
[2025-04-25 08:31:47,294][  INFO][PID:872607] Retry #1 (on other host)
[2025-04-25 08:31:47,337][  INFO][PID:872607] Trying to allocate VM: ResallocHost, ticket_id=2659407, requested_tags=['copr_builder', 'arch_x86_64']
[2025-04-25 08:31:50,383][  INFO][PID:872607] Allocated host ResallocHost, ticket_id=2659407, hostname=34.229.89.185, name=aws_x86_64_normalreserved_prod_03319449_20250425_082924, requested_tags=['copr_builder', 'arch_x86_64']
[2025-04-25 08:31:50,384][  INFO][PID:872607] Allocating ssh connection to builder
[2025-04-25 08:31:50,384][  INFO][PID:872607] Checking that builder machine is OK
[2025-04-25 08:31:50,704][  INFO][PID:872607] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2025-04-25 08:31:50,909][  INFO][PID:872607] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2025-04-25 08:31:50,910][  INFO][PID:872607] Filling build.info file with builder info
[2025-04-25 08:31:50,912][  INFO][PID:872607] Not re-notifying FE and msg buses for the new host.
[2025-04-25 08:31:50,913][  INFO][PID:872607] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8966030-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2025-04-25 08:31:51,267][  INFO][PID:872607] Downloading the builder-live.log file, attempt 1
[2025-04-25 08:31:51,269][  INFO][PID:872607] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@34.229.89.185 copr-rpmbuild-log
[2025-04-25 08:34:41,297][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 08:44:41,402][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 08:54:41,486][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 09:04:41,573][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 09:14:41,718][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 09:24:41,845][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 09:34:41,958][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 09:44:42,065][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 09:54:42,146][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 10:04:42,295][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 10:14:42,476][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 10:24:42,562][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 10:34:42,641][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 10:44:42,732][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 10:54:42,831][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 11:04:42,952][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 11:14:43,161][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 11:24:43,352][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 11:34:43,583][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 11:44:43,700][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 11:54:43,833][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 12:04:43,958][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 12:14:44,172][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 12:24:44,262][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 12:34:44,385][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 12:44:44,605][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 12:54:44,701][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 13:04:44,821][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 13:14:45,024][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 13:24:45,141][  INFO][PID:872607] Periodic builder liveness probe: alive
[2025-04-25 13:31:58,366][  INFO][PID:872607] Downloading results from builder
[2025-04-25 13:31:58,367][  INFO][PID:872607] rsyncing of mockbuilder@34.229.89.185:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966030-python-torch started
[2025-04-25 13:31:58,368][  INFO][PID:872607] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@34.229.89.185:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966030-python-torch/ &> /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966030-python-torch/build-08966030.rsync.log
[2025-04-25 13:32:00,607][  INFO][PID:872607] rsyncing finished.
[2025-04-25 13:32:00,607][  INFO][PID:872607] Releasing VM back to pool
[2025-04-25 13:32:00,618][  INFO][PID:872607] Searching for 'success' file in resultdir
[2025-04-25 13:32:00,618][ ERROR][PID:872607] Build failed: Backend process error: No success file => build failure
[2025-04-25 13:32:00,619][  INFO][PID:872607] Finished build: id=8966030 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control chroot=fedora-rawhide-x86_64 
[2025-04-25 13:32:00,621][ ERROR][PID:872607] Backend process error: No success file => build failure
[2025-04-25 13:32:00,622][  INFO][PID:872607] Worker failed build, took 18009.709374904633
[2025-04-25 13:32:00,622][  INFO][PID:872607] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "thrnciar",
            "project_name": "setuptools-78.1.1-control",
            "project_dirname": "setuptools-78.1.1-control",
            "submitter": "thrnciar",
            "ended_on": 1745587920.622221,
            "started_on": 1745569910.912846,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8966030-fedora-rawhide-x86_64",
            "build_id": 8966030,
            "package_name": "python-torch",
            "package_version": "2.5.1-19",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/thrnciar/setuptools-78.1.1-control/python-torch",
            "git_hash": "ef637f77198d8f7095165215e6d30b3218938b95",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "thrnciar/setuptools-78.1.1-control--thrnciar",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/",
                    "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch",
                    "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch"
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/thrnciar/setuptools-78.1.1-control",
            "result_dir": "08966030-python-torch",
            "built_packages": "",
            "id": 8966030,
            "mockchain_macros": {
                "copr_username": "thrnciar",
                "copr_projectname": "setuptools-78.1.1-control",
                "vendor": "Fedora Project COPR (thrnciar/setuptools-78.1.1-control)"
            }
        }
    ]
}
[2025-04-25 13:32:00,647][  INFO][PID:872607] Sending fedora-messaging bus message in build.end
[2025-04-25 13:32:00,669][  INFO][PID:872607] Compressing /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966030-python-torch/builder-live.log by gzip
[2025-04-25 13:32:00,670][  INFO][PID:872607] Running command 'gzip /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966030-python-torch/builder-live.log' as PID 3555950
[2025-04-25 13:32:00,768][  INFO][PID:872607] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966030-python-torch/builder-live.log)
[2025-04-25 13:32:00,768][  INFO][PID:872607] Compressing /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966030-python-torch/backend.log by gzip
[2025-04-25 13:32:00,769][  INFO][PID:872607] Running command 'gzip /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966030-python-torch/backend.log' as PID 3555957