[2025-03-26 09:09:58,610][  INFO][PID:6740] Marking build as starting
[2025-03-26 09:09:58,851][  INFO][PID:6740] Trying to allocate VM: ResallocHost, ticket_id=2050169, requested_tags=['copr_builder', 'arch_x86_64']
[2025-03-26 09:12:17,169][  INFO][PID:6740] Allocated host ResallocHost, ticket_id=2050169, hostname=35.172.129.18, name=aws_x86_64_spot_prod_02334626_20250326_091013, requested_tags=['copr_builder', 'arch_x86_64']
[2025-03-26 09:12:17,169][  INFO][PID:6740] Allocating ssh connection to builder
[2025-03-26 09:12:17,169][  INFO][PID:6740] Checking that builder machine is OK
[2025-03-26 09:12:17,528][  INFO][PID:6740] Running remote command: copr-builder-ready fedora-40-x86_64
[2025-03-26 09:12:17,743][  INFO][PID:6740] Red Hat subscription not needed for fedora-40-x86_64
Builder is ready to be used

[2025-03-26 09:12:17,744][  INFO][PID:6740] Filling build.info file with builder info
[2025-03-26 09:12:17,745][  INFO][PID:6740] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 172800,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "rezso",
            "project_name": "ML",
            "project_dirname": "ML",
            "submitter": "rezso",
            "ended_on": null,
            "started_on": 1742980337.7453127,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-40-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8821742-fedora-40-x86_64",
            "build_id": 8821742,
            "package_name": "pytorch",
            "package_version": "2.8.0-20250326.0.gita8d0c5c9.cu12_8",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rezso/ML/pytorch",
            "git_hash": "7925cc9130ba3ea0243915b10b4a21fb6e14254d",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "rezso/ML--rezso",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/rezso/ML/fedora-40-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/rezso/CUDA/fedora-40-x86_64/",
                    "id": "copr_rezso_CUDA",
                    "name": "Additional repo copr_rezso_CUDA"
                },
                {
                    "baseurl": "http://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64",
                    "id": "http_developer_download_nvidia_com_compute_cuda_repos_rhel9_x86_64",
                    "name": "Additional repo http_developer_download_nvidia_com_compute_cuda_repos_rhel9_x86_64"
                },
                {
                    "baseurl": "http://developer.download.nvidia.com/compute/cuda/repos/rhel9/sbsa",
                    "id": "http_developer_download_nvidia_com_compute_cuda_repos_rhel9_sbsa",
                    "name": "Additional repo http_developer_download_nvidia_com_compute_cuda_repos_rhel9_sbsa"
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/rezso/ML",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rezso/ML",
            "result_dir": "08821742-pytorch",
            "built_packages": "",
            "id": 8821742,
            "mockchain_macros": {
                "copr_username": "rezso",
                "copr_projectname": "ML",
                "vendor": "Fedora Project COPR (rezso/ML)"
            }
        }
    ]
}
[2025-03-26 09:14:55,955][  INFO][PID:6740] Sending fedora-messaging bus message in build.start
[2025-03-26 09:15:03,369][  INFO][PID:6740] Sending fedora-messaging bus message in chroot.start
[2025-03-26 09:15:03,496][  INFO][PID:6740] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8821742-fedora-40-x86_64 --chroot fedora-40-x86_64 --detached
[2025-03-26 09:15:04,292][  INFO][PID:6740] Downloading the builder-live.log file, attempt 1
[2025-03-26 09:15:04,307][  INFO][PID:6740] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.172.129.18 copr-rpmbuild-log
[2025-03-26 09:15:09,389][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 09:25:09,688][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 09:35:09,810][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 09:45:09,876][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 09:55:09,947][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 10:05:10,023][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 10:15:10,116][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 10:25:10,290][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 10:35:10,407][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 10:45:10,488][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 10:55:10,625][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 11:05:10,814][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 11:15:10,981][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 11:25:11,063][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 11:35:11,150][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 11:45:11,240][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 11:55:11,333][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 12:05:11,410][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 12:15:11,537][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 12:25:11,618][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 12:35:11,738][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 12:45:11,822][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 12:55:11,901][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 13:05:11,979][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 13:15:12,115][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 13:25:12,223][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 13:35:12,303][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 13:35:13,555][ ERROR][PID:6740] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-26 13:35:23,566][WARNING][PID:6740] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-26 13:35:33,573][  INFO][PID:6740] Downloading the builder-live.log file, attempt 2
[2025-03-26 13:35:33,612][  INFO][PID:6740] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.172.129.18 copr-rpmbuild-log
[2025-03-26 13:35:33,616][ ERROR][PID:6740] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-26 13:35:43,621][WARNING][PID:6740] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-26 13:35:53,621][  INFO][PID:6740] Downloading the builder-live.log file, attempt 3
[2025-03-26 13:35:53,624][  INFO][PID:6740] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.172.129.18 copr-rpmbuild-log
[2025-03-26 13:35:53,633][ ERROR][PID:6740] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-26 13:36:03,638][WARNING][PID:6740] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-26 13:36:13,638][  INFO][PID:6740] Downloading the builder-live.log file, attempt 4
[2025-03-26 13:36:13,640][  INFO][PID:6740] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@35.172.129.18 copr-rpmbuild-log
[2025-03-26 13:37:13,709][ ERROR][PID:6740] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-26 13:37:23,732][ ERROR][PID:6740] Giving up for unstable SSH, failures: 1742996123.563754, 1742996143.6209586, 1742996163.6381662, 1742996243.7127035
[2025-03-26 13:37:23,755][ ERROR][PID:6740] Re-try request for task on 'ResallocHost, ticket_id=2050169, hostname=35.172.129.18, name=aws_x86_64_spot_prod_02334626_20250326_091013, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-03-26 13:37:23,758][  INFO][PID:6740] Releasing VM back to pool
[2025-03-26 13:37:23,772][  INFO][PID:6740] Retry #1 (on other host)
[2025-03-26 13:37:23,857][  INFO][PID:6740] Trying to allocate VM: ResallocHost, ticket_id=2054180, requested_tags=['copr_builder', 'arch_x86_64']
[2025-03-26 13:37:26,973][  INFO][PID:6740] Allocated host ResallocHost, ticket_id=2054180, hostname=13.217.54.223, name=aws_x86_64_normalreserved_prod_02341594_20250326_133257, requested_tags=['copr_builder', 'arch_x86_64']
[2025-03-26 13:37:26,975][  INFO][PID:6740] Allocating ssh connection to builder
[2025-03-26 13:37:26,980][  INFO][PID:6740] Checking that builder machine is OK
[2025-03-26 13:37:27,318][  INFO][PID:6740] Running remote command: copr-builder-ready fedora-40-x86_64
[2025-03-26 13:37:27,515][  INFO][PID:6740] Red Hat subscription not needed for fedora-40-x86_64
Builder is ready to be used

[2025-03-26 13:37:27,521][  INFO][PID:6740] Filling build.info file with builder info
[2025-03-26 13:37:27,534][  INFO][PID:6740] Not re-notifying FE and msg buses for the new host.
[2025-03-26 13:37:27,540][  INFO][PID:6740] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8821742-fedora-40-x86_64 --chroot fedora-40-x86_64 --detached
[2025-03-26 13:37:27,895][  INFO][PID:6740] Downloading the builder-live.log file, attempt 1
[2025-03-26 13:37:27,897][  INFO][PID:6740] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@13.217.54.223 copr-rpmbuild-log
[2025-03-26 13:45:12,955][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 13:55:13,037][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 14:05:13,132][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 14:15:13,229][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 14:25:13,344][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 14:35:13,522][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 14:45:13,625][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 14:55:13,712][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 15:05:13,791][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 15:15:13,907][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 15:25:14,000][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 15:35:14,080][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 15:45:14,157][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 15:55:14,241][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 16:05:14,325][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 16:15:14,440][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 16:25:14,606][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 16:35:14,689][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 16:45:14,769][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 16:55:14,894][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 17:05:14,983][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 17:15:15,137][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 17:25:15,247][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 17:35:15,341][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 17:45:15,441][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 17:55:15,526][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 18:05:15,616][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 18:15:15,722][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 18:25:15,799][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 18:35:15,875][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 18:45:15,943][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 18:55:16,058][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 19:05:16,138][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 19:15:16,265][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 19:25:16,346][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 19:35:16,427][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 19:45:16,500][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 19:55:16,575][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 20:05:16,647][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 20:15:16,777][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 20:25:16,870][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 20:35:16,949][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 20:45:17,032][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 20:55:17,113][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 21:05:17,221][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 21:15:17,323][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 21:25:17,424][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 21:35:17,507][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 21:45:17,580][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 21:55:17,662][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 22:05:17,740][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 22:15:17,865][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 22:25:17,952][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 22:35:18,028][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 22:45:18,120][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 22:55:18,207][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 23:05:18,284][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 23:15:18,489][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 23:25:18,587][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 23:35:18,667][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 23:45:18,743][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-26 23:55:18,813][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 00:05:18,896][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 00:15:18,982][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 00:25:19,051][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 00:35:19,122][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 00:45:19,199][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 00:55:19,274][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 01:05:19,358][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 01:15:19,483][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 01:25:19,586][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 01:35:19,672][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 01:45:19,781][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 01:55:19,893][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 02:05:19,966][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 02:15:20,065][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 02:25:20,151][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 02:35:20,218][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 02:45:20,288][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 02:55:20,364][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 03:05:20,436][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 03:15:20,959][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 03:25:21,046][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 03:35:21,122][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 03:45:21,209][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 03:55:21,279][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 04:05:21,371][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 04:15:21,702][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 04:25:21,853][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 04:35:21,927][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 04:45:21,999][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 04:55:22,073][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 05:05:22,157][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 05:15:22,243][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 05:25:22,316][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 05:35:22,391][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 05:45:22,467][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 05:55:22,543][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 06:05:22,613][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 06:15:22,710][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 06:25:22,831][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 06:35:22,903][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 06:45:23,060][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 06:55:23,166][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 07:05:23,250][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 07:15:23,371][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 07:25:23,463][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 07:35:23,549][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 07:45:23,729][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 07:55:23,887][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 08:05:23,981][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 08:15:24,070][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 08:25:24,147][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 08:35:24,220][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 08:45:24,309][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 08:55:24,396][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 09:05:24,475][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 09:15:24,605][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 09:25:24,684][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 09:35:24,767][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 09:45:24,884][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 09:55:24,966][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 10:05:25,043][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 10:15:25,143][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 10:25:25,224][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 10:35:25,297][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 10:45:25,378][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 10:55:25,470][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 11:05:25,554][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 11:15:25,674][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 11:25:25,764][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 11:35:25,847][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 11:45:25,931][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 11:55:26,015][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 12:05:26,110][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 12:15:26,319][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 12:25:26,447][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 12:35:26,529][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 12:45:26,607][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 12:55:26,693][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 13:05:26,772][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 13:15:26,885][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 13:25:26,967][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 13:35:27,053][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 13:45:27,145][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 13:55:27,248][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 14:05:27,333][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 14:15:27,445][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 14:25:27,622][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 14:35:27,723][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 14:45:27,817][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 14:55:27,917][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 15:05:28,037][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 15:15:28,157][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 15:25:28,242][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 15:35:28,316][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 15:45:28,392][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 15:55:28,470][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 16:05:28,539][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 16:15:28,679][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 16:25:28,792][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 16:35:28,866][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 16:45:28,947][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 16:55:29,030][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 17:05:29,110][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 17:15:29,204][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 17:25:29,299][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 17:35:29,377][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 17:45:29,452][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 17:55:29,525][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 18:05:29,599][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 18:15:29,707][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 18:25:29,812][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 18:35:29,893][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 18:45:29,971][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 18:55:30,047][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 19:05:30,123][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 19:15:30,245][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 19:25:30,337][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 19:35:30,417][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 19:45:30,502][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 19:55:30,633][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 20:05:30,723][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 20:15:30,827][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 20:25:30,904][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 20:35:31,006][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 20:45:31,103][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 20:55:31,234][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 21:05:31,363][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 21:15:31,476][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 21:25:31,564][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 21:35:31,661][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 21:45:31,734][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 21:55:31,811][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 22:05:31,893][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 22:15:32,051][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 22:25:32,171][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 22:35:32,248][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 22:45:32,323][  INFO][PID:6740] Periodic builder liveness probe: alive
[2025-03-27 22:50:39,635][  INFO][PID:6740] Downloading results from builder
[2025-03-27 22:50:39,665][  INFO][PID:6740] rsyncing of mockbuilder@13.217.54.223:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch started
[2025-03-27 22:50:39,709][  INFO][PID:6740] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@13.217.54.223:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/ &> /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/build-08821742.rsync.log
[2025-03-27 22:50:41,903][  INFO][PID:6740] rsyncing finished.
[2025-03-27 22:50:41,907][  INFO][PID:6740] Releasing VM back to pool
[2025-03-27 22:50:41,925][  INFO][PID:6740] Searching for 'success' file in resultdir
[2025-03-27 22:50:41,943][  INFO][PID:6740] Going to sign pkgs from source: 8821742-fedora-40-x86_64 in chroot: /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64
[2025-03-27 22:50:41,962][  INFO][PID:6740] Calling '/bin/sign -u rezso#ML@copr.fedorahosted.org -p' (attempt #1)
[2025-03-27 22:50:42,300][  INFO][PID:6740] Calling '/bin/sign -4 -h sha256 -u rezso#ML@copr.fedorahosted.org -r /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/pytorch-2.8.0-20250326.0.gita8d0c5c9.cu12_8.fc40.src.rpm' (attempt #1)
[2025-03-27 22:50:43,136][  INFO][PID:6740] signed rpm: /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/pytorch-2.8.0-20250326.0.gita8d0c5c9.cu12_8.fc40.src.rpm
[2025-03-27 22:50:43,137][  INFO][PID:6740] Calling '/bin/sign -4 -h sha256 -u rezso#ML@copr.fedorahosted.org -r /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/pytorch-2.8.0-20250326.0.gita8d0c5c9.cu12_8.fc40.x86_64.rpm' (attempt #1)
[2025-03-27 22:50:52,082][  INFO][PID:6740] signed rpm: /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/pytorch-2.8.0-20250326.0.gita8d0c5c9.cu12_8.fc40.x86_64.rpm
[2025-03-27 22:50:52,084][  INFO][PID:6740] Calling '/bin/sign -4 -h sha256 -u rezso#ML@copr.fedorahosted.org -r /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/pytorch-python3-2.8.0-20250326.0.gita8d0c5c9.cu12_8.fc40.x86_64.rpm' (attempt #1)
[2025-03-27 22:50:53,012][  INFO][PID:6740] signed rpm: /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/pytorch-python3-2.8.0-20250326.0.gita8d0c5c9.cu12_8.fc40.x86_64.rpm
[2025-03-27 22:50:53,013][  INFO][PID:6740] Calling '/bin/sign -4 -h sha256 -u rezso#ML@copr.fedorahosted.org -r /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/pytorch-devel-2.8.0-20250326.0.gita8d0c5c9.cu12_8.fc40.x86_64.rpm' (attempt #1)
[2025-03-27 22:50:53,653][  INFO][PID:6740] signed rpm: /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/pytorch-devel-2.8.0-20250326.0.gita8d0c5c9.cu12_8.fc40.x86_64.rpm
[2025-03-27 22:50:53,654][  INFO][PID:6740] Sign done
[2025-03-27 22:50:53,661][  INFO][PID:6740] Incremental createrepo run, adding 08821742-pytorch into https://download.copr.fedorainfracloud.org/results/rezso/ML/fedora-40-x86_64, (auto-create-repo=True)
[2025-03-27 22:50:53,708][  INFO][PID:6740] Running command 'copr-repo --batched /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64 --add 08821742-pytorch --no-appstream-metadata' as PID 1017930
[2025-03-27 22:50:55,442][  INFO][PID:6740] Finished after 1 seconds with exit code 0 (copr-repo --batched /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64 --add 08821742-pytorch --no-appstream-metadata)
[2025-03-27 22:50:55,462][  INFO][PID:6740] Getting build details
[2025-03-27 22:50:55,464][  INFO][PID:6740] Listing built binary packages in /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch
[2025-03-27 22:50:55,466][  INFO][PID:6740] Built packages:
['pytorch 2.8.0', 'pytorch-python3 2.8.0', 'pytorch-devel 2.8.0']
[2025-03-27 22:50:55,562][  INFO][PID:6740] build details: {'built_packages': 'pytorch 2.8.0\npytorch-python3 2.8.0\npytorch-devel 2.8.0'}
[2025-03-27 22:50:55,566][  INFO][PID:6740] Retrieving pubkey
[2025-03-27 22:50:55,567][  INFO][PID:6740] Calling '/bin/sign -u rezso#ML@copr.fedorahosted.org -p' (attempt #1)
[2025-03-27 22:50:55,881][  INFO][PID:6740] Added pubkey for user rezso project ML into: /var/lib/copr/public_html/results/rezso/ML/pubkey.gpg
[2025-03-27 22:50:55,889][  INFO][PID:6740] Finished build: id=8821742 failed=False timeout=172800 destdir=/var/lib/copr/public_html/results/rezso/ML chroot=fedora-40-x86_64 
[2025-03-27 22:50:55,997][  INFO][PID:6740] Worker succeeded build, took 119608.4656534195
[2025-03-27 22:50:56,010][  INFO][PID:6740] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 172800,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "rezso",
            "project_name": "ML",
            "project_dirname": "ML",
            "submitter": "rezso",
            "ended_on": 1743115855.9967377,
            "started_on": 1742996247.5310843,
            "submitted_on": null,
            "status": 1,
            "chroot": "fedora-40-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8821742-fedora-40-x86_64",
            "build_id": 8821742,
            "package_name": "pytorch",
            "package_version": "2.8.0-20250326.0.gita8d0c5c9.cu12_8",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/rezso/ML/pytorch",
            "git_hash": "7925cc9130ba3ea0243915b10b4a21fb6e14254d",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "rezso/ML--rezso",
            "results": {
                "packages": [
                    {
                        "name": "pytorch",
                        "epoch": null,
                        "version": "2.8.0",
                        "release": "20250326.0.gita8d0c5c9.cu12_8.fc40",
                        "arch": "x86_64"
                    },
                    {
                        "name": "pytorch-python3",
                        "epoch": null,
                        "version": "2.8.0",
                        "release": "20250326.0.gita8d0c5c9.cu12_8.fc40",
                        "arch": "x86_64"
                    },
                    {
                        "name": "pytorch-devel",
                        "epoch": null,
                        "version": "2.8.0",
                        "release": "20250326.0.gita8d0c5c9.cu12_8.fc40",
                        "arch": "x86_64"
                    },
                    {
                        "name": "pytorch",
                        "epoch": null,
                        "version": "2.8.0",
                        "release": "20250326.0.gita8d0c5c9.cu12_8.fc40",
                        "arch": "src"
                    }
                ]
            },
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/rezso/ML/fedora-40-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/rezso/CUDA/fedora-40-x86_64/",
                    "id": "copr_rezso_CUDA",
                    "name": "Additional repo copr_rezso_CUDA"
                },
                {
                    "baseurl": "http://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64",
                    "id": "http_developer_download_nvidia_com_compute_cuda_repos_rhel9_x86_64",
                    "name": "Additional repo http_developer_download_nvidia_com_compute_cuda_repos_rhel9_x86_64"
                },
                {
                    "baseurl": "http://developer.download.nvidia.com/compute/cuda/repos/rhel9/sbsa",
                    "id": "http_developer_download_nvidia_com_compute_cuda_repos_rhel9_sbsa",
                    "name": "Additional repo http_developer_download_nvidia_com_compute_cuda_repos_rhel9_sbsa"
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/rezso/ML",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/rezso/ML",
            "result_dir": "08821742-pytorch",
            "built_packages": "pytorch 2.8.0\npytorch-python3 2.8.0\npytorch-devel 2.8.0",
            "id": 8821742,
            "mockchain_macros": {
                "copr_username": "rezso",
                "copr_projectname": "ML",
                "vendor": "Fedora Project COPR (rezso/ML)"
            }
        }
    ]
}
[2025-03-27 22:50:56,724][  INFO][PID:6740] Sending fedora-messaging bus message in build.end
[2025-03-27 22:50:57,155][  INFO][PID:6740] Compressing /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/builder-live.log by gzip
[2025-03-27 22:50:57,156][  INFO][PID:6740] Running command 'gzip /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/builder-live.log' as PID 1018031
[2025-03-27 22:50:57,225][  INFO][PID:6740] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/builder-live.log)
[2025-03-27 22:50:57,227][  INFO][PID:6740] Compressing /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/backend.log by gzip
[2025-03-27 22:50:57,228][  INFO][PID:6740] Running command 'gzip /var/lib/copr/public_html/results/rezso/ML/fedora-40-x86_64/08821742-pytorch/backend.log' as PID 1018033