[2025-06-07 02:29:33,464][  INFO][PID:2314508] Marking build as starting
[2025-06-07 02:29:33,571][  INFO][PID:2314508] Trying to allocate VM: ResallocHost, ticket_id=3394710, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-06-07 02:29:36,624][  INFO][PID:2314508] Allocated host ResallocHost, ticket_id=3394710, hostname=2620:52:3:1:dead:beef:cafe:c24c, name=copr_hv_ppc64le_02_prod_04659478_20250607_012334, requested_tags=['copr_builder', 'arch_ppc64le']
[2025-06-07 02:29:36,627][  INFO][PID:2314508] Allocating ssh connection to builder
[2025-06-07 02:29:36,629][  INFO][PID:2314508] Checking that builder machine is OK
[2025-06-07 02:29:37,427][  INFO][PID:2314508] Running remote command: copr-builder-ready fedora-41-ppc64le
[2025-06-07 02:29:37,893][  INFO][PID:2314508] Red Hat subscription not needed for fedora-41-ppc64le
Builder is ready to be used

[2025-06-07 02:29:37,895][  INFO][PID:2314508] Filling build.info file with builder info
[2025-06-07 02:29:37,899][  INFO][PID:2314508] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "@fedora-llvm-team",
            "project_name": "llvm-snapshots-big-merge-20250607",
            "project_dirname": "llvm-snapshots-big-merge-20250607",
            "submitter": "kkleine",
            "ended_on": null,
            "started_on": 1749263377.8990357,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-41-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9140378-fedora-41-ppc64le",
            "build_id": 9140378,
            "package_name": "llvm",
            "package_version": "20.1.6-7",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@fedora-llvm-team/llvm-snapshots-big-merge-20250607/llvm",
            "git_hash": "5dc1a1759191ff1d594be6cfd3798487ffd410c6",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@fedora-llvm-team/llvm-snapshots-big-merge-20250607--kkleine",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607/fedora-41-ppc64le/",
                    "id": "copr_base",
                    "module_hotfixes": true,
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-test-suite/fedora-41-ppc64le/",
                    "id": "copr_fedora_llvm_team_llvm_test_suite",
                    "name": "Additional repo copr_fedora_llvm_team_llvm_test_suite"
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [
                "snapshot_build"
            ],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607",
            "result_dir": "09140378-llvm",
            "built_packages": "",
            "id": 9140378,
            "mockchain_macros": {
                "copr_username": "@fedora-llvm-team",
                "copr_projectname": "llvm-snapshots-big-merge-20250607",
                "vendor": "Fedora Project COPR (@fedora-llvm-team/llvm-snapshots-big-merge-20250607)"
            }
        }
    ]
}
[2025-06-07 02:29:37,925][  INFO][PID:2314508] Sending fedora-messaging bus message in build.start
[2025-06-07 02:29:38,623][  INFO][PID:2314508] Sending fedora-messaging bus message in chroot.start
[2025-06-07 02:29:38,644][  INFO][PID:2314508] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/9140378-fedora-41-ppc64le --chroot fedora-41-ppc64le --detached
[2025-06-07 02:29:39,401][  INFO][PID:2314508] Downloading the builder-live.log file, attempt 1
[2025-06-07 02:29:39,404][  INFO][PID:2314508] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c24c copr-rpmbuild-log
[2025-06-07 02:29:44,413][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 02:39:44,597][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 02:49:44,769][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 02:59:44,924][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 03:09:45,113][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 03:19:45,303][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 03:29:45,494][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 03:39:45,685][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 03:49:45,911][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 03:59:46,217][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 04:09:46,568][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 04:19:47,003][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 04:29:47,184][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 04:39:47,377][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 04:49:47,552][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 04:59:47,743][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 05:09:48,003][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 05:19:48,204][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 05:29:48,489][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 05:39:48,678][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 05:49:48,854][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 05:59:49,024][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 06:09:49,200][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 06:19:49,390][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 06:29:49,552][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 06:39:49,707][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 06:49:49,947][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 06:59:50,226][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 07:09:50,531][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 07:19:50,775][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 07:29:51,109][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 07:39:51,383][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 07:49:51,564][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 07:59:51,740][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 08:09:51,931][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 08:19:52,130][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 08:29:52,306][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 08:39:52,467][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 08:49:52,634][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 08:59:52,787][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 09:09:52,951][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 09:19:53,104][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 09:29:53,266][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 09:39:53,424][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 09:49:53,587][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 09:59:53,748][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 10:09:53,932][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 10:19:54,104][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 10:29:54,524][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 10:39:55,175][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 10:49:55,358][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 10:59:55,537][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 11:09:55,776][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 11:19:55,955][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 11:29:56,182][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 11:39:56,420][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 11:49:56,568][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 11:59:56,734][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 12:09:56,928][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 12:19:57,115][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 12:29:57,311][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 12:39:57,463][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 12:49:57,756][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 12:59:57,986][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 13:09:58,324][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 13:19:58,609][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 13:29:58,925][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 13:39:59,158][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 13:49:59,384][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 13:59:59,597][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 14:09:59,865][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 14:20:00,064][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 14:30:00,316][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 14:40:00,553][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 14:50:00,761][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 15:00:00,989][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 15:10:01,438][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 15:20:01,710][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 15:30:02,036][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 15:40:02,343][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 15:50:02,560][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 16:00:02,884][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 16:10:03,242][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 16:20:03,553][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 16:30:03,724][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 16:40:03,967][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 16:50:04,194][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 17:00:04,409][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 17:10:04,752][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 17:20:05,007][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 17:30:05,225][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 17:40:05,415][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 17:50:05,644][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 18:00:05,847][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 18:10:06,120][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 18:20:06,370][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 18:30:06,530][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 18:40:06,686][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 18:50:06,903][  INFO][PID:2314508] Periodic builder liveness probe: alive
[2025-06-07 18:59:09,052][  INFO][PID:2314508] Downloading results from builder
[2025-06-07 18:59:09,087][  INFO][PID:2314508] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c24c]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607/fedora-41-ppc64le/09140378-llvm started
[2025-06-07 18:59:09,121][  INFO][PID:2314508] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c24c]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607/fedora-41-ppc64le/09140378-llvm/ &> /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607/fedora-41-ppc64le/09140378-llvm/build-09140378.rsync.log
[2025-06-07 18:59:43,760][  INFO][PID:2314508] rsyncing finished.
[2025-06-07 18:59:43,763][  INFO][PID:2314508] Releasing VM back to pool
[2025-06-07 18:59:43,786][  INFO][PID:2314508] Searching for 'success' file in resultdir
[2025-06-07 18:59:43,797][ ERROR][PID:2314508] Build failed: Backend process error: No success file => build failure
[2025-06-07 18:59:43,922][  INFO][PID:2314508] Finished build: id=9140378 failed=True timeout=108000 destdir=/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607 chroot=fedora-41-ppc64le 
[2025-06-07 18:59:44,044][ ERROR][PID:2314508] Backend process error: No success file => build failure
[2025-06-07 18:59:44,047][  INFO][PID:2314508] Worker failed build, took 59406.14850068092
[2025-06-07 18:59:44,066][  INFO][PID:2314508] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 108000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": true,
            "project_owner": "@fedora-llvm-team",
            "project_name": "llvm-snapshots-big-merge-20250607",
            "project_dirname": "llvm-snapshots-big-merge-20250607",
            "submitter": "kkleine",
            "ended_on": 1749322784.0475364,
            "started_on": 1749263377.8990357,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-41-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "9140378-fedora-41-ppc64le",
            "build_id": 9140378,
            "package_name": "llvm",
            "package_version": "20.1.6-7",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@fedora-llvm-team/llvm-snapshots-big-merge-20250607/llvm",
            "git_hash": "5dc1a1759191ff1d594be6cfd3798487ffd410c6",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@fedora-llvm-team/llvm-snapshots-big-merge-20250607--kkleine",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607/fedora-41-ppc64le/",
                    "id": "copr_base",
                    "module_hotfixes": true,
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-test-suite/fedora-41-ppc64le/",
                    "id": "copr_fedora_llvm_team_llvm_test_suite",
                    "name": "Additional repo copr_fedora_llvm_team_llvm_test_suite"
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [
                "snapshot_build"
            ],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607",
            "result_dir": "09140378-llvm",
            "built_packages": "",
            "id": 9140378,
            "mockchain_macros": {
                "copr_username": "@fedora-llvm-team",
                "copr_projectname": "llvm-snapshots-big-merge-20250607",
                "vendor": "Fedora Project COPR (@fedora-llvm-team/llvm-snapshots-big-merge-20250607)"
            }
        }
    ]
}
[2025-06-07 18:59:44,742][  INFO][PID:2314508] Sending fedora-messaging bus message in build.end
[2025-06-07 18:59:45,155][  INFO][PID:2314508] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607/fedora-41-ppc64le/09140378-llvm/builder-live.log by gzip
[2025-06-07 18:59:45,195][  INFO][PID:2314508] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607/fedora-41-ppc64le/09140378-llvm/builder-live.log' as PID 2652085
[2025-06-07 18:59:45,807][  INFO][PID:2314508] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607/fedora-41-ppc64le/09140378-llvm/builder-live.log)
[2025-06-07 18:59:45,820][  INFO][PID:2314508] Compressing /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607/fedora-41-ppc64le/09140378-llvm/backend.log by gzip
[2025-06-07 18:59:45,823][  INFO][PID:2314508] Running command 'gzip /var/lib/copr/public_html/results/@fedora-llvm-team/llvm-snapshots-big-merge-20250607/fedora-41-ppc64le/09140378-llvm/backend.log' as PID 2652146