[2025-01-20 12:13:07,333][  INFO][PID:5660] Marking build as starting
[2025-01-20 12:13:07,423][  INFO][PID:5660] Trying to allocate VM
[2025-01-20 12:15:25,711][  INFO][PID:5660] Allocated host ResallocHost, ticket_id=828670, hostname=2620:52:3:1:dead:beef:cafe:c296, name=copr_p09_01_prod_00712294_20250120_115602
[2025-01-20 12:15:25,712][  INFO][PID:5660] Allocating ssh connection to builder
[2025-01-20 12:15:25,712][  INFO][PID:5660] Checking that builder machine is OK
[2025-01-20 12:15:26,751][  INFO][PID:5660] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-01-20 12:15:27,341][  INFO][PID:5660] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-01-20 12:15:27,344][  INFO][PID:5660] Filling build.info file with builder info
[2025-01-20 12:15:27,346][  INFO][PID:5660] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "jplesnik",
            "project_name": "perl-test",
            "project_dirname": "perl-test",
            "submitter": "jplesnik",
            "ended_on": null,
            "started_on": 1737375327.346271,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8550597-fedora-rawhide-ppc64le",
            "build_id": 8550597,
            "package_name": "os-autoinst",
            "package_version": "4.6^20241125gitb64e219-2",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/jplesnik/perl-test/os-autoinst",
            "git_hash": "28982b36757800caaa52e8dd06d9680d3bd717e9",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "jplesnik/perl-test--jplesnik",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/jplesnik/perl-test/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/jplesnik/perl-test",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/jplesnik/perl-test",
            "result_dir": "08550597-os-autoinst",
            "built_packages": "",
            "id": 8550597,
            "mockchain_macros": {
                "copr_username": "jplesnik",
                "copr_projectname": "perl-test",
                "vendor": "Fedora Project COPR (jplesnik/perl-test)"
            }
        }
    ]
}
[2025-01-20 12:15:27,391][  INFO][PID:5660] Sending fedora-messaging bus message in build.start
[2025-01-20 12:15:28,452][  INFO][PID:5660] Sending fedora-messaging bus message in chroot.start
[2025-01-20 12:15:28,489][  INFO][PID:5660] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8550597-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-01-20 12:15:29,614][  INFO][PID:5660] Downloading the builder-live.log file, attempt 1
[2025-01-20 12:15:29,618][  INFO][PID:5660] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c296 copr-rpmbuild-log
[2025-01-20 12:15:34,635][  INFO][PID:5660] Periodic builder liveness probe: alive
[2025-01-20 12:20:08,229][ ERROR][PID:5660] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-01-20 12:20:18,230][WARNING][PID:5660] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-01-20 12:20:28,231][  INFO][PID:5660] Downloading the builder-live.log file, attempt 2
[2025-01-20 12:20:28,233][  INFO][PID:5660] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c296 copr-rpmbuild-log
[2025-01-20 12:21:28,306][ ERROR][PID:5660] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-01-20 12:21:38,307][WARNING][PID:5660] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-01-20 12:21:48,308][  INFO][PID:5660] Downloading the builder-live.log file, attempt 3
[2025-01-20 12:21:48,310][  INFO][PID:5660] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c296 copr-rpmbuild-log
[2025-01-20 12:22:48,378][ ERROR][PID:5660] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-01-20 12:22:58,389][WARNING][PID:5660] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-01-20 12:23:08,390][  INFO][PID:5660] Downloading the builder-live.log file, attempt 4
[2025-01-20 12:23:08,391][  INFO][PID:5660] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c296 copr-rpmbuild-log
[2025-01-20 12:24:08,459][ ERROR][PID:5660] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-01-20 12:24:18,460][ ERROR][PID:5660] Giving up for unstable SSH, failures: 1737375618.230725, 1737375698.3076954, 1737375778.3892646, 1737375858.4606616
[2025-01-20 12:24:18,461][ ERROR][PID:5660] Re-try request for task on 'ResallocHost, ticket_id=828670, hostname=2620:52:3:1:dead:beef:cafe:c296, name=copr_p09_01_prod_00712294_20250120_115602': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-01-20 12:24:18,462][  INFO][PID:5660] Releasing VM back to pool
[2025-01-20 12:24:18,472][  INFO][PID:5660] Retry #1 (on other host)
[2025-01-20 12:24:18,473][  INFO][PID:5660] Trying to allocate VM
[2025-01-20 12:24:21,502][  INFO][PID:5660] Allocated host ResallocHost, ticket_id=829011, hostname=2620:52:3:1:dead:beef:cafe:c2a4, name=copr_p09_01_prod_00712771_20250120_121711
[2025-01-20 12:24:21,503][  INFO][PID:5660] Allocating ssh connection to builder
[2025-01-20 12:24:21,503][  INFO][PID:5660] Checking that builder machine is OK
[2025-01-20 12:24:21,930][  INFO][PID:5660] Running remote command: copr-builder-ready fedora-rawhide-ppc64le
[2025-01-20 12:24:22,505][  INFO][PID:5660] Red Hat subscription not needed for fedora-rawhide-ppc64le
Builder is ready to be used

[2025-01-20 12:24:22,506][  INFO][PID:5660] Filling build.info file with builder info
[2025-01-20 12:24:22,510][  INFO][PID:5660] Not re-notifying FE and msg buses for the new host.
[2025-01-20 12:24:22,511][  INFO][PID:5660] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8550597-fedora-rawhide-ppc64le --chroot fedora-rawhide-ppc64le --detached
[2025-01-20 12:24:23,591][  INFO][PID:5660] Downloading the builder-live.log file, attempt 1
[2025-01-20 12:24:23,593][  INFO][PID:5660] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c2a4 copr-rpmbuild-log
[2025-01-20 12:25:38,615][  INFO][PID:5660] Periodic builder liveness probe: alive
[2025-01-20 12:29:58,461][  INFO][PID:5660] Downloading results from builder
[2025-01-20 12:29:58,462][  INFO][PID:5660] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a4]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/jplesnik/perl-test/fedora-rawhide-ppc64le/08550597-os-autoinst started
[2025-01-20 12:29:58,463][  INFO][PID:5660] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c2a4]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/jplesnik/perl-test/fedora-rawhide-ppc64le/08550597-os-autoinst/ &> /var/lib/copr/public_html/results/jplesnik/perl-test/fedora-rawhide-ppc64le/08550597-os-autoinst/build-08550597.rsync.log
[2025-01-20 12:30:01,120][  INFO][PID:5660] rsyncing finished.
[2025-01-20 12:30:01,120][  INFO][PID:5660] Releasing VM back to pool
[2025-01-20 12:30:01,136][  INFO][PID:5660] Searching for 'success' file in resultdir
[2025-01-20 12:30:01,136][ ERROR][PID:5660] Build failed: Backend process error: No success file => build failure
[2025-01-20 12:30:01,137][  INFO][PID:5660] Finished build: id=8550597 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/jplesnik/perl-test chroot=fedora-rawhide-ppc64le 
[2025-01-20 12:30:01,139][ ERROR][PID:5660] Backend process error: No success file => build failure
[2025-01-20 12:30:01,140][  INFO][PID:5660] Worker failed build, took 338.6297564506531
[2025-01-20 12:30:01,141][  INFO][PID:5660] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "jplesnik",
            "project_name": "perl-test",
            "project_dirname": "perl-test",
            "submitter": "jplesnik",
            "ended_on": 1737376201.1402576,
            "started_on": 1737375862.5105011,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8550597-fedora-rawhide-ppc64le",
            "build_id": 8550597,
            "package_name": "os-autoinst",
            "package_version": "4.6^20241125gitb64e219-2",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/jplesnik/perl-test/os-autoinst",
            "git_hash": "28982b36757800caaa52e8dd06d9680d3bd717e9",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "jplesnik/perl-test--jplesnik",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/jplesnik/perl-test/fedora-rawhide-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/jplesnik/perl-test",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/jplesnik/perl-test",
            "result_dir": "08550597-os-autoinst",
            "built_packages": "",
            "id": 8550597,
            "mockchain_macros": {
                "copr_username": "jplesnik",
                "copr_projectname": "perl-test",
                "vendor": "Fedora Project COPR (jplesnik/perl-test)"
            }
        }
    ]
}
[2025-01-20 12:30:01,180][  INFO][PID:5660] Sending fedora-messaging bus message in build.end
[2025-01-20 12:30:01,198][  INFO][PID:5660] Compressing /var/lib/copr/public_html/results/jplesnik/perl-test/fedora-rawhide-ppc64le/08550597-os-autoinst/builder-live.log by gzip
[2025-01-20 12:30:01,200][  INFO][PID:5660] Running command 'gzip /var/lib/copr/public_html/results/jplesnik/perl-test/fedora-rawhide-ppc64le/08550597-os-autoinst/builder-live.log' as PID 175660
[2025-01-20 12:30:01,232][  INFO][PID:5660] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/jplesnik/perl-test/fedora-rawhide-ppc64le/08550597-os-autoinst/builder-live.log)
[2025-01-20 12:30:01,233][  INFO][PID:5660] Compressing /var/lib/copr/public_html/results/jplesnik/perl-test/fedora-rawhide-ppc64le/08550597-os-autoinst/backend.log by gzip
[2025-01-20 12:30:01,235][  INFO][PID:5660] Running command 'gzip /var/lib/copr/public_html/results/jplesnik/perl-test/fedora-rawhide-ppc64le/08550597-os-autoinst/backend.log' as PID 175676