[2025-03-25 17:31:18,952][  INFO][PID:1962955] Marking build as starting
[2025-03-25 17:31:19,145][  INFO][PID:1962955] Trying to allocate VM
[2025-03-25 17:31:47,436][  INFO][PID:1962955] Allocated host ResallocHost, ticket_id=2035947, hostname=140.211.168.14, name=copr_osuosl_p09_prod_02315461_20250325_170931
[2025-03-25 17:31:47,440][  INFO][PID:1962955] Allocating ssh connection to builder
[2025-03-25 17:31:47,442][  INFO][PID:1962955] Checking that builder machine is OK
[2025-03-25 17:31:48,802][  INFO][PID:1962955] Running remote command: copr-builder-ready epel-7-ppc64le
[2025-03-25 17:31:49,890][  INFO][PID:1962955] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2025-03-25 17:31:49,894][  INFO][PID:1962955] Filling build.info file with builder info
[2025-03-25 17:31:49,902][  INFO][PID:1962955] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@copr",
            "project_name": "copr",
            "project_dirname": "copr",
            "submitter": "praiskup",
            "ended_on": null,
            "started_on": 1742923909.9017253,
            "submitted_on": null,
            "status": 3,
            "chroot": "epel-7-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8819721-epel-7-ppc64le",
            "build_id": 8819721,
            "package_name": "python-copr",
            "package_version": "2.1-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/copr/python-copr",
            "git_hash": "4154ba4e67020be53ed5a34bf0641bc5012c7d5f",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@copr/copr--praiskup",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/copr/epel-7-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@copr/copr",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/copr",
            "result_dir": "08819721-python-copr",
            "built_packages": "",
            "id": 8819721,
            "mockchain_macros": {
                "copr_username": "@copr",
                "copr_projectname": "copr",
                "vendor": "Fedora Project COPR (@copr/copr)"
            }
        }
    ]
}
[2025-03-25 17:31:49,977][  INFO][PID:1962955] Sending fedora-messaging bus message in build.start
[2025-03-25 17:31:52,874][  INFO][PID:1962955] Sending fedora-messaging bus message in chroot.start
[2025-03-25 17:31:52,901][  INFO][PID:1962955] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8819721-epel-7-ppc64le --chroot epel-7-ppc64le --detached
[2025-03-25 17:31:54,336][  INFO][PID:1962955] Downloading the builder-live.log file, attempt 1
[2025-03-25 17:31:54,354][  INFO][PID:1962955] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log
[2025-03-25 17:31:59,349][  INFO][PID:1962955] Periodic builder liveness probe: alive
[2025-03-25 17:41:59,646][  INFO][PID:1962955] Periodic builder liveness probe: alive
[2025-03-25 17:52:00,006][  INFO][PID:1962955] Periodic builder liveness probe: alive
[2025-03-25 18:02:00,139][  INFO][PID:1962955] Periodic builder liveness probe: alive
[2025-03-25 18:11:56,230][ ERROR][PID:1962955] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-25 18:12:00,350][  INFO][PID:1962955] Periodic builder liveness probe: dead
[2025-03-25 18:12:06,236][WARNING][PID:1962955] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-25 18:12:16,238][  INFO][PID:1962955] Downloading the builder-live.log file, attempt 2
[2025-03-25 18:12:16,247][  INFO][PID:1962955] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log
[2025-03-25 18:13:00,434][ ERROR][PID:1962955] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.14 port 22: Connection timed out

[2025-03-25 18:13:16,339][ ERROR][PID:1962955] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-25 18:13:26,346][WARNING][PID:1962955] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-25 18:13:36,347][  INFO][PID:1962955] Downloading the builder-live.log file, attempt 3
[2025-03-25 18:13:36,352][  INFO][PID:1962955] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log
[2025-03-25 18:14:10,535][ ERROR][PID:1962955] SSH connection lost on #2 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.14 port 22: Connection timed out

[2025-03-25 18:14:36,428][ ERROR][PID:1962955] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-25 18:14:46,435][WARNING][PID:1962955] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-03-25 18:14:56,436][  INFO][PID:1962955] Downloading the builder-live.log file, attempt 4
[2025-03-25 18:14:56,438][  INFO][PID:1962955] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@140.211.168.14 copr-rpmbuild-log
[2025-03-25 18:15:20,645][ ERROR][PID:1962955] SSH connection lost on #3 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.14 port 22: Connection timed out

[2025-03-25 18:15:56,511][ ERROR][PID:1962955] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-03-25 18:16:06,520][ ERROR][PID:1962955] Giving up for unstable SSH, failures: 1742926326.2366405, 1742926406.346943, 1742926486.4354732, 1742926566.5201073
[2025-03-25 18:16:30,759][ ERROR][PID:1962955] SSH connection lost on #4 attempt, let's retry after 10s, Connection broke:
OUT:

ERR:
ssh: connect to host 140.211.168.14 port 22: Connection timed out

[2025-03-25 18:16:40,765][ ERROR][PID:1962955] Can't ssh to cancel build.
[2025-03-25 18:16:40,769][ ERROR][PID:1962955] Re-try request for task on 'ResallocHost, ticket_id=2035947, hostname=140.211.168.14, name=copr_osuosl_p09_prod_02315461_20250325_170931': SSH problems when downloading live log: Resalloc ticket FAILED
[2025-03-25 18:16:40,773][  INFO][PID:1962955] Releasing VM back to pool
[2025-03-25 18:16:40,799][  INFO][PID:1962955] Retry #1 (on other host)
[2025-03-25 18:16:40,808][  INFO][PID:1962955] Trying to allocate VM
[2025-03-25 18:16:43,884][  INFO][PID:1962955] Allocated host ResallocHost, ticket_id=2037199, hostname=2620:52:3:1:dead:beef:cafe:c210, name=copr_hv_ppc64le_01_prod_02317773_20250325_181106
[2025-03-25 18:16:43,885][  INFO][PID:1962955] Allocating ssh connection to builder
[2025-03-25 18:16:43,886][  INFO][PID:1962955] Checking that builder machine is OK
[2025-03-25 18:16:44,888][  INFO][PID:1962955] Running remote command: copr-builder-ready epel-7-ppc64le
[2025-03-25 18:16:45,505][  INFO][PID:1962955] Checking Red Hat subscription (attempt #1)
Red Hat subscription active
Builder is ready to be used

[2025-03-25 18:16:45,505][  INFO][PID:1962955] Filling build.info file with builder info
[2025-03-25 18:16:45,507][  INFO][PID:1962955] Not re-notifying FE and msg buses for the new host.
[2025-03-25 18:16:45,507][  INFO][PID:1962955] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8819721-epel-7-ppc64le --chroot epel-7-ppc64le --detached
[2025-03-25 18:16:46,512][  INFO][PID:1962955] Downloading the builder-live.log file, attempt 1
[2025-03-25 18:16:46,529][  INFO][PID:1962955] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@2620:52:3:1:dead:beef:cafe:c210 copr-rpmbuild-log
[2025-03-25 18:16:55,381][  INFO][PID:1962955] Cancel request succeeded
out:
success
err:
Warning: Permanently added '2620:52:3:1:dead:beef:cafe:c210' (ED25519) to the list of known hosts.

[2025-03-25 18:16:55,382][  INFO][PID:1962955] Downloading results from builder
[2025-03-25 18:16:55,382][  INFO][PID:1962955] rsyncing of mockbuilder@[2620:52:3:1:dead:beef:cafe:c210]:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/@copr/copr/epel-7-ppc64le/08819721-python-copr started
[2025-03-25 18:16:55,384][  INFO][PID:1962955] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@[2620:52:3:1:dead:beef:cafe:c210]:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/@copr/copr/epel-7-ppc64le/08819721-python-copr/ &> /var/lib/copr/public_html/results/@copr/copr/epel-7-ppc64le/08819721-python-copr/build-08819721.rsync.log
[2025-03-25 18:16:55,998][  INFO][PID:1962955] rsyncing finished.
[2025-03-25 18:16:56,000][  INFO][PID:1962955] Releasing VM back to pool
[2025-03-25 18:16:56,032][  INFO][PID:1962955] Searching for 'success' file in resultdir
[2025-03-25 18:16:56,034][ ERROR][PID:1962955] Build failed: Backend process error: No success file => build failure
[2025-03-25 18:16:56,039][  INFO][PID:1962955] Finished build: id=8819721 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/@copr/copr chroot=epel-7-ppc64le 
[2025-03-25 18:16:56,049][ ERROR][PID:1962955] Backend process error: No success file => build failure
[2025-03-25 18:16:56,050][  INFO][PID:1962955] Worker failed build, took 10.54284143447876
[2025-03-25 18:16:56,053][  INFO][PID:1962955] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "@copr",
            "project_name": "copr",
            "project_dirname": "copr",
            "submitter": "praiskup",
            "ended_on": 1742926616.0500095,
            "started_on": 1742926605.507168,
            "submitted_on": null,
            "status": 0,
            "chroot": "epel-7-ppc64le",
            "arch": "ppc64le",
            "buildroot_pkgs": [],
            "task_id": "8819721-epel-7-ppc64le",
            "build_id": 8819721,
            "package_name": "python-copr",
            "package_version": "2.1-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/@copr/copr/python-copr",
            "git_hash": "4154ba4e67020be53ed5a34bf0641bc5012c7d5f",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "@copr/copr--praiskup",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": null,
            "background": false,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/@copr/copr/epel-7-ppc64le/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                }
            ],
            "tags": [
                "arch_ppc64le"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/@copr/copr",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/@copr/copr",
            "result_dir": "08819721-python-copr",
            "built_packages": "",
            "id": 8819721,
            "mockchain_macros": {
                "copr_username": "@copr",
                "copr_projectname": "copr",
                "vendor": "Fedora Project COPR (@copr/copr)"
            }
        }
    ]
}
[2025-03-25 18:16:56,122][  INFO][PID:1962955] Sending fedora-messaging bus message in build.end
[2025-03-25 18:16:56,177][  INFO][PID:1962955] Compressing /var/lib/copr/public_html/results/@copr/copr/epel-7-ppc64le/08819721-python-copr/builder-live.log by gzip
[2025-03-25 18:16:56,182][  INFO][PID:1962955] Running command 'gzip /var/lib/copr/public_html/results/@copr/copr/epel-7-ppc64le/08819721-python-copr/builder-live.log' as PID 2363826
[2025-03-25 18:16:56,195][  INFO][PID:1962955] Finished after 0 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/@copr/copr/epel-7-ppc64le/08819721-python-copr/builder-live.log)
[2025-03-25 18:16:56,202][  INFO][PID:1962955] Compressing /var/lib/copr/public_html/results/@copr/copr/epel-7-ppc64le/08819721-python-copr/backend.log by gzip
[2025-03-25 18:16:56,206][  INFO][PID:1962955] Running command 'gzip /var/lib/copr/public_html/results/@copr/copr/epel-7-ppc64le/08819721-python-copr/backend.log' as PID 2363829