[2025-04-25 07:17:36,782][  INFO][PID:901823] Marking build as starting
[2025-04-25 07:17:36,850][  INFO][PID:901823] Trying to allocate VM: ResallocHost, ticket_id=2657898, requested_tags=['copr_builder', 'arch_x86_64']
[2025-04-25 07:17:39,899][  INFO][PID:901823] Allocated host ResallocHost, ticket_id=2657898, hostname=44.201.229.78, name=aws_x86_64_spot_prod_03317068_20250425_070624, requested_tags=['copr_builder', 'arch_x86_64']
[2025-04-25 07:17:39,900][  INFO][PID:901823] Allocating ssh connection to builder
[2025-04-25 07:17:39,900][  INFO][PID:901823] Checking that builder machine is OK
[2025-04-25 07:17:40,095][  INFO][PID:901823] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2025-04-25 07:17:40,302][  INFO][PID:901823] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2025-04-25 07:17:40,303][  INFO][PID:901823] Filling build.info file with builder info
[2025-04-25 07:17:40,304][  INFO][PID:901823] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "thrnciar",
            "project_name": "setuptools-78.1.1-control",
            "project_dirname": "setuptools-78.1.1-control",
            "submitter": "thrnciar",
            "ended_on": null,
            "started_on": 1745565460.3044229,
            "submitted_on": null,
            "status": 3,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8966069-fedora-rawhide-x86_64",
            "build_id": 8966069,
            "package_name": "webkitgtk",
            "package_version": "2.49.1-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/thrnciar/setuptools-78.1.1-control/webkitgtk",
            "git_hash": "ced5489dc75a90833bb261e71fd0be72c664d9a6",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "thrnciar/setuptools-78.1.1-control--thrnciar",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/",
                    "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch",
                    "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch"
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/thrnciar/setuptools-78.1.1-control",
            "result_dir": "08966069-webkitgtk",
            "built_packages": "",
            "id": 8966069,
            "mockchain_macros": {
                "copr_username": "thrnciar",
                "copr_projectname": "setuptools-78.1.1-control",
                "vendor": "Fedora Project COPR (thrnciar/setuptools-78.1.1-control)"
            }
        }
    ]
}
[2025-04-25 07:17:40,330][  INFO][PID:901823] Sending fedora-messaging bus message in build.start
[2025-04-25 07:17:41,329][  INFO][PID:901823] Sending fedora-messaging bus message in chroot.start
[2025-04-25 07:17:41,351][  INFO][PID:901823] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8966069-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2025-04-25 07:17:41,734][  INFO][PID:901823] Downloading the builder-live.log file, attempt 1
[2025-04-25 07:17:41,735][  INFO][PID:901823] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.229.78 copr-rpmbuild-log
[2025-04-25 07:17:46,743][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 07:27:46,857][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 07:37:46,992][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 07:47:47,136][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 07:57:47,257][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 08:07:47,374][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 08:17:47,582][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 08:27:47,680][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 08:37:47,774][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 08:47:47,867][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 08:57:47,954][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 09:07:48,057][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 09:17:48,189][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 09:27:48,297][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 09:37:48,425][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 09:47:48,533][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 09:57:48,623][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 10:07:48,724][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 10:17:48,909][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 10:27:48,991][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 10:37:49,076][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 10:47:49,176][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 10:57:49,270][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 11:07:49,451][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 11:17:49,674][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 11:27:49,839][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 11:37:50,067][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 11:42:36,385][ ERROR][PID:901823] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-25 11:42:46,386][WARNING][PID:901823] SSH connection lost on #1 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-25 11:42:56,387][  INFO][PID:901823] Downloading the builder-live.log file, attempt 2
[2025-04-25 11:42:56,407][  INFO][PID:901823] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.229.78 copr-rpmbuild-log
[2025-04-25 11:43:56,476][ ERROR][PID:901823] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-25 11:44:06,482][WARNING][PID:901823] SSH connection lost on #2 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-25 11:44:16,483][  INFO][PID:901823] Downloading the builder-live.log file, attempt 3
[2025-04-25 11:44:16,484][  INFO][PID:901823] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.229.78 copr-rpmbuild-log
[2025-04-25 11:45:16,551][ ERROR][PID:901823] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-25 11:45:26,555][WARNING][PID:901823] SSH connection lost on #3 attempt, let's retry after 10s, Unable to finish after 1 SSH attempts
[2025-04-25 11:45:36,556][  INFO][PID:901823] Downloading the builder-live.log file, attempt 4
[2025-04-25 11:45:36,558][  INFO][PID:901823] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@44.201.229.78 copr-rpmbuild-log
[2025-04-25 11:46:36,626][ ERROR][PID:901823] SSH connection lost on #1 attempt, let's retry after 10s, Connection broke.
[2025-04-25 11:46:46,631][ ERROR][PID:901823] Giving up for unstable SSH, failures: 1745581366.3862543, 1745581446.4823098, 1745581526.5559258, 1745581606.631496
[2025-04-25 11:46:46,632][ ERROR][PID:901823] Re-try request for task on 'ResallocHost, ticket_id=2657898, hostname=44.201.229.78, name=aws_x86_64_spot_prod_03317068_20250425_070624, requested_tags=['copr_builder', 'arch_x86_64']': SSH problems when downloading live log: Stopped following builder for broken SSH: Unable to finish after 1 SSH attempts
[2025-04-25 11:46:46,633][  INFO][PID:901823] Releasing VM back to pool
[2025-04-25 11:46:46,645][  INFO][PID:901823] Retry #1 (on other host)
[2025-04-25 11:46:46,659][  INFO][PID:901823] Trying to allocate VM: ResallocHost, ticket_id=2661904, requested_tags=['copr_builder', 'arch_x86_64']
[2025-04-25 11:46:49,671][  INFO][PID:901823] Allocated host ResallocHost, ticket_id=2661904, hostname=54.160.174.14, name=aws_x86_64_normalreserved_prod_03324924_20250425_113945, requested_tags=['copr_builder', 'arch_x86_64']
[2025-04-25 11:46:49,672][  INFO][PID:901823] Allocating ssh connection to builder
[2025-04-25 11:46:49,672][  INFO][PID:901823] Checking that builder machine is OK
[2025-04-25 11:46:50,025][  INFO][PID:901823] Running remote command: copr-builder-ready fedora-rawhide-x86_64
[2025-04-25 11:46:50,238][  INFO][PID:901823] Red Hat subscription not needed for fedora-rawhide-x86_64
Builder is ready to be used

[2025-04-25 11:46:50,239][  INFO][PID:901823] Filling build.info file with builder info
[2025-04-25 11:46:50,244][  INFO][PID:901823] Not re-notifying FE and msg buses for the new host.
[2025-04-25 11:46:50,244][  INFO][PID:901823] Starting remote build: copr-rpmbuild --verbose --drop-resultdir --task-url https://copr.fedorainfracloud.org/backend/get-build-task/8966069-fedora-rawhide-x86_64 --chroot fedora-rawhide-x86_64 --detached
[2025-04-25 11:46:50,616][  INFO][PID:901823] Downloading the builder-live.log file, attempt 1
[2025-04-25 11:46:50,617][  INFO][PID:901823] Popen command started: ssh -F /home/copr/.ssh/config mockbuilder@54.160.174.14 copr-rpmbuild-log
[2025-04-25 11:47:50,632][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 11:57:50,780][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 12:07:50,926][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 12:17:51,100][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 12:27:51,199][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 12:37:51,382][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 12:47:51,567][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 12:57:51,679][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 13:07:51,800][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 13:17:51,991][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 13:27:52,108][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 13:37:52,261][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 13:47:52,436][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 13:57:52,572][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 14:07:52,717][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 14:17:52,911][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 14:27:53,063][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 14:37:53,186][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 14:47:53,338][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 14:57:53,473][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 15:07:53,653][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 15:17:53,885][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 15:27:54,074][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 15:37:54,225][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 15:47:54,353][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 15:57:54,493][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 16:07:54,662][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 16:17:54,978][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 16:27:55,143][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 16:37:55,234][  INFO][PID:901823] Periodic builder liveness probe: alive
[2025-04-25 16:46:57,703][  INFO][PID:901823] Downloading results from builder
[2025-04-25 16:46:57,735][  INFO][PID:901823] rsyncing of mockbuilder@54.160.174.14:/var/lib/copr-rpmbuild/results/ to /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966069-webkitgtk started
[2025-04-25 16:46:57,771][  INFO][PID:901823] Popen command started: /usr/bin/rsync -rltDvH --chmod=D755,F644 -e 'ssh -F /home/copr/.ssh/config' mockbuilder@54.160.174.14:/var/lib/copr-rpmbuild/results/ /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966069-webkitgtk/ &> /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966069-webkitgtk/build-08966069.rsync.log
[2025-04-25 16:46:58,366][  INFO][PID:901823] rsyncing finished.
[2025-04-25 16:46:58,369][  INFO][PID:901823] Releasing VM back to pool
[2025-04-25 16:46:58,390][  INFO][PID:901823] Searching for 'success' file in resultdir
[2025-04-25 16:46:58,395][ ERROR][PID:901823] Build failed: Backend process error: No success file => build failure
[2025-04-25 16:46:58,509][  INFO][PID:901823] Finished build: id=8966069 failed=True timeout=18000 destdir=/var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control chroot=fedora-rawhide-x86_64 
[2025-04-25 16:46:58,621][ ERROR][PID:901823] Backend process error: No success file => build failure
[2025-04-25 16:46:58,624][  INFO][PID:901823] Worker failed build, took 18008.379289388657
[2025-04-25 16:46:58,639][  INFO][PID:901823] Sending build state back to frontend: {
    "builds": [
        {
            "timeout": 18000,
            "frontend_base_url": "https://copr.fedorainfracloud.org",
            "memory_reqs": 2048,
            "enable_net": false,
            "project_owner": "thrnciar",
            "project_name": "setuptools-78.1.1-control",
            "project_dirname": "setuptools-78.1.1-control",
            "submitter": "thrnciar",
            "ended_on": 1745599618.6236873,
            "started_on": 1745581610.2443979,
            "submitted_on": null,
            "status": 0,
            "chroot": "fedora-rawhide-x86_64",
            "arch": "x86_64",
            "buildroot_pkgs": [],
            "task_id": "8966069-fedora-rawhide-x86_64",
            "build_id": 8966069,
            "package_name": "webkitgtk",
            "package_version": "2.49.1-1",
            "git_repo": "https://copr-dist-git.fedorainfracloud.org/git/thrnciar/setuptools-78.1.1-control/webkitgtk",
            "git_hash": "ced5489dc75a90833bb261e71fd0be72c664d9a6",
            "git_branch": null,
            "source_type": null,
            "source_json": null,
            "pkg_name": null,
            "pkg_main_version": null,
            "pkg_epoch": null,
            "pkg_release": null,
            "srpm_url": null,
            "uses_devel_repo": false,
            "sandbox": "thrnciar/setuptools-78.1.1-control--thrnciar",
            "results": null,
            "appstream": false,
            "allow_user_ssh": false,
            "ssh_public_keys": null,
            "storage": 0,
            "background": true,
            "fedora_review": false,
            "isolation": "default",
            "repo_priority": null,
            "repos": [
                {
                    "baseurl": "https://download.copr.fedorainfracloud.org/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/",
                    "id": "copr_base",
                    "name": "Copr repository",
                    "priority": null
                },
                {
                    "baseurl": "http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/",
                    "id": "http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch",
                    "name": "Additional repo http_kojipkgs_fedoraproject_org_repos_rawhide_latest_basearch"
                }
            ],
            "tags": [
                "arch_x86_64"
            ],
            "with_opts": [],
            "without_opts": [],
            "destdir": "/var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control",
            "results_repo_url": "https://download.copr.fedorainfracloud.org/results/thrnciar/setuptools-78.1.1-control",
            "result_dir": "08966069-webkitgtk",
            "built_packages": "",
            "id": 8966069,
            "mockchain_macros": {
                "copr_username": "thrnciar",
                "copr_projectname": "setuptools-78.1.1-control",
                "vendor": "Fedora Project COPR (thrnciar/setuptools-78.1.1-control)"
            }
        }
    ]
}
[2025-04-25 16:46:59,301][  INFO][PID:901823] Sending fedora-messaging bus message in build.end
[2025-04-25 16:46:59,717][  INFO][PID:901823] Compressing /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966069-webkitgtk/builder-live.log by gzip
[2025-04-25 16:46:59,756][  INFO][PID:901823] Running command 'gzip /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966069-webkitgtk/builder-live.log' as PID 867726
[2025-04-25 16:47:00,756][  INFO][PID:901823] Finished after 1 seconds with exit code 0 (gzip /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966069-webkitgtk/builder-live.log)
[2025-04-25 16:47:00,769][  INFO][PID:901823] Compressing /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966069-webkitgtk/backend.log by gzip
[2025-04-25 16:47:00,771][  INFO][PID:901823] Running command 'gzip /var/lib/copr/public_html/results/thrnciar/setuptools-78.1.1-control/fedora-rawhide-x86_64/08966069-webkitgtk/backend.log' as PID 867782