class Google::Apis::DataflowV1b3::WorkerPool

Describes one particular pool of Cloud Dataflow workers to be instantiated by the Cloud Dataflow service in order to perform the computations required by a job. Note that a workflow job may use multiple pools, in order to match the various computational requirements of the various stages of the job.

Attributes

autoscaling_settings[RW]

Settings for WorkerPool autoscaling. Corresponds to the JSON property `autoscalingSettings` @return [Google::Apis::DataflowV1b3::AutoscalingSettings]

data_disks[RW]

Data disks that are used by a VM in this workflow. Corresponds to the JSON property `dataDisks` @return [Array<Google::Apis::DataflowV1b3::Disk>]

default_package_set[RW]

The default package set to install. This allows the service to select a default set of packages which are useful to worker harnesses written in a particular language. Corresponds to the JSON property `defaultPackageSet` @return [String]

disk_size_gb[RW]

Size of root disk for VMs, in GB. If zero or unspecified, the service will attempt to choose a reasonable default. Corresponds to the JSON property `diskSizeGb` @return [Fixnum]

disk_source_image[RW]

Fully qualified source image for disks. Corresponds to the JSON property `diskSourceImage` @return [String]

disk_type[RW]

Type of root disk for VMs. If empty or unspecified, the service will attempt to choose a reasonable default. Corresponds to the JSON property `diskType` @return [String]

ip_configuration[RW]

Configuration for VM IPs. Corresponds to the JSON property `ipConfiguration` @return [String]

kind[RW]

The kind of the worker pool; currently only `harness` and `shuffle` are supported. Corresponds to the JSON property `kind` @return [String]

machine_type[RW]

Machine type (e.g. “n1-standard-1”). If empty or unspecified, the service will attempt to choose a reasonable default. Corresponds to the JSON property `machineType` @return [String]

metadata[RW]

Metadata to set on the Google Compute Engine VMs. Corresponds to the JSON property `metadata` @return [Hash<String,String>]

network[RW]

Network to which VMs will be assigned. If empty or unspecified, the service will use the network “default”. Corresponds to the JSON property `network` @return [String]

num_threads_per_worker[RW]

The number of threads per worker harness. If empty or unspecified, the service will choose a number of threads (according to the number of cores on the selected machine type for batch, or 1 by convention for streaming). Corresponds to the JSON property `numThreadsPerWorker` @return [Fixnum]

num_workers[RW]

Number of Google Compute Engine workers in this pool needed to execute the job. If zero or unspecified, the service will attempt to choose a reasonable default. Corresponds to the JSON property `numWorkers` @return [Fixnum]

on_host_maintenance[RW]

The action to take on host maintenance, as defined by the Google Compute Engine API. Corresponds to the JSON property `onHostMaintenance` @return [String]

packages[RW]

Packages to be installed on workers. Corresponds to the JSON property `packages` @return [Array<Google::Apis::DataflowV1b3::Package>]

pool_args[RW]

Extra arguments for this worker pool. Corresponds to the JSON property `poolArgs` @return [Hash<String,Object>]

sdk_harness_container_images[RW]

Set of SDK harness containers needed to execute this pipeline. This will only be set in the Fn API path. For non-cross-language pipelines this should have only one entry. Cross-language pipelines will have two or more entries. Corresponds to the JSON property `sdkHarnessContainerImages` @return [Array<Google::Apis::DataflowV1b3::SdkHarnessContainerImage>]

subnetwork[RW]

Subnetwork to which VMs will be assigned, if desired. Expected to be of the form “regions/REGION/subnetworks/SUBNETWORK”. Corresponds to the JSON property `subnetwork` @return [String]

taskrunner_settings[RW]

Taskrunner configuration settings. Corresponds to the JSON property `taskrunnerSettings` @return [Google::Apis::DataflowV1b3::TaskRunnerSettings]

teardown_policy[RW]

Sets the policy for determining when to turndown worker pool. Allowed values are: `TEARDOWN_ALWAYS`, `TEARDOWN_ON_SUCCESS`, and `TEARDOWN_NEVER`. ` TEARDOWN_ALWAYS` means workers are always torn down regardless of whether the job succeeds. `TEARDOWN_ON_SUCCESS` means workers are torn down if the job succeeds. `TEARDOWN_NEVER` means the workers are never torn down. If the workers are not torn down by the service, they will continue to run and use Google Compute Engine VM resources in the user's project until they are explicitly terminated by the user. Because of this, Google recommends using the `TEARDOWN_ALWAYS` policy except for small, manually supervised test jobs. If unknown or unspecified, the service will attempt to choose a reasonable default. Corresponds to the JSON property `teardownPolicy` @return [String]

worker_harness_container_image[RW]

Required. Docker container image that executes the Cloud Dataflow worker harness, residing in Google Container Registry. Deprecated for the Fn API path. Use sdk_harness_container_images instead. Corresponds to the JSON property `workerHarnessContainerImage` @return [String]

zone[RW]

Zone to run the worker pools in. If empty or unspecified, the service will attempt to choose a reasonable default. Corresponds to the JSON property `zone` @return [String]

Public Class Methods

new(**args) click to toggle source
# File lib/google/apis/dataflow_v1b3/classes.rb, line 6603
def initialize(**args)
   update!(**args)
end

Public Instance Methods

update!(**args) click to toggle source

Update properties of this object

# File lib/google/apis/dataflow_v1b3/classes.rb, line 6608
def update!(**args)
  @autoscaling_settings = args[:autoscaling_settings] if args.key?(:autoscaling_settings)
  @data_disks = args[:data_disks] if args.key?(:data_disks)
  @default_package_set = args[:default_package_set] if args.key?(:default_package_set)
  @disk_size_gb = args[:disk_size_gb] if args.key?(:disk_size_gb)
  @disk_source_image = args[:disk_source_image] if args.key?(:disk_source_image)
  @disk_type = args[:disk_type] if args.key?(:disk_type)
  @ip_configuration = args[:ip_configuration] if args.key?(:ip_configuration)
  @kind = args[:kind] if args.key?(:kind)
  @machine_type = args[:machine_type] if args.key?(:machine_type)
  @metadata = args[:metadata] if args.key?(:metadata)
  @network = args[:network] if args.key?(:network)
  @num_threads_per_worker = args[:num_threads_per_worker] if args.key?(:num_threads_per_worker)
  @num_workers = args[:num_workers] if args.key?(:num_workers)
  @on_host_maintenance = args[:on_host_maintenance] if args.key?(:on_host_maintenance)
  @packages = args[:packages] if args.key?(:packages)
  @pool_args = args[:pool_args] if args.key?(:pool_args)
  @sdk_harness_container_images = args[:sdk_harness_container_images] if args.key?(:sdk_harness_container_images)
  @subnetwork = args[:subnetwork] if args.key?(:subnetwork)
  @taskrunner_settings = args[:taskrunner_settings] if args.key?(:taskrunner_settings)
  @teardown_policy = args[:teardown_policy] if args.key?(:teardown_policy)
  @worker_harness_container_image = args[:worker_harness_container_image] if args.key?(:worker_harness_container_image)
  @zone = args[:zone] if args.key?(:zone)
end