class Google::Apis::DataflowV1b3::Source
A source that records can be read and decoded from.
Attributes
While splitting, sources may specify the produced bundles as differences against another source, in order to save backend-side memory and allow bigger jobs. For details, see SourceSplitRequest
. To support this use case, the full set of parameters of the source is logically obtained by taking the latest explicitly specified value of each parameter in the order: base_specs
(later items win), spec (overrides anything in base_specs
). Corresponds to the JSON property `baseSpecs` @return [Array<Hash<String,Object>>]
The codec to use to decode data read from the source. Corresponds to the JSON property `codec` @return [Hash<String,Object>]
Setting this value to true hints to the framework that the source doesn't need splitting, and using SourceSplitRequest
on it would yield SOURCE_SPLIT_OUTCOME_USE_CURRENT. E.g. a file splitter may set this to true when splitting a single file into a set of byte ranges of appropriate size, and set this to false when splitting a filepattern into individual files. However, for efficiency, a file splitter may decide to produce file subranges directly from the filepattern to avoid a splitting round-trip. See SourceSplitRequest
for an overview of the splitting process. This field is meaningful only in the Source
objects populated by the user (e.g. when filling in a DerivedSource
). Source
objects supplied by the framework to the user don' t have this field populated. Corresponds to the JSON property `doesNotNeedSplitting` @return [Boolean]
Setting this value to true hints to the framework that the source doesn't need splitting, and using SourceSplitRequest
on it would yield SOURCE_SPLIT_OUTCOME_USE_CURRENT. E.g. a file splitter may set this to true when splitting a single file into a set of byte ranges of appropriate size, and set this to false when splitting a filepattern into individual files. However, for efficiency, a file splitter may decide to produce file subranges directly from the filepattern to avoid a splitting round-trip. See SourceSplitRequest
for an overview of the splitting process. This field is meaningful only in the Source
objects populated by the user (e.g. when filling in a DerivedSource
). Source
objects supplied by the framework to the user don' t have this field populated. Corresponds to the JSON property `doesNotNeedSplitting` @return [Boolean]
Metadata about a Source
useful for automatically optimizing and tuning the pipeline, etc. Corresponds to the JSON property `metadata` @return [Google::Apis::DataflowV1b3::SourceMetadata]
The source to read from, plus its parameters. Corresponds to the JSON property `spec` @return [Hash<String,Object>]
Public Class Methods
# File lib/google/apis/dataflow_v1b3/classes.rb, line 4511 def initialize(**args) update!(**args) end
Public Instance Methods
Update properties of this object
# File lib/google/apis/dataflow_v1b3/classes.rb, line 4516 def update!(**args) @base_specs = args[:base_specs] if args.key?(:base_specs) @codec = args[:codec] if args.key?(:codec) @does_not_need_splitting = args[:does_not_need_splitting] if args.key?(:does_not_need_splitting) @metadata = args[:metadata] if args.key?(:metadata) @spec = args[:spec] if args.key?(:spec) end