class Shrine::Storage::S3
Constants
- COPY_OPTIONS
- MAX_MULTIPART_PARTS
- MIN_PART_SIZE
- MULTIPART_THRESHOLD
Attributes
Public Class Methods
Initializes a storage for uploading to S3
. All options are forwarded to [‘Aws::S3::Client#initialize`], except the following:
:bucket : (Required). Name of the S3
bucket.
:client : By default an ‘Aws::S3::Client` instance is created internally from
additional options, but you can use this option to provide your own client. This can be an `Aws::S3::Client` or an `Aws::S3::Encryption::Client` object.
:prefix : “Directory” inside the bucket to store files into.
:upload_options : Additional options that will be used for uploading files, they will
be passed to [`Aws::S3::Object#put`], [`Aws::S3::Object#copy_from`] and [`Aws::S3::Bucket#presigned_post`].
:copy_options : Additional options that will be used for copying files, they will
be passed to [`Aws::S3::Object#copy_from`].
:multipart_threshold : If the input file is larger than the specified size, a parallelized
multipart will be used for the upload/copy. Defaults to `{upload: 15*1024*1024, copy: 100*1024*1024}` (15MB for upload requests, 100MB for copy requests).
:max_multipart_parts : Limits the number of parts if parellized multipart upload/copy is used. Defaults to 10_000.
In addition to specifying the ‘:bucket`, you’ll also need to provide AWS credentials. The most common way is to provide them directly via ‘:access_key_id`, `:secret_access_key`, and `:region` options. But you can also use any other way of authentication specified in the [AWS SDK documentation][configuring AWS SDK].
[‘Aws::S3::Object#put`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#put-instance_method [`Aws::S3::Object#copy_from`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#copy_from-instance_method [`Aws::S3::Bucket#presigned_post`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_post-instance_method [`Aws::S3::Client#initialize`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Client.html#initialize-instance_method [configuring AWS SDK]: docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html
# File lib/shrine/storage/s3.rb, line 71 def initialize(bucket:, client: nil, prefix: nil, upload_options: {}, multipart_threshold: {}, max_multipart_parts: nil, signer: nil, public: nil, copy_options: COPY_OPTIONS, **s3_options) raise ArgumentError, "the :bucket option is nil" unless bucket @client = client || Aws::S3::Client.new(**s3_options) @bucket = Aws::S3::Bucket.new(name: bucket, client: @client) @prefix = prefix @upload_options = upload_options @copy_options = copy_options @multipart_threshold = MULTIPART_THRESHOLD.merge(multipart_threshold) @max_multipart_parts = max_multipart_parts || MAX_MULTIPART_PARTS @signer = signer @public = public end
Public Instance Methods
If block is given, deletes all objects from the storage for which the block evaluates to true. Otherwise deletes all objects from the storage.
s3.clear! # or s3.clear! { |object| object.last_modified < Time.now - 7*24*60*60 }
# File lib/shrine/storage/s3.rb, line 220 def clear!(&block) objects_to_delete = bucket.objects(prefix: prefix) objects_to_delete = objects_to_delete.lazy.select(&block) if block delete_objects(objects_to_delete) end
Deletes the file from the storage.
# File lib/shrine/storage/s3.rb, line 200 def delete(id) object(id).delete end
Deletes objects at keys starting with the specified prefix.
s3.delete_prefixed("somekey/derivatives/")
# File lib/shrine/storage/s3.rb, line 207 def delete_prefixed(delete_prefix) # We need to make sure to combine with storage prefix, and # that it ends in '/' cause S3 can be squirrely about matching interior. delete_prefix = delete_prefix.chomp("/") + "/" bucket.objects(prefix: [*prefix, delete_prefix].join("/")).batch_delete! end
Returns true file exists on S3
.
# File lib/shrine/storage/s3.rb, line 128 def exists?(id) object(id).exists? end
Returns an ‘Aws::S3::Object` for the given id.
# File lib/shrine/storage/s3.rb, line 228 def object(id) bucket.object(object_key(id)) end
Returns a ‘Down::ChunkedIO` object that downloads S3
object content on-demand. By default, read content will be cached onto disk so that it can be rewinded, but if you don’t need that you can pass ‘rewindable: false`. A required character encoding can be passed in `encoding`; the default is `Encoding::BINARY` via `Down::ChunkedIO`.
Any additional options are forwarded to [‘Aws::S3::Object#get`].
[‘Aws::S3::Object#get`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#get-instance_method
# File lib/shrine/storage/s3.rb, line 119 def open(id, rewindable: true, encoding: nil, **options) chunks, length = get(id, **options) Down::ChunkedIO.new(chunks: chunks, rewindable: rewindable, size: length, encoding: encoding) rescue Aws::S3::Errors::NoSuchKey raise Shrine::FileNotFound, "file #{id.inspect} not found on storage" end
Returns URL, params, headers, and verb for direct uploads.
s3.presign("key") #=> # { # url: "https://my-bucket.s3.amazonaws.com/...", # fields: { ... }, # blank for PUT presigns # headers: { ... }, # blank for POST presigns # method: "post", # }
By default it calls [‘Aws::S3::Object#presigned_post`] which generates data for a POST request, but you can also specify `method: :put` for PUT uploads which calls [`Aws::S3::Object#presigned_url`].
s3.presign("key", method: :post) # for POST upload (default) s3.presign("key", method: :put) # for PUT upload
Any additional options are forwarded to the underlying AWS SDK method.
[‘Aws::S3::Object#presigned_post`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_post-instance_method [`Aws::S3::Object#presigned_url`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_url-instance_method
# File lib/shrine/storage/s3.rb, line 189 def presign(id, method: :post, **presign_options) options = {} options[:acl] = "public-read" if public options.merge!(@upload_options) options.merge!(presign_options) send(:"presign_#{method}", id, options) end
If the file is an UploadedFile
from S3
, issues a COPY command, otherwise uploads the file. For files larger than ‘:multipart_threshold` a multipart upload/copy will be used for better performance and more resilient uploads.
It assigns the correct “Content-Type” taken from the MIME type, because by default S3
sets everything to “application/octet-stream”.
# File lib/shrine/storage/s3.rb, line 92 def upload(io, id, shrine_metadata: {}, **upload_options) content_type, filename = shrine_metadata.values_at("mime_type", "filename") options = {} options[:content_type] = content_type if content_type options[:content_disposition] = ContentDisposition.inline(filename) if filename options[:acl] = "public-read" if public options.merge!(@upload_options) options.merge!(upload_options) if copyable?(io) copy(io, id, **options) else put(io, id, **options) end end
Returns the presigned URL to the file.
:host : This option replaces the host part of the returned URL, and is
typically useful for setting CDN hosts (e.g. `http://abc123.cloudfront.net`)
:public : Returns the unsigned URL to the S3
object. This requires the S3
object to be public.
All other options are forwarded to [‘Aws::S3::Object#presigned_url`] or [`Aws::S3::Object#public_url`].
[‘Aws::S3::Object#presigned_url`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#presigned_url-instance_method [`Aws::S3::Object#public_url`]: docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#public_url-instance_method
# File lib/shrine/storage/s3.rb, line 148 def url(id, public: self.public, host: nil, **options) if public || signer url = object(id).public_url(**options) else url = object(id).presigned_url(:get, **options) end if host uri = URI.parse(url) uri.path = uri.path.match(/^\/#{bucket.name}/).post_match unless uri.host.include?(bucket.name) url = URI.join(host, uri.request_uri[1..-1]).to_s end if signer url = signer.call(url, **options) end url end
Private Instance Methods
Copies an existing S3
object to a new location. Uses multipart copy for large files.
# File lib/shrine/storage/s3.rb, line 247 def copy(io, id, **copy_options) # don't inherit source object metadata or AWS tags options = { metadata_directive: "REPLACE", } if io.size && io.size >= @multipart_threshold[:copy] # pass :content_length on multipart copy to avoid an additional HEAD request options.merge!(multipart_copy: true, content_length: io.size) end options.merge!(@copy_options) options.merge!(copy_options) object(id).copy_from(io.storage.object(io.id), **options) end
The file is copyable if it’s on S3
and on the same Amazon account.
# File lib/shrine/storage/s3.rb, line 336 def copyable?(io) io.is_a?(UploadedFile) && io.storage.is_a?(Storage::S3) && io.storage.client.config.access_key_id == client.config.access_key_id end
Deletes all objects in fewest requests possible (S3
only allows 1000 objects to be deleted at once).
# File lib/shrine/storage/s3.rb, line 344 def delete_objects(objects) objects.each_slice(1000) do |objects_batch| delete_params = { objects: objects_batch.map { |object| { key: object.key } } } bucket.delete_objects(delete: delete_params) end end
# File lib/shrine/storage/s3.rb, line 304 def get(id, **params) enum = object(id).enum_for(:get, **params) begin content_length = Integer(enum.peek.last["content-length"]) rescue StopIteration content_length = 0 end chunks = Enumerator.new { |y| loop { y << enum.next.first } } [chunks, content_length] end
Returns object key with potential prefix.
# File lib/shrine/storage/s3.rb, line 352 def object_key(id) [*prefix, id].join("/") end
Determins the part size that should be used when uploading the given IO object via multipart upload.
# File lib/shrine/storage/s3.rb, line 290 def part_size(io) return unless io.respond_to?(:size) && io.size if io.size <= MIN_PART_SIZE * @max_multipart_parts # <= 50 GB MIN_PART_SIZE else # > 50 GB (io.size.to_f / @max_multipart_parts).ceil end end
Generates parameters for a POST upload request.
# File lib/shrine/storage/s3.rb, line 265 def presign_post(id, options) presigned_post = object(id).presigned_post(options) { method: :post, url: presigned_post.url, fields: presigned_post.fields } end
Generates parameters for a PUT upload request.
# File lib/shrine/storage/s3.rb, line 272 def presign_put(id, options) url = object(id).presigned_url(:put, options) # When any of these options are specified, the corresponding request # headers must be included in the upload request. headers = {} headers["Content-Length"] = options[:content_length] if options[:content_length] headers["Content-Type"] = options[:content_type] if options[:content_type] headers["Content-Disposition"] = options[:content_disposition] if options[:content_disposition] headers["Content-Encoding"] = options[:content_encoding] if options[:content_encoding] headers["Content-Language"] = options[:content_language] if options[:content_language] headers["Content-MD5"] = options[:content_md5] if options[:content_md5] { method: :put, url: url, headers: headers } end
Uploads the file to S3
. Uses multipart upload for large files.
# File lib/shrine/storage/s3.rb, line 235 def put(io, id, **options) if io.respond_to?(:size) && io.size && io.size <= @multipart_threshold[:upload] object(id).put(body: io, **options) else # multipart upload object(id).upload_stream(part_size: part_size(io), **options) do |write_stream| IO.copy_stream(io, write_stream) end end end