class Google::Cloud::Gemserver::Backend::StorageSync

# Storage Sync

A set of methods that manage syncing files between the local file system that the gemserver runs on (a container on Google App Engine) and Google Cloud Storage. By doing so, when the gemserver is restarted , for whatever reason, the gems pushed to the gemserver will persist. Without such a system in place all gems / files on the gemserver will be lost as it runs on a container.

Constants

DIR_LOCK

A lock to ensure the .gemstash directory, used to store gem files, is created atomically.

Public Class Methods

new() click to toggle source

Creates an instance of the Singleton StorageSync class with a background thread and asynchronous components to run methods asynchronously.

Calls superclass method
# File lib/google/cloud/gemserver/backend/storage_sync.rb, line 64
def initialize
  super
end

Public Instance Methods

download_service() click to toggle source

@private The downloading service that downloads gem files from Google Cloud Storage to the local file system.

# File lib/google/cloud/gemserver/backend/storage_sync.rb, line 138
def download_service
  puts "Running downloading service..."
  prepare_dir

  files = GCS.files
  return unless files
  files.each { |file| try_download file.name }
end
file_changed?(file) click to toggle source

@private Determines if a file on the local file system has changed from the corresponding file on Google Cloud Storage, if it exists.

@param [String] file Name of the file

@return [Boolean]

# File lib/google/cloud/gemserver/backend/storage_sync.rb, line 175
def file_changed? file
  return true unless File.exist? file
  return true unless GCS.get_file(file)
  gcs_md5 = GCS.get_file(file).md5
  local_md5 = Digest::MD5.file(file).base64digest
  gcs_md5 != local_md5
end
gemstash_dir() click to toggle source

@private The directory used to store gem data.

@return [String]

# File lib/google/cloud/gemserver/backend/storage_sync.rb, line 90
def gemstash_dir
  if ENV["APP_ENV"] == "production"
    Configuration::GEMSTASH_DIR
  else
    File.expand_path("~/.gemstash")
  end
end
instance() click to toggle source

Delegate the run and download_service methods to the Singleton via .instance.

# File lib/google/cloud/gemserver/backend/storage_sync.rb, line 56
def_delegators :instance, :run, :upload_service, :download_service,
  :try_upload, :try_download, :file_changed?
prepare_dir() click to toggle source

@private Create the directory used to store gem data.

# File lib/google/cloud/gemserver/backend/storage_sync.rb, line 100
def prepare_dir
  Filelock DIR_LOCK do
    FileUtils.mkpath gemstash_dir
  end
end
run() click to toggle source

Runs a background gem files syncing service to ensure they are up to date with respect to the files on Google Cloud Storage. This allows allow the gem metadata on the gemserver (in the container) to persist in case of situations where the gemserver goes down.

# File lib/google/cloud/gemserver/backend/storage_sync.rb, line 73
def run
  async.sync
end
sync() click to toggle source

@private Runs the uploader to send updated gem files to Google Cloud Storage (source of truth) then updates the rest of the gem files by downloading them off Google Cloud Storage.

# File lib/google/cloud/gemserver/backend/storage_sync.rb, line 81
def sync
  upload_service
  download_service
end
try_download(file) click to toggle source

@private Downloads a file to the local file sytem from Google Cloud Storage only if there is sufficient space and the local copy's hash differs from the cloud copy's hash.

@param [String] file Name of the file to download.

# File lib/google/cloud/gemserver/backend/storage_sync.rb, line 153
def try_download file
  total = `df -k /`.split(" ")[11].to_f
  used = `df -k /`.split(" ")[12].to_f
  usage = used / total
  if usage < 0.95
    if File.exist? file
      Filelock(file) { GCS.sync file if file_changed? file }
    else
      GCS.copy_to_host file
    end
  else
    raise "Error downloading: disk usage at #{usage}! Increase disk space!"
  end
end
try_upload(file) click to toggle source

@private Uploads a file to Google Cloud Storage only if the file has yet to be uploaded or has a different hash from the Cloud copy.

@param [String] file The path to the file to be uploaded.

# File lib/google/cloud/gemserver/backend/storage_sync.rb, line 127
def try_upload file
  GCS.upload(file) unless GCS.on_gcs?(file)
  return unless file_changed?(file)
  Filelock file do
    GCS.upload file
  end
end
upload_service() click to toggle source

@private The uploading service that uploads gem files from the local file system to Google Cloud Storage. It does not upload any cached files.

# File lib/google/cloud/gemserver/backend/storage_sync.rb, line 110
def upload_service
  puts "Running uploading service..."
  prepare_dir

  entries = Dir.glob("#{gemstash_dir}/**/*").reject do |e|
    e.include? "gem_cache"
  end
  entries.each do |e|
    try_upload e if File.file? e
  end
end