class WinRM::Transport::FileTransporter

Object which can upload one or more files or directories to a remote host over WinRM using PowerShell scripts and CMD commands. Note that this form of file transfer is not ideal and extremely costly on both the local and remote sides. Great pains are made to minimize round trips to the remote host and to minimize the number of PowerShell sessions being invoked which can be 2 orders of magnitude more expensive than vanilla CMD commands.

This object is supported by either a `WinRM::WinRMWebService` or `CommandExecutor` instance as it depends on the `#run_cmd` and `#run_powershell_script` API contracts.

An optional logger can be supplied, assuming it can respond to the `#debug` and `#debug?` messages.

@author Fletcher Nichol <fnichol@nichol.ca> @author Matt Wrock <matt@mattwrock.com>

Constants

BASE64_PACK

@return [String] the Array pack template for Base64 encoding a stream

of data

@api private

MAX_ENCODED_WRITE

@return [Integer] the maximum number of bytes that can be supplied on

a Windows CMD prompt without exceeded the maximum command line
length

@api private

Attributes

logger[R]

@return [#debug,#debug?] the logger @api private

service[R]

@return [WinRM::WinRMWebService,Winrm::CommandExecutor] a WinRM web

service object

@api private

Public Class Methods

new(service, logger = nil, opts = {}) click to toggle source

Creates a FileTransporter given a service object and optional logger. The service object may be a `WinRM::WinRMWebService` or `CommandExecutor` instance.

@param service [WinRM::WinRMWebService,CommandExecutor] a

winrm web service object

@param logger [#debug,#debug?] an optional logger/ui object that

responds to `#debug` and `#debug?` (default: `nil`)
# File lib/winrm/transport/file_transporter.rb, line 65
def initialize(service, logger = nil, opts = {})
  @service  = service
  @logger   = logger
  @id_generator = opts.fetch(:id_generator) { -> { SecureRandom.uuid } }
end

Public Instance Methods

upload(locals, remote) click to toggle source

Uploads a collection of files and/or directories to the remote host.

**TODO Notes:**

  • options could specify zip mode, zip options, etc.

  • maybe option to set tmpfile base dir to override $env:PATH?

  • progress yields block like net-scp progress

  • final API: def upload(locals, remote, _options = {}, &_progress)

@param locals [Array<String>,String] one or more local file or

directory paths

@param remote [String] the base destination path on the remote host @return [Hash] report hash, keyed by the local MD5 digest

# File lib/winrm/transport/file_transporter.rb, line 83
def upload(locals, remote)
  files = nil

  elapsed = Benchmark.measure do
    files = make_files_hash(Array(locals), remote)

    report = check_files(files)
    merge_with_report!(files, report)

    report = stream_upload_files(files)
    merge_with_report!(files, report)

    report = decode_files(files)
    merge_with_report!(files, report)

    cleanup(files)
  end

  debug {
    "Uploaded #{files.keys.size} items " \
    "in #{duration(elapsed.real)}"
  }

  files
end

Private Instance Methods

add_directory_hash!(hash, dir, remote) click to toggle source

Adds an entry to a files Hash (keyed by local MD5 digest) for a directory. When a directory is added, a temporary Zip file is created containing the contents of the directory and any file-related data such as MD5 digest, size, etc. will be referring to the Zip file.

@param hash [Hash] hash to be mutated @param dir [String] directory path to be Zipped and added @param remote [String] path to destination on remote host @api private

# File lib/winrm/transport/file_transporter.rb, line 140
def add_directory_hash!(hash, dir, remote)
  zip_io = TmpZip.new(dir, logger)
  zip_md5 = md5sum(zip_io.path)

  hash[zip_md5] = {
    "src"     => dir,
    "src_zip" => zip_io.path.to_s,
    "zip_io"  => zip_io,
    "tmpzip"  => "$env:TEMP\\tmpzip-#{zip_md5}.zip",
    "dst"     => remote,
    "size"    => File.size(zip_io.path)
  }
end
add_file_hash!(hash, local, remote) click to toggle source

Adds an entry to a files Hash (keyed by local MD5 digest) for a file.

@param hash [Hash] hash to be mutated @param local [String] file path @param remote [String] path to destination on remote host @api private

# File lib/winrm/transport/file_transporter.rb, line 160
def add_file_hash!(hash, local, remote)
  hash[md5sum(local)] = {
    "src"   => local,
    "dst"   => "#{remote}\\#{File.basename(local)}",
    "size"  => File.size(local)
  }
end
check_files(files) click to toggle source

Runs the check_files PowerShell script against a collection of destination path/MD5 checksum pairs. The PowerShell script returns its results as a CSV-formatted report which is converted into a Ruby Hash.

@param files [Hash] files hash, keyed by the local MD5 digest @return [Hash] a report hash, keyed by the local MD5 digest @api private

# File lib/winrm/transport/file_transporter.rb, line 176
def check_files(files)
  debug { "Running check_files.ps1" }
  hash_file = create_remote_hash_file(check_files_ps_hash(files))
  vars = %{$hash_file = "#{hash_file}"\n}
  output = service.run_powershell_script(
    [vars, check_files_script].join("\n")
  )
  parse_response(output)
end
check_files_ps_hash(files) click to toggle source

Constructs a collection of destination path/MD5 checksum pairs as a String representation of the contents of a PowerShell Hash Table.

@param files [Hash] files hash, keyed by the local MD5 digest @return [String] the inner contents of a PowerShell Hash Table @api private

# File lib/winrm/transport/file_transporter.rb, line 192
def check_files_ps_hash(files)
  ps_hash(Hash[
    files.map { |md5, data| [data.fetch("tmpzip", data["dst"]), md5] }
  ])
end
check_files_script() click to toggle source

@return [String] the check_files PowerShell script @api private

# File lib/winrm/transport/file_transporter.rb, line 200
def check_files_script
  @check_files_script ||= IO.read(File.join(
    File.dirname(__FILE__), %W[.. .. .. support check_files.ps1]
  ))
end
cleanup(files) click to toggle source

Performs any final cleanup on the report Hash and removes any temporary files/resources used in the upload task.

@param files [Hash] a files hash @api private

# File lib/winrm/transport/file_transporter.rb, line 211
def cleanup(files)
  files.select { |_, data| data.key?("zip_io") }.each do |md5, data|
    data.fetch("zip_io").unlink
    files.fetch(md5).delete("zip_io")
    debug { "Cleaned up src_zip #{data["src_zip"]}" }
  end
end
clixml_to_s(clixml) click to toggle source

Parses CLIXML String into regular String (without any XML syntax). Inspired by github.com/WinRb/WinRM/issues/106.

@param clixml [String] clixml text @return [String] parsed clixml into String

# File lib/winrm/transport/file_transporter.rb, line 355
def clixml_to_s(clixml)
  doc = REXML::Document.new(clixml)
  text = doc.get_elements("//S").map(&:text).join
  text.gsub(/_x(\h\h\h\h)_/) do
    code = Regexp.last_match[1]
    code.hex.chr
  end
end
create_remote_hash_file(hash) click to toggle source

Creates a remote Base64-encoded temporary file containing a PowerShell hash table.

@param hash [String] a String representation of a PowerShell hash

table

@return [String] the remote path to the temporary file @api private

# File lib/winrm/transport/file_transporter.rb, line 226
def create_remote_hash_file(hash)
  hash_file = "$env:TEMP\\hash-#{@id_generator.call}.txt"
  hash.lines.each { |line| debug { line.chomp } }
  StringIO.open(hash) { |io| stream_upload(io, hash_file) }
  hash_file
end
decode_files(files) click to toggle source

Runs the decode_files PowerShell script against a collection of temporary file/destination path pairs. The PowerShell script returns its results as a CSV-formatted report which is converted into a Ruby Hash. The script will not be invoked if there are no “dirty” files present in the incoming files Hash.

@param files [Hash] files hash, keyed by the local MD5 digest @return [Hash] a report hash, keyed by the local MD5 digest @api private

# File lib/winrm/transport/file_transporter.rb, line 242
def decode_files(files)
  decoded_files = decode_files_ps_hash(files)

  if decoded_files == ps_hash(Hash.new)
    debug { "No remote files to decode, skipping" }
    Hash.new
  else
    debug { "Running decode_files.ps1" }
    hash_file = create_remote_hash_file(decoded_files)
    vars = %{$hash_file = "#{hash_file}"\n}

    output = service.run_powershell_script(
      ["#{decode_files_script}",
       vars,
       "Decode-Files (Invoke-Input $hash_file) | ConvertTo-Csv -NoTypeInformation"
      ].join("\n")
    )
    parse_response(output)
  end
end
decode_files_ps_hash(files) click to toggle source

Constructs a collection of temporary file/destination path pairs for all “dirty” files as a String representation of the contents of a PowerShell Hash Table. A “dirty” file is one which has the `“chk_dirty”` option set to `“True”` in the incoming files Hash.

@param files [Hash] files hash, keyed by the local MD5 digest @return [String] the inner contents of a PowerShell Hash Table @api private

# File lib/winrm/transport/file_transporter.rb, line 271
def decode_files_ps_hash(files)
  result = files.select { |_, data| data["chk_dirty"] == "True" }.map { |_, data|
    val = { "dst" => data["dst"] }
    val["tmpzip"] = data["tmpzip"] if data["tmpzip"]

    [data["tmpfile"], val]
  }

  ps_hash(Hash[result])
end
decode_files_script() click to toggle source

@return [String] the decode_files PowerShell script @api private

# File lib/winrm/transport/file_transporter.rb, line 284
def decode_files_script
  @decode_files_script ||= IO.read(File.join(
    File.dirname(__FILE__), %W[.. .. .. support decode_files.ps1]
  ))
end
duration(total) click to toggle source

Returns a formatted string representing a duration in seconds.

@param total [Integer] the total number of seconds @return [String] a formatted string of the form (XmYY.00s)

# File lib/winrm/transport/file_transporter.rb, line 294
def duration(total)
  total = 0 if total.nil?
  minutes = (total / 60).to_i
  seconds = (total - (minutes * 60))
  format("(%dm%.2fs)", minutes, seconds)
end
make_files_hash(locals, remote) click to toggle source

Contructs a Hash of files or directories, keyed by the local MD5 digest. Each file entry has a source and destination set, at a minimum.

@param locals [Array<String>] a collection of local files or

directories

@param remote [String] the base destination path on the remote host @return [Hash] files hash, keyed by the local MD5 digest @api private

# File lib/winrm/transport/file_transporter.rb, line 310
def make_files_hash(locals, remote)
  hash = Hash.new
  locals.each do |local|
    expanded = File.expand_path(local)
    expanded += local[-1] if local.end_with?("/", "\\")

    if File.file?(expanded)
      add_file_hash!(hash, expanded, remote)
    elsif File.directory?(expanded)
      add_directory_hash!(hash, expanded, remote)
    else
      raise Errno::ENOENT, "No such file or directory #{expanded}"
    end
  end
  hash
end
md5sum(local) click to toggle source

@return [String] the MD5 digest of a local file @api private

# File lib/winrm/transport/file_transporter.rb, line 329
def md5sum(local)
  Digest::MD5.file(local).hexdigest
end
merge_with_report!(files, report) click to toggle source

Destructively merges a report Hash into an existing files Hash. Note: this method mutates the files Hash.

@param files [Hash] files hash, keyed by the local MD5 digest @param report [Hash] report hash, keyed by the local MD5 digest @api private

# File lib/winrm/transport/file_transporter.rb, line 339
def merge_with_report!(files, report)
  files.merge!(report) { |_, oldval, newval| oldval.merge(newval) }
end
pad(depth = 0) click to toggle source

@param depth [Integer] number of padding characters (default: `0`) @return [String] a whitespace padded string of the given length @api private

# File lib/winrm/transport/file_transporter.rb, line 346
def pad(depth = 0)
  " " * depth
end
parse_response(output) click to toggle source

Parses response of a PowerShell script or CMD command which contains a CSV-formatted document in the standard output stream.

@param output [WinRM::Output] output object with stdout, stderr, and

exit code

@return [Hash] report hash, keyed by the local MD5 digest @api private

# File lib/winrm/transport/file_transporter.rb, line 371
def parse_response(output)
  exitcode = output[:exitcode]
  stderr = output.stderr
  if stderr.include?("The command line is too long")
    # The powershell script which should result in `output` parameter
    # is too long, remove some newlines, comments, etc from it.
    raise StandardError, "The command line is too long" \
      " (powershell script is too long)"
  end
  pretty_stderr = clixml_to_s(stderr)

  if exitcode != 0
    raise FileTransporterFailed, "[#{self.class}] Upload failed " \
      "(exitcode: #{exitcode})\n#{pretty_stderr}"
  elsif stderr != '\r\n' && stderr != ""
    raise FileTransporterFailed, "[#{self.class}] Upload failed " \
      "(exitcode: 0), but stderr present\n#{pretty_stderr}"
  end

  array = CSV.parse(output.stdout, :headers => true).map(&:to_hash)
  array.each { |h| h.each { |key, value| h[key] = nil if value == "" } }
  Hash[array.map { |entry| [entry.fetch("src_md5"), entry] }]
end
ps_hash(obj, depth = 0) click to toggle source

Converts a Ruby hash into a PowerShell hash table, represented in a String.

@param obj [Object] source Hash or object when used in recursive

calls

@param depth [Integer] padding depth, used in recursive calls

(default: `0`)

@return [String] a PowerShell hash table @api private

# File lib/winrm/transport/file_transporter.rb, line 404
def ps_hash(obj, depth = 0)
  if obj.is_a?(Hash)
    obj.map { |k, v|
      %{#{pad(depth + 2)}#{ps_hash(k)} = #{ps_hash(v, depth + 2)}}
    }.join(";\n").insert(0, "@{\n").insert(-1, "\n#{pad(depth)}}")
  else
    %{"#{obj}"}
  end
end
stream_upload(input_io, dest) click to toggle source

Uploads an IO stream to a Base64-encoded destination file.

**Implementation Note:** Some of the code in this method may appear slightly too dense and while adding additional variables would help, the code is written very precisely to avoid unwanted allocations which will bloat the Ruby VM's object space (and memory footprint). The goal here is to stream potentially large files to a remote host while not loading the entire file into memory first, then Base64 encoding it–duplicating the file in memory again.

@param input_io [#read] a readable stream or object to be uploaded @param dest [String] path to the destination file on the remote host @return [Integer,Integer] the number of resulting upload chunks and

the number of bytes transferred to the remote host

@api private

# File lib/winrm/transport/file_transporter.rb, line 429
def stream_upload(input_io, dest)
  dest_cmd = dest.sub("$env:TEMP", "%TEMP%")
  read_size = (MAX_ENCODED_WRITE.to_i / 4) * 3
  chunk, bytes = 1, 0
  buffer = ""
  service.run_cmd(%{echo|set /p=>"#{dest_cmd}"}) # truncate empty file
  while input_io.read(read_size, buffer)
    bytes += (buffer.bytesize / 3 * 4)
    service.run_cmd([buffer].pack(BASE64_PACK).
      insert(0, "echo ").concat(%{ >> "#{dest_cmd}"}))
    debug { "Wrote chunk #{chunk} for #{dest}" } if chunk % 25 == 0
    chunk += 1
  end
  buffer = nil # rubocop:disable Lint/UselessAssignment

  [chunk - 1, bytes]
end
stream_upload_file(src, tmpfile) click to toggle source

Uploads a local file to a Base64-encoded temporary file.

@param src [String] path to a local file @param tmpfile [String] path to the temporary file on the remote

host

@return [Integer,Integer] the number of resulting upload chunks and

the number of bytes transferred to the remote host

@api private

# File lib/winrm/transport/file_transporter.rb, line 455
def stream_upload_file(src, tmpfile)
  debug { "Uploading #{src} to encoded tmpfile #{tmpfile}" }
  chunks, bytes = 0, 0
  elapsed = Benchmark.measure do
    File.open(src, "rb") do |io|
      chunks, bytes = stream_upload(io, tmpfile)
    end
  end
  debug {
    "Finished uploading #{src} to encoded tmpfile #{tmpfile} " \
    "(#{bytes.to_f / 1000} KB over #{chunks} chunks) " \
    "in #{duration(elapsed.real)}"
  }

  [chunks, bytes]
end
stream_upload_files(files) click to toggle source

Uploads a collection of “dirty” files to the remote host as Base64-encoded temporary files. A “dirty” file is one which has the `“chk_dirty”` option set to `“True”` in the incoming files Hash.

@param files [Hash] files hash, keyed by the local MD5 digest @return [Hash] a report hash, keyed by the local MD5 digest @api private

# File lib/winrm/transport/file_transporter.rb, line 479
def stream_upload_files(files)
  response = Hash.new
  files.each do |md5, data|
    src = data.fetch("src_zip", data["src"])
    if data["chk_dirty"] == "True"
      tmpfile = "$env:TEMP\\b64-#{md5}.txt"
      response[md5] = { "tmpfile" => tmpfile }
      chunks, bytes = stream_upload_file(src, tmpfile)
      response[md5]["chunks"] = chunks
      response[md5]["xfered"] = bytes
    else
      debug { "File #{data["dst"]} is up to date, skipping" }
    end
  end
  response
end