module Speculation::Test

Constants

S

This is a Ruby translation of clojure.spec:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj

All credit belongs with Rich Hickey and contributors for their original work.


This is a Ruby translation of clojure.spec:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj

All credit belongs with Rich Hickey and contributors for their original work.


This is a Ruby translation of clojure.spec:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj

All credit belongs with Rich Hickey and contributors for their original work.


This is a Ruby translation of clojure.spec:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj

All credit belongs with Rich Hickey and contributors for their original work.


This is a Ruby translation of clojure.spec:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj

All credit belongs with Rich Hickey and contributors for their original work.


This is a Ruby translation of clojure.spec:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj

All credit belongs with Rich Hickey and contributors for their original work.


This is a Ruby translation of clojure.spec:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj

All credit belongs with Rich Hickey and contributors for their original work.


This is a Ruby translation of clojure.spec:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj

All credit belongs with Rich Hickey and contributors for their original work.


This is a Ruby translation of clojure.spec:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj

All credit belongs with Rich Hickey and contributors for their original work.


This is a Ruby translation of clojure.spec:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj

All credit belongs with Rich Hickey and contributors for their original work.


This is a Ruby translation of clojure.spec:

https://github.com/clojure/clojure/blob/master/src/clj/clojure/spec.clj

All credit belongs with Rich Hickey and contributors for their original work.


Attributes

instrument_enabled[RW]

if false, instrumented methods call straight through

Public Class Methods

abbrev_result(x) click to toggle source

Given a check result, returns an abbreviated version suitable for summary use. @param x [Hash] @return [Hash]

# File lib/speculation/test.rb, line 175
def self.abbrev_result(x)
  if x[:failure]
    x.reject { |k, _| k == :ret }.
      merge(:spec    => x[:spec].inspect,
            :failure => unwrap_failure(x[:failure]))
  else
    x.reject { |k, _| [:spec, :ret].include?(k) }
  end
end
check(method_or_methods = nil, opts = {}) click to toggle source

Run generative tests for spec conformance on method_or_methods. If method_or_methods is not specified, check all checkable methods.

@param method_or_methods [Array<Method>, Method] @param opts [Hash] @option opts :num_tests [Integer] (1000) number of times to generatively test each method @option opts :gen [Hash] map from spec names to generator overrides.

Generator overrides are passed to Speculation.gen when generating method args.

@return [Array<Hash>] an array of check result hashes with the following keys:

* :spec       the spec tested
* :method     optional method tested
* :failure    optional test failure
* :result     optional boolean as to whether all generative tests passed
* :num_tests  optional number of generative tests ran

:failure is a hash that will contain a :failure key with possible values:

* :check_failed   at least one checked return did not conform
* :no_args_spec   no :args spec provided
* :no_fspec       no fspec provided
* :no_gen         unable to generate :args
* :instrument     invalid args detected by instrument
# File lib/speculation/test.rb, line 158
def self.check(method_or_methods = nil, opts = {})
  method_or_methods ||= checkable_methods

  checkable = Set(checkable_methods(opts))
  checkable.map!(&S.method(:MethodIdentifier))

  methods = Set(method_or_methods)
  methods.map!(&S.method(:MethodIdentifier))

  pmap(methods.intersection(checkable)) { |ident|
    check1(ident, S.get_spec(ident), opts)
  }
end
check_method(method, spec, opts = {}) click to toggle source

Runs generative tests for method using spec and opts. @param method [Method] @param spec [Spec] @param opts [Hash] @return [Hash] @see check see check for options and return

# File lib/speculation/test.rb, line 118
def self.check_method(method, spec, opts = {})
  validate_check_opts(opts)
  check1(S.MethodIdentifier(method), spec, opts)
end
checkable_methods(opts = {}) click to toggle source

@param opts [Hash] an opts hash as per `check` @return [Array<Method>] the array of methods that can be checked.

# File lib/speculation/test.rb, line 125
def self.checkable_methods(opts = {})
  validate_check_opts(opts)

  S.
    registry.
    keys.
    select { |k| fn_spec_name?(k) && !k.instance_method? }.
    concat(Hash(opts[:spec]).keys).
    map(&method(:Method))
end
enumerate_methods(*modules) click to toggle source

@param modules [Module, Class] @return [Array<Method>] an array of public and protected singleton

methods belonging to modules
# File lib/speculation/test.rb, line 212
def self.enumerate_methods(*modules)
  modules.flat_map { |mod| mod.methods(false).map(&mod.method(:method)) } # method
end
instrument(method_or_methods = instrumentable_methods, opts = {}) click to toggle source

@param method_or_methods [Method, Array<Method>]

Instruments the methods named by method-or-methods, a method or collection
of methods, or all instrumentable methods if method_or_methods is not
specified.
If a method has an :args fn-spec, replaces the method with a method that
checks arg conformance (throwing an exception on failure) before
delegating to the original method.

@param opts [Hash] opts hash can be used to override registered specs, and/or to replace

method implementations entirely. Opts for methods not included in
method-or-methods are ignored. This facilitates sharing a common options
hash across many different calls to instrument

@option opts :spec [Hash] a map from methods to override specs.

:spec overrides registered method specs with specs you provide. Use :spec
overrides to provide specs for libraries that do not have them, or to
constrain your own use of a fn to a subset of its spec'ed contract.
:spec can be used in combination with :stub or :replace.

@option opts :stub [Set, Array] a set of methods to be replaced by stubs.

:stub replaces a fn with a stub that checks :args, then uses the :ret spec
to generate a return value.

@option opts :gen [Hash{Symbol => Proc}] a map from spec names to generator overrides.

:gen overrides are used only for :stub generation.

@option opts :replace [Hash{Method => Proc}] a map from methods to replacement procs.

:replace replaces a method with a method that checks args conformance,
then invokes the method/proc you provide, enabling arbitrary stubbing and
mocking.

@return [Array<Method>] a collection of methods instrumented.

# File lib/speculation/test.rb, line 84
def self.instrument(method_or_methods = instrumentable_methods, opts = {})
  if opts[:gen]
    gens = opts[:gen].reduce({}) { |h, (k, v)| h.merge(S.MethodIdentifier(k) => v) }
    opts = opts.merge(:gen => gens)
  end

  Array(method_or_methods).
    map { |method| S.MethodIdentifier(method) }.
    uniq.
    map { |ident| instrument1(ident, opts) }.
    compact.
    map(&method(:Method))
end
instrumentable_methods(opts = {}) click to toggle source

Given an opts hash as per instrument, returns the set of methods that can be instrumented. @param opts [Hash] @return [Array<Method>]

# File lib/speculation/test.rb, line 40
def self.instrumentable_methods(opts = {})
  if opts[:gen]
    unless opts[:gen].keys.all? { |k| k.is_a?(Method) || k.is_a?(Symbol) }
      raise ArgumentError, "instrument :gen expects Method or Symbol keys"
    end
  end

  S.registry.keys.select(&method(:fn_spec_name?)).to_set.tap { |set|
    set.merge(opts[:spec].keys)    if opts[:spec]
    set.merge(opts[:stub])         if opts[:stub]
    set.merge(opts[:replace].keys) if opts[:replace]
  }.map(&method(:Method))
end
summarize_results(check_results, &summary_result) click to toggle source

Given a collection of check_results, e.g. from `check`, pretty prints the summary_result (default abbrev_result) of each.

@param check_results [Array] a collection of check_results @yield [Hash] @return [Hash] a hash with :total, the total number of results, plus a key with a

count for each different :type of result.

@see check see check for check_results @see abbrev_result

# File lib/speculation/test.rb, line 194
def self.summarize_results(check_results, &summary_result)
  summary_result ||= method(:abbrev_result)

  check_results.reduce(:total => 0) { |summary, result|
    pp summary_result.call(result)

    result_key = result_type(result)

    summary.merge(
      :total     => summary[:total].next,
      result_key => summary.fetch(result_key, 0).next
    )
  }
end
unstrument(method_or_methods = nil) click to toggle source

Undoes instrument on the method_or_methods, specified as in instrument. With no args, unstruments all instrumented methods. @param method_or_methods [Method, Array<Method>] @return [Array<Method>] a collection of methods unstrumented

# File lib/speculation/test.rb, line 102
def self.unstrument(method_or_methods = nil)
  method_or_methods ||= @instrumented_methods.value.keys

  Array(method_or_methods).
    map { |method| S.MethodIdentifier(method) }.
    map { |ident| unstrument1(ident) }.
    compact.
    map(&method(:Method))
end
with_instrument_disabled() { || ... } click to toggle source

Disables instrument's checking of calls within a block

# File lib/speculation/test.rb, line 29
def self.with_instrument_disabled
  instrument_enabled.value = false
  yield
ensure
  instrument_enabled.value = true
end

Private Class Methods

Method(x) click to toggle source

if x is an MethodIdentifier, return its method

# File lib/speculation/test.rb, line 566
def Method(x)
  case x
  when MethodIdentifier      then x.get_method
  when Method, UnboundMethod then x
  else raise ArgumentError, "unexpected method-like object #{x}"
  end
end
Set(x) click to toggle source
# File lib/speculation/test.rb, line 574
def Set(x)
  case x
  when Set then x
  when Enumerable then Set.new(x)
  else Set[x]
  end
end
backtrace_relevant_to_instrument(backtrace) click to toggle source
# File lib/speculation/test.rb, line 462
def backtrace_relevant_to_instrument(backtrace)
  backtrace.drop_while { |line| line.include?(__FILE__) }
end
check1(ident, spec, opts) click to toggle source
# File lib/speculation/test.rb, line 433
def check1(ident, spec, opts)
  specd = S.spec(spec)

  reinstrument = unstrument(ident).any?
  method = ident.get_method

  if specd.args
    check_result = quick_check(method, spec, opts)
    make_check_result(method, spec, check_result)
  else
    failure = { :info    => "No :args spec",
                :failure => :no_args_spec }

    { :failure => failure,
      :method  => method,
      :spec    => spec }
  end
ensure
  instrument(ident) if reinstrument
end
check_call(method, spec, args, block) click to toggle source

Returns true if call passes specs, otherwise returns a hash with :backtrace, :cause and :data keys. :data will have a :failure key.

# File lib/speculation/test.rb, line 359
def check_call(method, spec, args, block)
  conformed_args = S.conform(spec.args, args) if spec.args

  if conformed_args == :"Speculation/invalid"
    return explain_check(args, spec.args, args, :args)
  end

  conformed_block = S.conform(spec.block, block) if spec.block

  if conformed_block == :"Speculation/invalid"
    return explain_check(block, spec.block, block, :block)
  end

  ret = method.call(*args, &block)

  conformed_ret = S.conform(spec.ret, ret) if spec.ret

  if conformed_ret == :"Speculation/invalid"
    return explain_check(args, spec.ret, ret, :ret)
  end

  return true unless spec.args && spec.ret && spec.fn

  if S.valid?(spec.fn, :args => conformed_args, :block => conformed_block, :ret => conformed_ret)
    true
  else
    explain_check(args, spec.fn, { :args => conformed_args, :block => conformed_block, :ret => conformed_ret }, :fn)
  end
end
explain_check(args, spec, v, role) click to toggle source
# File lib/speculation/test.rb, line 343
def explain_check(args, spec, v, role)
  data = unless S.valid?(spec, v)
           S._explain_data(spec, [role], [], [], v).
             merge(:args    => args,
                   :val     => v,
                   :failure => :check_failed)
         end

  S::Error.new("Specification-based check failed", data).tap do |e|
    e.set_backtrace(caller)
  end
end
failure_type(x) click to toggle source

check reporting ###

# File lib/speculation/test.rb, line 542
def failure_type(x)
  x.data[:failure] if x.is_a?(S::Error)
end
fn_spec_name?(spec_name) click to toggle source
# File lib/speculation/test.rb, line 466
def fn_spec_name?(spec_name)
  spec_name.is_a?(S::MethodIdentifier)
end
instrument1(ident, opts) click to toggle source
# File lib/speculation/test.rb, line 277
def instrument1(ident, opts)
  spec = S.get_spec(ident)

  raw, wrapped = @instrumented_methods.
    value.
    fetch(ident, {}).
    values_at(:raw, :wrapped)

  current = ident.get_method
  to_wrap = wrapped == current ? raw : current

  ospec = instrument_choose_spec(spec, ident, opts[:spec])
  raise no_fspec(ident, spec) unless ospec

  ofn = instrument_choose_fn(to_wrap, ospec, ident, opts)

  checked = spec_checking_fn(ident, ofn, ospec)

  ident.redefine_method!(checked)

  wrapped = ident.get_method

  @instrumented_methods.swap do |methods|
    methods.merge(ident => { :raw => to_wrap, :wrapped => wrapped })
  end

  ident
end
instrument_choose_fn(f, spec, ident, opts) click to toggle source
# File lib/speculation/test.rb, line 306
def instrument_choose_fn(f, spec, ident, opts)
  stubs   = (opts[:stub] || []).map(&S.method(:MethodIdentifier))
  over    = opts[:gen] || {}
  replace = (opts[:replace] || {}).reduce({}) { |h, (k, v)| h.merge(S.MethodIdentifier(k) => v) }

  if stubs.include?(ident)
    Gen.generate(S.gen(spec, over))
  else
    replace.fetch(ident, f)
  end
end
instrument_choose_spec(spec, ident, overrides) click to toggle source
# File lib/speculation/test.rb, line 318
def instrument_choose_spec(spec, ident, overrides)
  (overrides || {}).
    reduce({}) { |h, (k, v)| h.merge(S.MethodIdentifier(k) => v) }.
    fetch(ident, spec)
end
make_check_result(method, spec, check_result) click to toggle source
# File lib/speculation/test.rb, line 417
def make_check_result(method, spec, check_result)
  result = { :spec   => spec,
             :ret    => check_result,
             :method => method }

  if check_result[:result] && check_result[:result] != true
    result[:failure] = check_result[:result]
  end

  if check_result[:shrunk]
    result[:failure] = check_result[:shrunk][:result]
  end

  result
end
no_fspec(ident, spec) click to toggle source
# File lib/speculation/test.rb, line 273
def no_fspec(ident, spec)
  S::Error.new("#{ident} not spec'ed", :method => ident, :spec => spec, :failure => :no_fspec)
end
quick_check(method, spec, opts) click to toggle source
# File lib/speculation/test.rb, line 389
def quick_check(method, spec, opts)
  gen = opts[:gen]
  num_tests = opts.fetch(:num_tests, 1000)

  args_gen = begin
               S.gen(spec.args, gen)
             rescue => e
               return { :result => e }
             end

  block_gen = if spec.block
                begin
                  S.gen(spec.block, gen)
                rescue => e
                  return { :result => e }
                end
              else
                Utils.constantly(nil)
              end

  arg_block_gen = Gen.tuple(args_gen, block_gen)

  generator_guard = ->(genned_val) { S.valid?(spec.args, genned_val) }
  rantly_quick_check(arg_block_gen, num_tests, generator_guard) do |(args, block)|
    check_call(method, spec, args, block)
  end
end
rantly_quick_check(gen, num_tests, generator_guard) { |args, blk| ... } click to toggle source

Reimplementation of Rantly's `check` since it does not provide direct access to results (shrunk data etc.), instead printing them to STDOUT.

# File lib/speculation/test.rb, line 472
def rantly_quick_check(gen, num_tests, generator_guard, &invariant)
  i = 0
  limit = 100

  Rantly.singleton.generate(num_tests, limit, gen) do |val|
    args, blk = val
    i += 1

    result = yield([args, blk]) rescue $!

    unless result == true
      args = ::Tuple.new(args) # This is a Rantly Tuple.

      shrunk = shrink(generator_guard, args, result, ->(v) { invariant.call([v, blk]) })

      shrunk[:smallest] = { :args => shrunk[:smallest].array, :block => blk }

      return { :fail      => { :args => args.array, :block => blk },
               :num_tests => i,
               :result    => result,
               :shrunk    => shrunk }
    end
  end

  { :num_tests => i,
    :result    => true }
end
result_type(ret) click to toggle source

Returns the type of the check result. This can be any of the :failure symbols documented in 'check', or:

:check_passed all checked method returns conformed :check_raised checked fn threw an exception

# File lib/speculation/test.rb, line 555
def result_type(ret)
  failure = ret[:failure]

  if failure.nil?
    :check_passed
  else
    failure_type(failure) || :check_raised
  end
end
shrink(generator_guard, value, result, invariant, depth = 0, iteration = 0) click to toggle source

reimplementation of Rantly's shrinking.

# File lib/speculation/test.rb, line 501
def shrink(generator_guard, value, result, invariant, depth = 0, iteration = 0)
  smallest = value
  max_depth = depth

  if value.shrinkable?
    while iteration < 1024
      shrunk_value = value.shrink

      unless generator_guard.call(shrunk_value.array)
        iteration += 1
        value = shrunk_value
        value.shrinkable? ? next : break
      end

      res = invariant.call(shrunk_value.array) rescue $!

      unless res == true
        shrunk = shrink(generator_guard, shrunk_value, res, invariant, depth + 1, iteration + 1)

        branch_smallest, branch_depth, iteration, branch_result =
          shrunk.values_at(:smallest, :depth, :iteration, :result)

        if branch_depth > max_depth
          max_depth = branch_depth
          smallest = branch_smallest
          result = branch_result
        end
      end

      break unless value.retry?
    end
  end

  { :depth     => max_depth,
    :iteration => iteration,
    :result    => result,
    :smallest  => smallest }
end
spec_checking_fn(ident, method, fspec) click to toggle source
# File lib/speculation/test.rb, line 219
def spec_checking_fn(ident, method, fspec)
  fspec = S.send(:maybe_spec, fspec)

  conform = ->(args, block) do
    conformed_args = S.conform(fspec.args, args)
    conformed_block = S.conform(fspec.block, block) if fspec.block

    if conformed_args == :"Speculation/invalid"
      backtrace = backtrace_relevant_to_instrument(caller)

      ed = S.
        _explain_data(fspec.args, [:args], [], [], args).
        merge(:args => args, :failure => :instrument, :caller => backtrace.first)

      io = StringIO.new
      S.explain_out(ed, io)
      msg = io.string

      raise Speculation::Error.new("Call to '#{ident}' did not conform to spec:\n#{msg}", ed)
    elsif conformed_block == :"Speculation/invalid"
      backtrace = backtrace_relevant_to_instrument(caller)

      ed = S.
        _explain_data(fspec.block, [:block], [], [], block).
        merge(:block => block, :failure => :instrument, :caller => backtrace.first)

      io = StringIO.new
      S.explain_out(ed, io)
      msg = io.string

      raise Speculation::Error.new("Call to '#{ident}' did not conform to spec:\n#{msg}", ed)
    end
  end

  ->(*args, &block) do
    method = method.bind(self) if method.is_a?(UnboundMethod)

    if Test.instrument_enabled.value
      Test.with_instrument_disabled do
        conform.call(args, block) if fspec.args

        begin
          Test.instrument_enabled.value = true
          method.call(*args, &block)
        ensure
          Test.instrument_enabled.value = false
        end
      end
    else
      method.call(*args, &block)
    end
  end
end
unstrument1(ident) click to toggle source
# File lib/speculation/test.rb, line 324
def unstrument1(ident)
  instrumented = @instrumented_methods.value[ident]
  return unless instrumented

  raw, wrapped = instrumented.values_at(:raw, :wrapped)

  @instrumented_methods.swap do |h|
    h.reject { |k, _v| k == ident }
  end

  current = ident.get_method

  # Only redefine to original if it has not been modified since it was
  # instrumented.
  if wrapped == current
    ident.tap { |i| i.redefine_method!(raw) }
  end
end
unwrap_failure(x) click to toggle source
# File lib/speculation/test.rb, line 546
def unwrap_failure(x)
  failure_type(x) ? x.data : x
end
validate_check_opts(opts) click to toggle source
# File lib/speculation/test.rb, line 454
def validate_check_opts(opts)
  return unless opts[:gen]

  unless opts[:gen].keys.all? { |k| k.is_a?(Method) || k.is_a?(Symbol) }
    raise ArgumentErorr, "check :gen expects Method or Symbol keys"
  end
end