module GECS
Gem for Experimental Computer Science
- Author
-
David Flater <dflater@nist.gov>
- Copyright
-
Public domain
- License
-
Unlicense
The Gem for Experimental Computer Science (GECS
) is for managing the data resulting from experiments on software and IT systems. It realizes a data model that disambiguates the vocabulary of experimentation and measurement for a computer science audience. It also provides convenience functions to determine confidence intervals, analyze main effects and interactions, and export data in an R-compatible format for further analysis and visualization.
This software is experimental. NIST assumes no responsibility whatsoever for its use by other parties and makes no guarantees, expressed or implied, about its quality, reliability, or any other characteristic.
Conventions used within the GECS
module¶ ↑
The following abbreviations are used:
- Parm
-
parameter
- Est
-
estimation or estimate
- Inv
-
interval
- Ind
-
independent
- Dep
-
dependent
- Var
-
variable
- Treat
-
treatment (a vector of factor levels)
Arrays of character strings (or in one case, ParmDef
structs) are used as a kind of “enum.” Values of the enum pseudo-type are unsigned integers that simply index the array to provide a concise identifier for whatever the referenced character string (or ParmDef
) describes.
A good many helper methods that should not be exposed through the GECS
API unfortunately are exposed and get listed by rdoc. These have been tagged (Private) in their comments.
Rdoc lists struct definitions in the Constants section.
Dependencies¶ ↑
To use the ::bootstrapMeans
method:
-
The Ruby gem parallel must be installed. Tested version: 0.9.2.
-
The R environment for statistical computing must be runnable from a shell command line. Tested version: 3.1.0.
-
The R package bootBCa must be installed. Tested version: 1.0.
To use the ::quickMeans
method:
-
The Ruby gem statistics2 must be installed. Tested version: 0.54.
Both ::bootstrapMeans
and ::quickMeans
use set, a standard pre-installed class.
Bad behaviors¶ ↑
::bootstrapMeans
writes temporary files into the current working directory. Normally, they will be deleted when no longer in use.
The inverse t distribution in statistics2 that is used by ::quickMeans
agrees with R and Octave only to 4 decimals or so.
Constants
- BagOfHolding
Normative struct definition.
BagOfHolding
is a bag containing everything that is loaded from or saved to an export file except for theGECS
version number.- parms
-
Array of strings defining pseudo-enum as used in a given data file. See
Parms
. - estMethods
-
Array of strings defining pseudo-enum as used in a given data file. See
EstMethods
. - estMethodParms
-
Array of strings defining pseudo-enum as used in a given data file. See
EstMethodParms
. - invTypes
-
Array of strings defining pseudo-enum as used in a given data file. See
InvTypes
. - parmDefs
-
Array of
ParmDef
structs defining pseudo-enum as used in a given data file. - experiments
-
Array of
Experiment
structs (index by experiment id). - data
-
Hash from
Key
to array (per depVars) of arrays (measurement values in chronological order). - ests
-
Hash from
Key
to array (per depVars) of hashes (from parmDefs index toParmData
).
- DoubleBag
Normative struct definition.
DoubleBag
is a bag containing aGECS
version number and aBagOfHolding
. The format version identifier is added/removed by save/load.- EstMethodParms
Array of character strings (pseudo-enum definition) used to identify parameters of estimation methods.
Different estimation methods will have different parameters. Interval type matters for asymmetrical distributions. Nested bootstrap replica count is used for bootstrap-t. An adaptive bootstrap may vary the replica count to achieve a precision specified as a numerical tolerance.
-
coverage probability
-
interval type
-
bootstrap replica count
-
nested bootstrap replica count
-
numerical tolerance of adaptive bootstrap
-
estimated attained precision of adaptive bootstrap
This is a non-prescriptive definition for example or default use. Each export file encapsulates its own definitions.
-
- EstMethods
Array of character strings (pseudo-enum definition) used to identify estimation methods.
-
original
-
bootstrap, percentile interval
-
bootstrap, BCa interval
-
bootstrap-t interval
This is a non-prescriptive definition for example or default use. Each export file encapsulates its own definitions.
-
- Experiment
Normative struct definition.
- id
-
Primary key.
- indVars
-
Array of factor identifiers (enum style).
- depVars
-
Array of output variable identifiers (enum style).
- description
-
Everything else as verbose text.
- InvTypes
Array of character strings (pseudo-enum definition) used to identify types of confidence intervals.
-
probabilistically symmetric
-
shortest
This is a non-prescriptive definition for example or default use. Each export file encapsulates its own definitions.
-
- Key
Normative struct definition.
A
Key
is used to retrieve either data or parameter estimates.- experimentId
-
References experiments.
- treat
-
Array of factor values ordered per indVars. For data, all values must be specified. For parms, nil works like a wildcard. E.g., for main effects, only one factor will have a specified level and all others will be nil. Factor values (levels) are not necessarily numeric.
- ParPod
(Private) Unit of parallelization.
- ParmData
Normative struct definition.
- est
-
Value (or, if necessary, an array of values).
- lo
-
Bounds are presumed inclusive unless infinite (or nil).
- hi
-
Bounds are presumed inclusive unless infinite (or nil).
- estMethodParms
-
Result-specific context (or nil).
- ParmDef
Normative struct definition.
A
ParmDef
is a “parameterized parameter” providing additional context to disambiguate alternative ways of estimating the parameter. Additional result-specific context can be included in theParmData
if necessary.- parm
-
Index into parms.
- estMethod
-
Index into estMethods.
- estMethodParms
-
Hash from estMethodParms index to values.
- Parms
Array of character strings (pseudo-enum definition) used to identify parameters.
Parameters are theoretically fixed but unknown metrics of a population that is bigger than the sample. They can only be estimated, and with estimation methods potentially being computationally expensive and complex, it may be important to preserve those estimates.
-
mean
-
standard deviation
-
variance
This is a non-prescriptive definition for example or default use. Each export file encapsulates its own definitions.
-
- Version
Integer constant indicating the version of
GECS
that has been loaded. There is no major/minor/patchlevel encoding. It just increments.
Public Class Methods
Add the following parameter to the results and effects of a specified experiment: mean, bootstrap method, BCa interval, 95% confidence, adaptive determination of bootstrap replica count to achieve the requested numerical tolerances for each depvar. The estimate provided is the sample mean, not the bootstrap estimate. Count will be at least 50000.
This function is parallelized. All available CPUs will be used to run bootstrap calculations in R.
- bagOfHolding
-
A
BagOfHolding
. - id
-
Experiment
id. - delta
-
Array of numbers specifying desired numerical tolerances for each depvar. To reduce variation below the resolution of a typical plot of height 1000 pixels, you'd want delta something like (max(y)-min(y))/2000 for whatever range of y is being plotted.
# File lib/GECS.rb, line 682 def GECS.bootstrapMeans(bagOfHolding,id,delta) require 'parallel' throw "Bag is nil" if bagOfHolding.nil? throw "Experiments are nil" if bagOfHolding.experiments.nil? throw "No such experiment" if id >= bagOfHolding.experiments.length exp = bagOfHolding.experiments[id] throw "Null experiment" if exp.nil? throw "Null indvars" if exp.indVars.nil? throw "Null depvars" if exp.depVars.nil? numfacs = exp.indVars.length numdeps = exp.depVars.length throw "Not enough indvars" if numfacs < 1 throw "Not enough depvars" if numdeps < 1 throw "Wrong number of deltas" if delta.length != numdeps parmDef = ParmDef.new(bagOfHolding.getOrAddParm("mean"), bagOfHolding.getOrAddEstMethod("bootstrap, BCa interval"), {bagOfHolding.getOrAddEstMethodParm("coverage probability")=>0.95}) meanId = bagOfHolding.getOrAddParmDef(parmDef) # Make a list of the data and effects that need fixing with a unique # serial number assigned to each. sernum = 0 pods = Array.new enumerateKeys(bagOfHolding,id).each{|key| for di in 0..numdeps-1 pods.push(ParPod.new(key,di,(sernum+=1),nil)) end } # Run numCPUs instances of BCa.R in parallel. pods = Parallel.map(pods) do |pod| poddata = refactorExtract(bagOfHolding,pod.key,pod.di) unless poddata.nil? descript = "Treatment " + pod.key.treat.to_s + " depvar " + pod.di.to_s fnam = "bootstrap-in-" + pod.sernum.to_s + ".txt" fp = File.open(fnam,"w") fp.puts(poddata) fp.close # This R script that has been mangled onto the command line to avoid # adding another external dependency is mostly just a wrapper for # BCa.R, but the bootstrap estimate of the mean is replaced by the # sample mean. cmd = "Rscript " + "-e 'library(\"bootBCa\")' " + "-e 'data <- unlist(read.table(\"" + fnam + "\",header=F,colClasses=\"numeric\"))' " + "-e 'out <- BCa(data,as.numeric(" + delta[pod.di].to_s + "),mean)' " + "-e 'cat(sprintf(\"%d %0.16f %0.16f %0.16f %0.16f\\n\",out[1],out[2],mean(data),out[4],out[5]))'" pod.out = `#{cmd}`.split File.delete(fnam) if pod.out.length < 5 # This should never happen. print "Bootstrap failure\n" print " ", descript, "\n" print " Depvar: ", bagOfHolding.experiments[id].depVars[pod.di], "\n" throw "Bootstrap failure" end # Verbose progress reporting print descript + ", " + pod.out[0] + " iterations done\n" end pod end # Copy back parameter estimates. bagOfHolding.ests ||= Hash.new pods.each{|pod| pd = parmRet(bagOfHolding,delta,pod) unless pd.nil? bagOfHolding.ests[pod.key] ||= Array.new(numdeps,nil) bagOfHolding.ests[pod.key][pod.di] ||= Hash.new bagOfHolding.ests[pod.key][pod.di][meanId] = pd end } end
Print out parameter metadata for an experiment.
# File lib/GECS.rb, line 420 def GECS.describeParms(bagOfHolding,id) experiment = bagOfHolding.experiments[id] ests = bagOfHolding.ests.select{|k,v| k.experimentId==id} ests.each{|k,v| print "Treatment " + k.treat.to_s + "\n" v.each_index{|di| print " Depvar ", experiment.depVars[di], "\n" if v[di].nil? print " Not applicable\n" else v[di].each{|pk,parmData| parmDef = bagOfHolding.parmDefs[pk] print " Parameter: ", bagOfHolding.parms[parmDef.parm], "\n" print " Estimation method: ", bagOfHolding.estMethods[parmDef.estMethod], "\n" print " Global estimation method parameters:", "\n" printEstMethodParms(bagOfHolding, parmDef.estMethodParms) print " Local estimation method parameters:", "\n" printEstMethodParms(bagOfHolding, parmData.estMethodParms) } end } } end
Dump an experiment's raw data (not parameter estimates) as an R table (with header) with a column for each dependent variable and N rows for each treatment. Short and missing series are padded with NAs.
# File lib/GECS.rb, line 311 def GECS.dumpData(bagOfHolding,id) throw "Bag is nil" if bagOfHolding.nil? throw "Experiments are nil" if bagOfHolding.experiments.nil? throw "No such experiment" if id >= bagOfHolding.experiments.length exp = bagOfHolding.experiments[id] raise "Experiment has no independent variables!" if exp.indVars.empty? raise "Experiment has no dependent variables!" if exp.depVars.empty? dump = quotesome(exp.indVars) + " " + quotesome(exp.depVars) + "\n" results = bagOfHolding.data.select{|k,v| k.experimentId==id} results.each{|k,cellarray| raise "Null treatment data" if cellarray.nil? raise "Bad treatment data" if cellarray.length != exp.depVars.length maxlen = cellarray.map{|cell| cell.nil? ? 1 : cell.length}.max for iteration in 0..maxlen-1 dump << quotesome(k.treat) cellarray.each{|cell| dump << " " + quotemaybe(cell.nil? ? nil : cell[iteration]).to_s } dump << "\n" end } dump end
Dump parameter estimates for an experiment as an R table (with header) with crudely constructed column names: depVar X parmDefId X [est, lo, hi, optionally prec]. Estimates are assumed to be scalars. If prec is true, add a prec column for each parameter containing the value of the estMethodParm “estimated attained precision of adaptive bootstrap”.
# File lib/GECS.rb, line 340 def GECS.dumpParms(bagOfHolding,id,prec=false) throw "Bag is nil" if bagOfHolding.nil? throw "Experiments are nil" if bagOfHolding.experiments.nil? throw "No such experiment" if id >= bagOfHolding.experiments.length experiment = bagOfHolding.experiments[id] ests = bagOfHolding.ests.select{|k,v| k.experimentId==id} parms = ests[ests.keys[0]][0].keys.sort precparm = bagOfHolding.getOrAddEstMethodParm("estimated attained precision of adaptive bootstrap") if prec dump = "# Key to parameter ID numbers:\n" parms.each{|x| parmDef = bagOfHolding.parmDefs[x] dump += "# " + x.to_s + " = " + bagOfHolding.parms[parmDef.parm] + ", " + bagOfHolding.estMethods[parmDef.estMethod] parmDef.estMethodParms.each{|k,v| dump += ", " + bagOfHolding.estMethodParms[k] + "=" + v.to_s } dump += "\n" } dump += quotesome(experiment.indVars) experiment.depVars.each{|d| parms.each{|p| dump += " \"" + d + " " + p.to_s + " est\"" + " \"" + d + " " + p.to_s + " lo\"" + " \"" + d + " " + p.to_s + " hi\"" dump += " \"" + d + " " + p.to_s + " prec\"" if prec } } dump += "\n" ests.each{|k,cells| dump += quotesome(k.treat) cells.each{|cell| parms.each{|p| if cell.nil? dump += " NA NA NA" dump += " NA" if prec else parm = cell[p] dump += " " + quotemaybe(parm.est).to_s + " " + quotemaybe(parm.lo).to_s + " " + quotemaybe(parm.hi).to_s if prec if parm.estMethodParms.nil? dump += " NA" else dump += " " + quotemaybe(parm.estMethodParms[precparm]).to_s end end end } } dump += "\n" } dump end
(Private) Make a list of the Keys for all treatments, main effects, and 2-way interactions for an experiment. An attempt is made to suppress interactions for which there are no data at all (combinations of levels that don't occur).
# File lib/GECS.rb, line 509 def GECS.enumerateKeys(bagOfHolding,id) require 'set' throw "Bag is nil" if bagOfHolding.nil? throw "Experiments are nil" if bagOfHolding.experiments.nil? throw "No such experiment" if id >= bagOfHolding.experiments.length exp = bagOfHolding.experiments[id] throw "Null experiment" if exp.nil? throw "Null indvars" if exp.indVars.nil? throw "Null depvars" if exp.depVars.nil? numfacs = exp.indVars.length numdeps = exp.depVars.length throw "Not enough indvars" if numfacs < 1 throw "Not enough depvars" if numdeps < 1 if numfacs==1 # Short cut for single-factor experiments. bagOfHolding.data.select{|k,v| k.experimentId==id}.keys else # Enumerate the levels of all of the factors while adding all of the # treatments. r = Array.new levels = Array.new(numfacs){Set.new} bagOfHolding.data.each_key{|k| if k.experimentId==id r.push(k) for fac in 0..numfacs-1 levels[fac].add(k.treat[fac]) end end } treat = Array.new(numfacs,nil) # Main effects. Single-factor experiments were already excluded. for fac in 0..numfacs-1 treat.fill(nil) for lvl in levels[fac] treat[fac] = lvl r.push(Key.new(id,Array.new(treat))) end end # 2-way interactions. if numfacs > 2 # Don't duplicate the treatments when numfacs==2. for fac1 in 0..numfacs-2 for fac2 in fac1+1..numfacs-1 treat.fill(nil) for lvl1 in levels[fac1] treat[fac1] = lvl1 for lvl2 in levels[fac2] treat[fac2] = lvl2 key = Key.new(id,Array.new(treat)) r.push(key) if matchesSomething(bagOfHolding,key) end end end end end r end end
Load a GECS
database from a file. Returns a BagOfHolding
.
# File lib/GECS.rb, line 290 def GECS.load(filename) temp = Marshal.load(File.open(filename,"r")) if temp.version > Version raise "File format version is later than GECS.rb version" end temp.bagOfHolding end
(Private) Return true if a given key matches any data at all.
# File lib/GECS.rb, line 491 def GECS.matchesSomething(bagOfHolding,key) id = key.experimentId bagOfHolding.data.each{|k,v| if k.experimentId==id if treatEq(key.treat,k.treat) unless v.nil? return true # Need to check every v[di] too? end end end } false end
Simplify creation of a new database by nilling out the enums. Since enums are looked up using a find-or-create pattern, there is no harm in starting with nils.
- experiments
-
Array of
Experiment
structs (index by experiment id). - data
-
Hash from
Key
to array (per depVars) of arrays (measurement values in chronological order).
# File lib/GECS.rb, line 304 def GECS.newBag(experiments,data) BagOfHolding.new(nil, nil, nil, nil, nil, experiments, data, nil) end
(Private) Create ParmData
from ParPod
return.
# File lib/GECS.rb, line 655 def GECS.parmRet(bagOfHolding,delta,pod) if pod.out.nil? nil else ParmData.new(pod.out[2].to_f, pod.out[3].to_f, pod.out[4].to_f, { bagOfHolding.getOrAddEstMethodParm("bootstrap replica count")=>pod.out[0].to_i, bagOfHolding.getOrAddEstMethodParm("numerical tolerance of adaptive bootstrap")=>delta[pod.di], bagOfHolding.getOrAddEstMethodParm("estimated attained precision of adaptive bootstrap")=>pod.out[1].to_f}) end end
(Private) Helper method for ::describeParms
.
# File lib/GECS.rb, line 445 def GECS.printEstMethodParms(bagOfHolding,estMethodParms) if estMethodParms.nil? puts " nil" else estMethodParms.each{|k,v| print " ", bagOfHolding.estMethodParms[k], " = ", v, "\n" } end end
(Private) Calculate the quick interval for the mean of some data. Returns a ParmData
.
# File lib/GECS.rb, line 591 def GECS.quickInterval(data) require 'statistics2' if data.nil? nil else count = data.length sum = data.reduce(:+) mean = sum.to_f/count variance = data.map{|x| (mean-x)**2}.inject(:+)/(count-1.0) meanU = Math.sqrt(variance/count)*Statistics2::ptdist(count-1,0.975) ParmData.new(mean,mean-meanU,mean+meanU,nil) end end
Add the following parameter to the data and effects of a specified experiment: mean, original method, 95% confidence. This is a quick way to summarize results when more complicated options are not needed. All values are computed as Floats with no respect for the original data type or its precision.
- bagOfHolding
-
A
BagOfHolding
. - id
-
Experiment
id.
# File lib/GECS.rb, line 613 def GECS.quickMeans(bagOfHolding,id) throw "Bag is nil" if bagOfHolding.nil? throw "Experiments are nil" if bagOfHolding.experiments.nil? throw "No such experiment" if id >= bagOfHolding.experiments.length exp = bagOfHolding.experiments[id] throw "Null experiment" if exp.nil? throw "Null indvars" if exp.indVars.nil? throw "Null depvars" if exp.depVars.nil? numfacs = exp.indVars.length numdeps = exp.depVars.length throw "Not enough indvars" if numfacs < 1 throw "Not enough depvars" if numdeps < 1 parmDef = ParmDef.new(bagOfHolding.getOrAddParm("mean"), bagOfHolding.getOrAddEstMethod("original"), {bagOfHolding.getOrAddEstMethodParm("coverage probability")=>0.95}) meanId = bagOfHolding.getOrAddParmDef(parmDef) bagOfHolding.ests ||= Hash.new enumerateKeys(bagOfHolding,id).each{|key| bagOfHolding.ests[key] ||= Array.new(numdeps,nil) for di in 0..numdeps-1 cell = refactorExtract(bagOfHolding,key,di) unless cell.nil? bagOfHolding.ests[key][di] ||= Hash.new bagOfHolding.ests[key][di][meanId] = quickInterval(cell) end end } end
(Private) Helper method to quote values that aren't numeric.
- oneval
-
A single value to be quoted or not. Nil becomes NA.
# File lib/GECS.rb, line 396 def GECS.quotemaybe(oneval) if oneval.nil? "NA" elsif oneval.is_a?(String) # R 3.0.2 looks like it is re-escaping strings on input so that \\ # turns into \\\\, yet this is the minimum amount of escaping that gets # everything through read.table without choking. allowEscapes=F only # makes it choke even more. The worst case seems to be when a string # ends with a backslash. "\""+oneval.gsub('\\'){'\\\\'}.gsub("\"","\\\"")+"\"" else oneval end end
(Private) Helper method to quote values that aren't numeric. Were they always well-behaved values, k.treat.join(“ ”) would suffice.
- treat
-
An array of values that might need to be quoted.
# File lib/GECS.rb, line 415 def GECS.quotesome(treat) treat.map{|level| quotemaybe(level)}.join(" ") end
(Private) Refactor the data of an experiment according to a specified effect and extract the data from the specified cell. If there are no nils in key, this is just a slow way of doing bagOfHolding.data[di].
- di
-
depvar index
# File lib/GECS.rb, line 474 def GECS.refactorExtract(bagOfHolding,key,di) id = key.experimentId r = nil bagOfHolding.data.each{|k,v| if k.experimentId==id if treatEq(key.treat,k.treat) unless v.nil? or v[di].nil? r ||= Array.new r.concat(v[di]) end end end } r end
Save a GECS
database to a file.
- bagOfHolding
-
A
BagOfHolding
.
# File lib/GECS.rb, line 285 def GECS.save(filename, bagOfHolding) Marshal.dump(DoubleBag.new(Version,bagOfHolding), open(filename,"w")) end
(Private) Equality test for treatments that implements nil as wildcard.
# File lib/GECS.rb, line 460 def GECS.treatEq(a,b) throw "Nil treatment passed to treatEq" if a.nil? or b.nil? throw "Length mismatch" if a.length != b.length a.each_index{|i| return false if !a[i].nil? and !b[i].nil? and a[i]!=b[i] } true end