A worked example
As an example, we now create some new functions and show both how
they can be used interactively and how they can be integrated with the
makeDataReport() function. These new functions are:
- New
summaryFunctions:countZeros(): A newsummaryFunctionthat counts the number of occurrences of the value0in a variable. This function will be used in the summarize step ofmakeDataReport().meanSummary(): A newsummaryFunctionthat computes the mean of numerical variables. This function is also intended to be used in the summarize step.
- New
visualFunctions:mosaicVisual(): A newvisualFunctionthat produces mosaic plots. This function will be used in the visualize step ofmakeDataReport().prettierHist(): A newvisualFunctionthat makesggplot2histograms, but with contours around each bar (as is the default in thegraphicshistograms). This function will also be used in the visualize step.
- New
checkFunctions:isID(): A newcheckFunctionintended for use in the pre-check-stage. This function checks whether a variable consists exclusively of long (at least 8 characters/digits) entries that are all of equal length, as this might be personal identification codes that we do not wish to print out in the data summary.identifyColons()A newcheckFunctionthat flags variables in which some observations have colons that appear in between other characters. This is practical for identifying autogenerated interaction effects. This function will be used in the check step ofmakeDataReport().
These functions are defined in turn below, and afterwards, an example
of how they can be called from makeDataReport() is
provided.
summaryFunction examples: countZeros() and meanSummary()
We start by defining two summary functions. For both functions, we
use summaryResult() for creating the function output and
then we use summaryFunction() in order to add the functions
to the output from a allSummaryFunctions() call.
countZeros — a simple summaryFunction
First, we will make a summary function that counts how many times the
value 0 occurs in a variable - either as a character or as
a numeric value. We define this summaryFunction in the
following lines of code:
countZeros <- function(v, ...) {
val <- length(which(v == 0))
summaryResult(list(feature = "No. zeros", result = val, value = val))
}As this function computes an integer (the number of zeros), there is
no difference between the elements $result and
$value. If, on the other hand, the result had been a
character string, extra formatting might be required in the
$result entry (such as escaping of quotation marks), and in
this scenario, the two entries would have differed.
Because the result is returned as a summaryResult
object, a printing method is automatically called when
countZeros() is used interactively:
## No. zeros: 5## No. zeros: 5Note that letters is a globally defined vector
consisting of all the letters in the (English) alphabet as
characters.
We change the class of countZeros() in order to make it
appear in allSummaryFunctions() calls. Moreover, we wish to
emphasize that the function is not intended to be called on all variable
types, as zeros have different roles in Date and
logical variables:
countZeros <- summaryFunction(countZeros,
description = "Count number of zeros",
classes = c("character", "factor", "integer",
"labelled", "numeric"))Please note that this is just meta-information: The function can
still be called on Date and logical variables,
but then the user will know that they are acting against the
recommendations of the programmer. In order to control what input
variables can actually be used, we suggest writing the function as an S3
generic with methods for some, but not all, data classes, as the next
example will show.
meanSummary — an S3 generic summary function
In this example, we will use the concept of method dispatch and
generic S3 functions in order to control what data classes a
summaryFunction can be used for. If you are not familiar
with S3 classes, we refer to Hadley Wickham’s Advanced R for a
excellent, though somewhat computer science heavy, introduction to
object oriented programming in R.
We would like to define a very simple summary function that computes
the arithmetic mean of a variable. This is of course only meaningful for
numerical variables, i.e. variables with the classes
numeric,integerorlogical`. Therefore, we will
use the S3 framework to force users to only call our function on these
appropriate classes.
First, we define the generic function:
This is just an empty shell: All it does is to say that if the
function meanSummary is called, R should look for
class-specific methods called meanSummary. However, no such
methods are defined yet, and therefore an error is produced if the
function is called:
## Error in UseMethod("meanSummary"): no applicable method for 'meanSummary' applied to an object of class "c('double', 'numeric')"We now define a helper function that input a variable v,
removes NAs, computes the mean and outputs this mean as a
proper summaryResult:
meanSummaryHelper <- function(v, maxDecimals) {
#remove missing observations
v <- na.omit(v)
#compute mean and store "raw" output in `val`
val <- mean(v)
#store printable (rounded) output in `res`
res <- round(val, maxDecimals)
#output summaryResult
summaryResult(list(feature = "Mean", result = res, value = val))
}and we then assign this function as the meanSummary
method used for logical, numeric and
integer variables:
#logical
meanSummary.logical <- function(v, maxDecimals = 2) {
meanSummaryHelper(v, maxDecimals)
}
#numeric
meanSummary.numeric <- function(v, maxDecimals = 2) {
meanSummaryHelper(v, maxDecimals)
}
#integer
meanSummary.integer <- function(v, maxDecimals = 2) {
meanSummaryHelper(v, maxDecimals)
}and now we can see what happens when the function is called on supported - and non-supported - variables:
## Mean: 0.07#called on a character variable - produces error as there is
#no method for characters
meanSummary(letters)## Error in UseMethod("meanSummary"): no applicable method for 'meanSummary' applied to an object of class "character"Finally, we will make it a proper summaryFunction.
Because we defined meanSummary() as an S3 generic function,
we do not have to specify which classes it should be called up -
summmaryFunction() will automatically look up what classes
have methods and are thus supported:
And now it appears - together with countZeros - in a
allSummaryFunctions() call:
| name | description | classes |
|---|---|---|
| countZeros | Count number of zeros | character, factor, integer, labelled, numeric |
| meanSummary | Compute arithmetic mean | integer, logical, numeric |
| centralValue | Compute median for numeric variables, mode for categorical variables | character, Date, factor, integer, labelled, haven_labelled, logical, numeric |
| countMissing | Compute proportion of missing observations | character, Date, factor, integer, labelled, haven_labelled, logical, numeric |
| minMax | Find minimum and maximum values | integer, numeric, Date |
| quartiles | Compute 1st and 3rd quartiles | Date, integer, numeric |
| refCat | Find reference level | factor |
| uniqueValues | Count number of unique values | character, Date, factor, integer, labelled, haven_labelled, logical, numeric |
| variableType | Data class of variable | character, Date, factor, integer, labelled, haven_labelled, logical, numeric |
visualFunction examples: mosaicVisual() and prettierHist()
We will now define two different visual functions. The first one is a
graphics-based function, while the second one is
ggplot2-based.
mosaicVisual — a new visualFunction for categorical data
In the base R package graphics, a function
which produces mosaic plots is available. It is intended to be called on
one-way tables, as. e.g. returned from table(). This
function is named mosiacplot and results in the following
type of plots:
#construct a character variable by sampling 100 values that are
#either "a" (probability 0.3) or "b" (probability 0.7):
x <- sample(c("a", "b"), size = 100, replace = TRUE,
prob = c(0.3, 0.7))
#draw a mosaic plot of the distribution:
mosaicplot(table(x))
As evident from the figure, a mosaic plot consists of several
rectangles, one for each category in the data, and the area of each such
rectangle is proportional to the proportion of the data that takes that
value - much like a pie chart. We intend to use this function as part of
a new visualization function. We will define the new function such that
it gets the full dataMaid functionality. This can be done
using the following code which sets up the call using the existing
function mosaicplot().
mosaicVisual <- function(v, vnam, doEval) {
#Define a (unevaluated) call to mosaicplot
thisCall <- call("mosaicplot", table(v), main = vnam, xlab = "")
#if doEval is TRUE, evaluate the call, thereby producing a plot
#if doEval is FALSE, return the deparsed call
if (doEval) {
return(eval(thisCall))
} else return(deparse(thisCall))
}This function can now be called directly or used in
makeDataReport(), as will presented in an example below.
Depending on the doEval argument, either a text string with
code or a plot is produced. We can for instance inspect the code for the
plot produced in the above by calling mosaicVisual() on
x with doEval = FALSE:
## [1] "mosaicplot(structure(c(a = 22L, b = 78L), dim = 2L, dimnames = list("
## [2] " v = c(\"a\", \"b\")), class = \"table\"), main = \"Variable x\", "
## [3] " xlab = \"\")"Even though mosaicVisual(), as written above, follows
the style of a visualFunction, it is not yet truly one and
therefore, it will not appear in an allVisualFunctions()
call. In order to get this functionality, we need to change its object
class. This can be done by writing
mosaicVisual <- visualFunction(mosaicVisual,
description = "Mosaic plots using graphics",
classes = setdiff(allClasses(),
c("numeric",
"integer",
"Date")))Here, we use the function allClasses() to quickly obtain
a vector of all the seven variable classes supported by
dataMaid, and we use setdiff() to choose all
classes except the numeric, integer and
Date classes, as the mosaic plot is most suited for
categorical data. Note that if mosaicVisual() were an S3
generic function, this argument could have been left as
NULL and then the classes for which methods are available
would be added automatically by visualFunction().
prettierHist() — a customized ggplot2 histogram
Next, we will define a ggplot2-based plotting function.
We will simply make a slight alteration of the looks of the typical
ggplot2 histogram, but it serves of a general example of
how ggplot2 visual functions can be built.
But first, we define a helper function that does the actual plotting.
This makes it simpler to write a visual function using the
call()/eval() structure presented in the
above, thereby making it easy to provide a visual function that can both
be used interactively (with doEval = TRUE) and in
makeDataReport() (with doEval = FALSE). The
helper function is defined like this:
library(ggplot2)
prettierHistHelper <- function(v, vnam) {
#define a ggplot2 histogram
p <- ggplot(data.frame(v = v), aes(x = v)) +
geom_histogram(col = "white", bins = 20) +
xlab(vnam)
#return the plot
p
}We use col = "white" to add white contours around the
bars in the histogram. Let’s look at an example of this function called
on a variable consisting of 100 random draws from the standard normal
distribution:
Now, we can define a visual function that calls
prettierHistHelper():
#define visualFunction-style prettierHist()-function
prettierHist <- function(v, vnam, doEval = TRUE) {
#define the call
thisCall <- call("prettierHistHelper", v = v, vnam = vnam)
#evaluate or deparse
if (doEval) {
return(eval(thisCall))
} else return(deparse(thisCall))
}
#Make it a proper visualFunction:
prettierHist <- visualFunction(prettierHist,
description = "ggplot2 style histogram with contours",
classes = c("numeric", "integer", "logical", "Date"))We specify that the prettierHist() should only be called
on variables of the classes numeric, integer,
logical or Date, as is reasonable for a
histogram. And we will now find our new visualFunctions in
a allVisualFunctions()-call:
| name | description | classes |
|---|---|---|
| mosaicVisual | Mosaic plots using graphics | character, factor, labelled, haven_labelled, logical |
| prettierHist | ggplot2 style histogram with contours | numeric, integer, logical, Date |
| basicVisual | Histograms and barplots using graphics | character, Date, factor, integer, labelled, haven_labelled, logical, numeric |
| standardVisual | Histograms and barplots using ggplot2 | character, Date, factor, integer, labelled, haven_labelled, logical, numeric |
| tableVisual | Distribution tables | character, factor, labelled, haven_labelled |
checkFunction examples: isID() and identifyColons()
Next up is two new check functions. First, we will define a check
function that is intended for use in the pre-check stage, i.e. used for
screening variables for eligibility for the
summarize/visualize/check-steps. This will be a function that checks the
entries in a variable for a certain structure which often used in ID
type variables and we will name the new function isID().
Afterwards, we will define a check function that looks for
: in categorical variables, as these might be the result of
unintended interactions introduced in the data wrangling process.
isID — a new checkFunction without problem values
First, let’s define the isID() function. As this
function is not supposed to list problematic values, it falls within the
category of checkFunctions represented by
myFullVarCheckFunction() above. We do not particularly wish
to use this function interactively, so we will stick to the minimal
requirements of a checkFunction used in
check() (see the overview table in the beginning of this
vignette). We are interested in checking whether variables that are
neither logical nor Date are restricted to a
certain structure. In this example, we simply check if the all
observations, when converted to characters, have the same lengths and
that length is at least 8. This might be regarded as minimal
restrictions for a lot of different ID type information, including US
social security numbers and credit card numbers, and of course, a more
specific ID check function can be built as an expansion of
isID(). We define the function:
isID <- function(v, nMax = NULL, ...) {
#define minimal output. Note that this is not a
#proper checkResult
out <- list(problem = FALSE, message = "")
#only perform check if the variable is neither a logical nor a Date
if (class(v) %in% setdiff(allClasses(), c("logical", "Date"))) {
#count the number of characters in each entry of v
v <- as.character(v)
lengths <- c(nchar(v))
#check if all entries of v have at least 8 characters
#and whether they all have the same length. If so,
#flag as a problem.
if (all(lengths >= 8) & length(unique(lengths)) == 1) {
out$problem <- TRUE
out$message <- "Warning: This variable seems to contain ID codes."
}
}
#return result of the check
out
}This is essentially all we need to do in order to include this
function as a pre-check-function in makeDataReport().
However, we should note that e.g. not using checkResult()
for the output means that it does not have a convenient
print() method available; now, the output just really is
displayed as a list:
#define 9-character ID variable:
idVar <- c("1234-1233", "9221-0289",
"9831-1201", "6722-1243")
#check for ID resemblance for the ID variable
isID(idVar)## $problem
## [1] TRUE
##
## $message
## [1] "Warning: This variable seems to contain ID codes."## $problem
## [1] FALSE
##
## $message
## [1] ""As we have not changed the class of this check function to be a
checkFunction, it should be noted that it will not
show up in the output of a allCheckFunctions() call.
identifyColons — a new checkFunction with problem values
The last function we will define in this vignette is
identifyColons(). We define it using the helper function
messageGenerator() to obtain a properly escaped message,
and we use checkResult() to make its output print neatly.
As mentioned above, the purpose of this check function is to identify
variables where colons appear, as they might have been introduced by
mistake when loading or wrangling the data. In the code we use regular
expressions through the gregexpr() function to identify
colons that appear in between other characters:
identifyColons <- function(v, nMax = Inf, ... ) {
#remove duplicates (for speed) and missing values:
v <- unique(na.omit(v))
#Define the message displayed if a problem is found:
problemMessage <- "Note that the following values include colons:"
#Initialize the problem indicator (`problem`) and
#the faulty values (`problemValues`)
problem <- FALSE
problemValues <- NULL
#Identify values in v that has the structure: First something (.),
#then a colon (:), and then something again (.), i.e. values with
#non-trailing colons:
problemValues <- v[sapply(gregexpr(".:.", v),
function(x) all(x != -1))]
#If any problem values are identified, set the problem indicator
#accordingly
if (length(problemValues) > 0) {
problem <- TRUE
}
#Combine the problem indicator and the problem values
#into a problem status object that can be passed to
#the messageGenerator() helper function that will
#make sure the outputted message is properly escaped
#for inclusion in the dataMaid report
problemStatus <- list(problem = problem,
problemValues = problemValues)
outMessage <- messageGenerator(problemStatus, problemMessage, nMax)
#Output a checkResult with the problem, the escaped
#message and the raw problem values.
checkResult(list(problem = problem,
message = outMessage,
problemValues = problemValues))
}And we change the class of the function so that it becomes a proper
checkFunction object:
identifyColons <- checkFunction(identifyColons,
description = "Identify non-trailing colons",
classes = c("character", "factor", "labelled"))Note, that for checkFunctions, the description provided
in a checkFunction() call will appear in the report
produced by makeDataReport() (in the Data cleaning
summary section), so for this type of function, the change of class
is not only done for the sake of the allCheckFunctions()
output.
Let’s see the results of identifyColons() when used
interactively on a potentially problematic variable:
#define a variable as an interaction between between two factors:
iaVar <- factor(c("a", "b", "a", "c")):factor(c(1, 2, 3, 4))
#Check iaVar for colons:
identifyColons(iaVar)## Note that the following values include colons: a:1, a:3, b:2, c:4.And when used on a variable that does not contain colons:
## No problems found.Calling the new summarize/visualize/check functions from makeDataReport()
Now, we are ready to use all the new functions in a
makeDataReport() call. We wish to produce a report that
utilizes the newly defined functions in the following way:
- We want to add the new pre-check function,
isID(), to the already existing pre-checks. - We wish to change the plot type for
factorvariables to bemosaicVisual(), whilenumeric,integerandDatevariables should useprettierHist()for their visualizations. - We want the new summary function,
countZeros(), to be added to the summaries performed on all variable types except forDateandlogical. - We want the new summary function,
meanSummary(), to be used fornumeric,integerandlogicalvariables. - We want the new check function,
identifyColons(), to be added to the checks performed oncharacter,factorandlabelledvariables.
We will make a data report for the built-in dataset
artData that is based on the Master Works of Art
data from Data
Explorer. The dataset contains 200 observations (corresponding to
paintings) of 11 variables, describing different properties of the
paintings and their artists. The first 5 observations look like
this:
## ArtistID ArtistName NoOfMiddlenames
## 1 29da884a Georges Seurat 0
## 2 f055cbb8 Giacomo Balla 0
## 3 5323b3b8 Lucas Cranach the Elder 2
## 4 ca07b695 Hugo van der Goes 2
## 5 02161eeb Grant Wood 0
## Title Year
## 1 A Sunday Afternoon on the Island of La Grande Jatte 1886
## 2 Abstract Speed and Sound 1914
## 3 Adam and Eve in Paradise 1531
## 4 Adoration of the Kings 1470
## 5 American Gothic 1930
## Location Continent Width Height Media
## 1 Art Institute of Chicago North America 308.00 207.60 oil paint
## 2 Guggenheim Museum North America 76.52 54.61 oil paint
## 3 Gemaldegalerie Europe 35.50 50.40 oil paint
## 4 Gemaldegalerie Europe 242.00 147.00 oil paint
## 5 Art Institute of Chicago North America 62.40 74.30 oil paint
## Movement
## 1 Post-Impressionist:Neo-Impressionist:Pointillist
## 2 Futurism
## 3 German Renaissance
## 4 Northern Renaissance
## 5 RegionalistThe 11 variables of artData can be summarized as
follows:
ArtistID: A unique ID used for cataloging the artists (fictional)ArtistName: The name of the artistNoOfMiddleNames: The number of middle names the artist hasTitle: The title of the paintingYear: The approximate year in which the painting was madeLocation: The current location of the paintingContinent: The continent of the current location of the paintingWidth: The width of the painting, in centimetersHeight: The height of the painting, in centimetersMedia: The media/materials of the paintingMovement: The artistic movement(s) the painting belongs to
The options for the data report described above are specified as follows:
makeDataReport(artData,
#add extra precheck function
preChecks = c("isKey", "isSingular", "isSupported", "isID"),
#Add the extra summaries - countZeros() for character, factor,
#integer, labelled and numeric variables and meanSummary() for integer,
#numeric and logical variables:
summaries = setSummaries(
character = defaultCharacterSummaries(add = "countZeros"),
factor = defaultFactorSummaries(add = "countZeros"),
labelled = defaultLabelledSummaries(add = "countZeros"),
numeric = defaultNumericSummaries(add = c("countZeros", "meanSummary")),
integer = defaultIntegerSummaries(add = c("countZeros", "meanSummary")),
logical = defaultLogicalSummaries(add = c("meanSummary"))
),
#choose mosaicVisual() for categorical variables,
#prettierHist() for all others:
visuals = setVisuals(
factor = "mosaicVisual",
numeric = "prettierHist",
integer = "prettierHist",
Date = "prettierHist"
),
#Add the new checkFunction, identifyColons, for character, factor and
#labelled variables:
checks = setChecks(
character = defaultCharacterChecks(add = "identifyColons"),
factor = defaultFactorChecks(add = "identifyColons"),
labelled = defaultLabelledChecks(add = "identifyColons")
),
#overwrite old versions of the report, render to html and don't
#open the html file automatically:
replace = TRUE,
output = "html",
open = FALSE
)We have chosen the output to be html (output = "html")
so that it can easily be included in this vignette by use of the
includeHTML() function from the htmltools
package. We have set open = FALSE so that the outputted
html file is not opened automatically. The outputted report is available
at the end of this vignette.
