Modeling Example
In the following, we will demonstrate parametric continous norming based on the beta-binomial distribution using the example of the development of receptive vocabulary in childhood and adolescence. For this purpose, we have integrated a data set into cNORM that was collected using the German version of the Peabody Picture Vocabulary Test 4 (PPVT-4). If you would like to reproduce the modeling yourself please make sure that you have installed a current version of cNORM.
Data Preparation
With the following code you can load the cNORM library and view the structure of the ppvt data set:
## Loads the package and displays the data
library(cNORM)
#> Good morning star-shine!
#> cNORM is free software. Please report bugs: https://github.com/WLenhard/cNORM/issues
str(ppvt)
#> 'data.frame': 4542 obs. of 6 variables:
#> $ age : num 2.6 2.6 2.62 2.86 2.88 ...
#> $ sex : num 1 1 1 1 1 1 1 1 1 1 ...
#> $ migration: num 0 0 0 0 0 0 0 0 0 0 ...
#> $ region : chr "west" "west" "west" "south" ...
#> $ raw : num 120 67 23 50 44 55 69 38 61 72 ...
#> $ group : num 3.16 3.16 3.16 3.16 3.16 ...
plot(ppvt$age, ppvt$raw, main="PPVT Raw Scores by Age",
xlab="Age", ylab="Raw score")
The dataset contains two columns that encode the explanatory variable: a continuous age variable and a discrete group variable (see also Data Preparation). The latter is primarily used for visualization purposes in parametric norming. Furthermore, the dataset includes the raw score and three background variables that can be used for stratification or weighting, namely sex, migration, and region.
The ‘Raw Scores by Age’ plot exhibits a pronounced curvilinear pattern, which is characteristic of vocabulary development. A rapid increase in scores is observed during the preschool years, followed by a deceleration of growth during adolescence.
We will initially conduct the modeling without stratification or weighting.
Model Fitting
To fit the beta-binomial distribution to the data, we employ the ‘cnorm.betabinomial()’ function. You have to specify the variables ‘age’ for the explanatory variable and ‘score’ for the raw score to be modeled. Additionally, it is necessary to indicate the number of items, which for the PPVT-4, is 228. Note that it is important to employ the actual number of items here, not the maximum achievable score.
# Models the data across a continuos explanatroy variable such as age,
# thereby assuming that the raw scores follow a beta-binomial distribution
# at a given age level:
model.betabinomial <- cnorm.betabinomial(age = ppvt$age, score = ppvt$raw, n = 228)
Fit indices can be obtained using the ‘diagnostic.betabinomial()’ function or the ‘summary()’ function. If the age variable and raw score variable are additionally specified, the output will also include R2, RMSE, and bias in relation to the observed data.
# Provides fit indices
diagnostics.betabinomial(model.betabinomial, age = ppvt$age, score = ppvt$raw)
#> $type
#> [1] "cnormBetaBinomial2"
#>
#> $converged
#> [1] TRUE
#>
#> $n_evaluations
#> function
#> 77
#>
#> $n_gradient
#> gradient
#> 77
#>
#> $final_value
#> [1] 19920.32
#>
#> $message
#> [1] "CONVERGENCE: REL_REDUCTION_OF_F <= FACTR*EPSMCH"
#>
#> $AIC
#> [1] 39856.65
#>
#> $BIC
#> [1] 39908.01
#>
#> $log_likelihood
#> [1] -19920.32
#>
#> $max_gradient
#> [1] NA
#>
#> $n_params
#> [1] 8
#>
#> $n_obs
#> [1] 4542
#>
#> $param_estimates
#> alpha_0 alpha_1 alpha_2 alpha_3 beta_0 beta_1
#> 2.97720857 0.10471760 -0.08619017 0.07852854 1.85645572 -0.44910542
#> beta_2 beta_3
#> 0.04655368 0.05662815
#>
#> $param_se
#> alpha_0 alpha_1 alpha_2 alpha_3 beta_0 beta_1 beta_2
#> 0.03272454 0.04498795 0.02469288 0.01847899 0.03179832 0.04478740 0.02385909
#> beta_3
#> 0.01839787
#>
#> $z_values
#> alpha_0 alpha_1 alpha_2 alpha_3 beta_0 beta_1 beta_2
#> 90.977863 2.327681 -3.490487 4.249612 58.382189 -10.027494 1.951192
#> beta_3
#> 3.077974
#>
#> $p_values
#> alpha_0 alpha_1 alpha_2 alpha_3 beta_0 beta_1
#> 0.000000e+00 1.992905e-02 4.821404e-04 2.141411e-05 0.000000e+00 0.000000e+00
#> beta_2 beta_3
#> 5.103415e-02 2.084134e-03
#>
#> $R2
#> [1] 0.9550196
#>
#> $rmse
#> [1] 2.113115
#>
#> $bias
#> [1] 0.1586252By default, the function fits a third-degree polynomial for both \(\alpha\) and \(\beta\). However, you can also specify the degrees of the polynomials yourself. Use the parameters ‘alpha’ and ‘beta’ for this purpose. When the ‘mode’ parameter of the function is set to 1, the adjustment is performed using the parameters for mean and standard deviation (\(\mu\) and \(\sigma\)) instead of \(\alpha\) and \(\beta\). If the number of items is not specified using the ‘n’ parameter, the function will use the maximum raw score (which we do not recommend).
Norm scores are returned as T-scores by default, but IQ scores, z-scores, percentiles, or customized norm scores specified by a vector ‘c(M, SD)’ are also possible.
By default, a plot of the manifest and modeled percentile curves is returned, too. Diagramms can also be generated using the ‘plot()’ function. As with distribution-free modeling, it is always important to have a close look at the percentile curves. Wavy patterns indicate overfitting.
To determine norm scores, please use the ‘predict()’ function and provide the model object along with a vector for the desired age levels and raw scores. Please use the midpoint of the desired age interval to specify age. In the example, T-scores for 10-year-old children are provided in 3-month intervals for the raw score 200.
Post stratification and weighting
Post-stratification is a method in which individual cases are weighted after data collection is complete, aiming to minimize deviations from representativeness in the sample. In parametric modeling, such weights can be applied in the same way as in distribution-free modeling. The ‘computeWeights()’ function calculates the necessary weights using a technique called ‘raking’. In this example, we weight individual cases to account for population proportions of gender and immigration status when computing the model. Consequently, the resulting norm scores should better represent the target population compared to the unweighted model.
# Calculates weights and models the data:
margins <- data.frame(variables = c("sex", "sex",
"migration", "migration"),
levels = c(1, 2, 0, 1),
share = c(.52, .48, .7, .3))
weights <- computeWeights(ppvt, margins)
#> Raking converged normally after 3 iterations.
model_weighted <- cnorm.betabinomial(ppvt$age, ppvt$raw, weights = weights)
#> n parameter not specified, using the maximum score in the data instead. Consider to provide n manually.
These norm tables provide the raw scores (x), their probabilities (Px), the cumulative probabilities (Pcum), percentiles, z-scores, and norm scores (if not specified otherwise: T-scores) and in this specific case the 95%-confidence intervals with respect to the norm score and percentile.
Please consult the vignette on WeightedRegression for
more information on post-stratification and weighting techniques.
Generation of Norm tables
Norm tables can be generated using the ‘normTable.betabinomial()’ function. This function requires the model object and an age specification or a vector with multiple age specifications for which you want to generate norm tables. When specifying ages, please note that the midpoint of the desired age interval should be used. In this example, norms for 14-year-old children are provided in 6-month intervals. If a confidence coefficient and a reliability coefficient are additionally specified in the ‘normTable.betabinomial()’ function, the table will be returned with confidence intervals. By default, the function limits the norm scores to +/- 3 standard deviations.
# Generates norm tables for age 14.25 and 14.75 and computes 95%-confidence
# intervals with a reliability of .97.
tables <- normTable.betabinomial(model.betabinomial, c(14.25, 14.75), CI = .95, reliability = .97)
head(tables[[1]]) # head is used to show only the first few rows of the table
#> x Px Pcum Percentile z norm lowerCI upperCI lowerCI_PR
#> 1 0 9.672759e-29 9.672759e-29 4.836379e-27 -3 20 17.55655 24.24345 0.05886057
#> 2 1 2.125895e-27 2.222623e-27 1.159675e-25 -3 20 17.55655 24.24345 0.05886057
#> 3 2 2.440727e-26 2.662990e-26 1.442626e-24 -3 20 17.55655 24.24345 0.05886057
#> 4 3 1.948144e-25 2.214443e-25 1.240371e-23 -3 20 17.55655 24.24345 0.05886057
#> 5 4 1.214127e-24 1.435571e-24 8.285076e-23 -3 20 17.55655 24.24345 0.05886057
#> 6 5 6.292131e-24 7.727702e-24 4.581636e-22 -3 20 17.55655 24.24345 0.05886057
#> upperCI_PR
#> 1 0.5002518
#> 2 0.5002518
#> 3 0.5002518
#> 4 0.5002518
#> 5 0.5002518
#> 6 0.5002518These norm tables provide the raw scores (x), their probabilities (Px), cumulative probabilities (Pcum), percentiles, z-scores, and norm scores (in this case, T-scores) and in this specific case the confidence intervals for the norm score and the percentiles for each age. This allows for the interpretation of an individual’s raw score in relation to their age group.
Simulation Study
In simulation studies (W. Lenhard & Lenhard, 2021), we demonstrated that distribution-free continuous norming using Taylor polynomials, as applied in cNORM, is significantly superior to conventional norming methods. This superiority was more pronounced with smaller normative samples. Only with unrealistically large samples did the relevant difference between continuous and conventional norming disappear. In another simulation study (A. Lenhard et al., 2019), we compared distribution-free with parametric norming. These investigations showed no general superiority of one method over the other. Instead, the quality of norming depended on the raw score distribution and sample size. A crucial factor for norming quality was whether an appropriate distribution could be found in parametric norming that fit the raw score distributions within each individual age group. However, the beta-binomial distribution was not included in our simulation study at that time.
Recent publications by Urban et al. (2024) have now shown that parametric modeling using the beta-binomial distribution can very effectively approximate the norm scores of some typical psychological test scales. Our own (unpublished) simulations with 1-PL IRT scales also pointed in this direction.
With the simulation study presented below, we aim to extend our previous results comparing parametric and distribution-free modeling to include the beta-binomial distribution. The simulations exclusively modeled scales without time limitations (i.e., without speed effects). In theory, the beta-binomial distribution should be particularly well-suited for modeling these scales.
Our simulation includes scales of varying difficulty and item numbers, and normative samples of different sizes.
We conducted five simulation runs with the following parameters:
Number of items: 10, 30, and 50
Scale difficulty: easy, medium, and hard
Sample sizes: 700, 1,400, 2,800, and 5,600 cases across 7 age groups
Cross-validation sample: 70,000 cases
In the results, we compare conventional norming with 6-month intervals, distribution-free continuous norming, and parametric continuous norming by means of beta-binomial distribution. The resulting norm scores for each method are compared with manifest T-scores simulated for an ideally representative, large population using an IRT process. This approach allows us to assess model quality while minimizing biases that occur in real norming studies. Since the person parameters are known in the simulation, we can calculate the mean bias, Root Mean Square Error (RMSE), R2, and Mean Absolute Difference (MAD) for each norming method. For simplicity, we will focus on R2 in the cross-validation sample in the following. However, detailed results and the complete simulation code are available in our OSF repository.
Results and discussion
```markdown 
Our results demonstrate that parametric continuous norming with beta-binomial distribution performs exceptionally well for the simulated scales. Even with relatively small samples, it closely approximates the ‘ideal’ norm scores simulated for the large and representative cross-validation sample. Parametric norming with the beta-binomial distribution is markedly superior to conventional norming with 6-month intervals. In fact, for the simple 1-PL scales simulated here, it performs even slightly better than distribution-free continuous norming with Taylor polynomials. However, the differences between the two continuous methods are so minimal that they are unlikely to have practical significance in most cases.
Moreover, based on the results presented here, it cannot be assumed that parametric norming using the beta-binomial distribution generally yields better results than distribution-free continuous norming. At least in unsystematic simulations for individual age groups, we found a better fit with Taylor polynomials for speeded test scales. We are currently preparing additional systematic simulations, the results of which we will report as soon as possible.
Conclusion: When to Use What?
A diverse methodological toolkit is invaluable in psychometrics. Taylor polynomials, as implemented in cNORM, have proven effective in a wide range of norming scenarios. Their distribution-free nature makes them applicable to various tests, including those with continuous or negative raw scores. Parametric modeling with beta-binomial distribution, on the other hand, is particularly well-suited for unspeeded tests with dichotomous items and fixed maximum scores. Both methods yield significantly better results than conventional norming for typical sample sizes and should therefore be preferentially applied in test construction.
The choice between these approaches should be guided by the following factors:
- Characteristics of raw scores (e.g., discrete vs. continuous raw scores, negative values)
- Test scale features (e.g., speeded vs. unspeeded)
Regardless of the method used, it is always necessary to thoroughly evaluate the model fit and transparently report its quality in the test manuals. Particular attention should be paid to the shape of the percentile curves, which should align with theoretical expectations. For instance, wavy percentile curves are generally not expected. Such a pattern indicates overfitting and should be avoided. In such cases, review the model parameters or employ an alternative modeling approach.
In summary, both distribution-free and parametric continuous norming with cNORM offer significant advantages over conventional norming. The specific choice between the two continuous norming methods should be made through careful consideration of the test type and available data.