IntervalWAgg {aggreCAT} | R Documentation |
Aggregation Method: IntervalWAgg
Description
Calculate one of several types of linear-weighted best estimates where the weights are dependent on the lower and upper bounds of three-point elicitation (interval widths).
Usage
IntervalWAgg(
expert_judgements,
type = "IntWAgg",
name = NULL,
placeholder = FALSE,
percent_toggle = FALSE,
round_2_filter = TRUE
)
Arguments
expert_judgements |
A dataframe in the format of data_ratings. |
type |
One of |
name |
Name for aggregation method. Defaults to |
placeholder |
Toggle the output of the aggregation method to impute placeholder data. |
percent_toggle |
Change the values to probabilities. Default is |
round_2_filter |
Note that the IDEA protocol results in both a Round 1 and Round 2 set of probabilities for each claim. Unless otherwise specified, we will assume that the final Round 2 responses (after discussion) are being referred to. |
Details
The width of the interval provided by individuals may be an indicator of certainty, and arguably of accuracy of the best estimate contained between the bounds of the interval.
type
may be one of the following:
IntWAgg: Weighted according to the interval width across individuals for that claim, rewarding narrow interval widths. \[w\_Interval_{i,c}= \frac{1}{U_{i,c} - L_{i,c}}\] \[\hat{p}_c( IntWAgg) = \sum_{i=1}^N \tilde{w}\_Interval_{i,c}B_{i,c}\]
where \(U_{i,d} - L_{i,d}\) are individual \(i\)'s judgements for claim \(d\). Then
IndIntWAgg: Weighted by the rescaled interval width (interval width relative to largest interval width provided by that individual)
Because of the variability in interval widths between individuals across claims, it may be beneficial to account for this individual variability by rescaling interval widths across all claims per individual. This results in a re-scaled interval width weight, for individual \(i\) for claim \(c\), relative to the widest interval provided by that individual across all claims \(C\):
\[w\_nIndivInterval_{i,c}= \frac{1}{\frac{U_{i,c} - L_{i,c}}{max({ (U_{i,d} - L_{i,d}): d = 1,\dots, C})}}\] \[\hat{p}_c\left( IndIntWAgg \right) = \sum_{i=1}^N \tilde{w}\_nIndivInterval_{i,c}B_{i,c}\]AsymWAgg: Weighted by the asymmetry of individuals' intervals, rewarding increasing asymmetry.
We use the asymmetry of an interval relative to the corresponding best estimate to define the following weights:
\[w\_asym_{i,c}= \begin{cases} 1 - 2 \cdot \frac{U_{i,c}-B_{i,c}}{U_{i,c}-L_{i,c}}, \text{for}\ B_{i,c} \geq \frac{U_{i,c}-L_{i,c}}{2}+L_{i,c}\cr 1 - 2 \cdot \frac{B_{i,c}-L_{i,c}}{U_{i,c}-L_{i,c}}, \text{otherwise} \end{cases}\]then,
\[\hat{p}_c(AsymWAgg) = \sum_{i=1}^N \tilde{w}\_asym_{i,c}B_{i,c}.\]IndIntAsymWAgg: Weighted by individuals’ interval widths and asymmetry
This rewards both asymmetric and narrow intervals. We simply multiply the weights calculated
in the "AsymWAgg"
and "IndIntWAgg"
methods.
VarIndIntWAgg: Weighted by the variation in individuals’ interval widths
A higher variance in individuals' interval width across claims may indicate a higher responsiveness to the supporting evidence of different claims. Such responsiveness might be predictive of more accurate assessors. We define:
\[w\_varIndivInterval_{i}= var{(U_{i,c} - L_{i,c}): c = 1,\dots, C},\]where the variance (\(var\)) is calculated across all claims for individual \(i\). Then,
\[\hat{p}_c(VarIndIntWAgg) = \sum_{i=1}^N \tilde{w}\_varIndivInterval_{i}B_{i,c}\]KitchSinkWAgg: Weighted by everything but the kitchen sink
This method is informed by the intuition that we want to reward narrow and asymmetric intervals,
as well as the variability of individuals' interval widths (across their estimates). Again, we multiply
the weights calculated in the "AsymWAgg"
, "IndIntWAgg"
and "VarIndIntWAgg"
methods above.
Value
A tibble of confidence scores cs
for each paper_id
.
Examples
IntervalWAgg(data_ratings)