Jake Thompson

We’re excited to announce the release of measr 0.2.1. The goal of measr is to provide applied researchers and psychometricians with a user friendly interface for estimating and evaluating diagnostic classification models (DCMs). DCMs are a class of psychometric models that provide classification of students into profiles of proficiency on a predefined set of knowledge, skills, or understandings.

measr 0.2.1

I’m stoked to announce the release of measr 0.3.1. The goal of measr is to provide applied researchers and psychometricians with a user friendly interface for estimating and evaluating diagnostic classification models (DCMs). This update is a minor release to introduce a couple of enhancements to model and prior specifications.

measr 0.3.1

An Application of <i>M</i><sub>2</sub> Statistic to Evaluate the Fit of Cognitive Diagnostic Models

Assessing Fit of Cognitive Diagnostic Models A Case Study

Obtaining Diagnostic Classification Model Estimates Using Mplus

Bayesian Psychometrics for Diagnostic Assessments: A Proof of Concept

measr: Bayesian psychometric measurement using
Stan

I am thrilled to announce the release of measr 1.0.0. The goal of measr is to provide a user-friendly interface for estimating and evaluating diagnostic classification models (DCMs; also called cognitive diagnostic models [CDMs]). This is a major release to mark the conclusion of the initial development work that was funded by the Institute of Education Sciences. Importantly, this does not mean that measr is going dormant!

measr 1.0.0

r-dcm blog

I am thrilled to announce the release of [measr](https://measr.info)
1.0.0. The goal of measr is to provide a user-friendly interface for
estimating and evaluating diagnostic classification models (DCMs; also
called cognitive diagnostic models \[CDMs\]). This is a major release to
mark the conclusion of the initial development work that was funded by
the [Institute of Education Sciences](https://ies.ed.gov). Importantly,
this does not mean that measr is going dormant! We are still actively
developing measr, so we'll continue to [add new features and respond to
bug reports](https://github.com/wjakethompson/measr/issues).

You can install the updated version of measr from CRAN with:

::: {#cb1 .sourceCode style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
install.packages("measr")
```
:::

This blog post will highlight the major improvements, including improved
documentation and vignettes, as well as a some minor updates.

:::: {.cell}
::: {#cb2 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
library(measr)
```
:::
::::

::: {#improved-documentation .section .level2}
## Improved Documentation {.anchored anchor-id="improved-documentation"}

The focus of recent work on measr has been to improve documentation. To
that end, the existing vignettes have been updated, and several new
vignettes have been added.

First, the [getting started](https://measr.info/articles/measr) vignette
has been updated to include additional installation information. This
should help reduce the friction for setting up the components needed for
measr, namely, installing [*Stan*](https://mc-stan.org). Because measr
interfaces with *Stan* to estimate DCMs, a working *Stan* installation
is needed for the package to function properly. The updated vignette
includes helpful guidance for installing both the
[rstan](https://mc-stan.org/rstan) and
[cmdstanr](https://mc-stan.org/cmdstanr) packages for using *Stan*. Only
one of the two is required, but both are supported so you can choose
whichever *Stan* front end you prefer.

A new vignette on [model
evaluation](https://measr.info/articles/model-evaluation) has been added
to demonstrate different methods for assessing the performance of a DCM
after it has been estimated. This includes the *M~2~* statistic [(Liu et
al., 2016)]{.citation cites="liu2016"}, as well as posterior predictive
model checks for evaluating both overall and item-level model fit
[(Sinharay & Almond, 2007; Thompson, 2019)]{.citation
cites="sinharay2007 thompson2019"}. To demonstrate how these methods are
implemented for measr, a simulated data set is used where we know how
well different types of DCMs should fit. The existing [model
estimation](https://measr.info/articles/model-estimation) vignette was
also updated to use this same simulated data set so that we can compare
parameter estimates derived from measr to the known parameter values
that were used to generate the data.

Because the model estimation and evaluation vignettes now use simulated
data, we also added a [case study](https://measr.info/articles/ecpe) to
walk through a DCM-based analysis from start to finish using a real data
set. In this case study, data from the *Examination for the Certificate
of Proficiency in English* (ECPE), which has been widely used in the DCM
literature [(e.g., Templin & Hoffman, 2013)]{.citation
cites="templin-emip-2013"}. We start with exploratory analyses, then we
estimate a DCM, examine parameter estimates, and interpret model fit and
reliability analyses.

Finally, example uses have been added for all functions included in
measr. These examples can be found on the documentation pages for each
function. All functions are indexed on the [reference
page](https://measr.info/reference/), which has been reorganized to
group functions with related functionality.
:::

::: {#minor-changes .section .level2}
## Minor Changes {.anchored anchor-id="minor-changes"}

There were also a number of minor improvements:

- The Stan code for estimating models has been updated to be compatible
  with the new array syntax.
- We published an article on measr in the *Journal of Open Source
  Software* [(Thompson, 2023)]{.citation cites="thompson-joss-2023"},
  and the [preferred citation](https://measr.info/authors#citation) when
  using measr has been updated.

For a complete list of changes, check out the
[changelog](https://measr.info/news/index.html#measr-100).
:::

::: {#additional-resources .section .level2}
## Additional Resources {.anchored anchor-id="additional-resources"}

If you are interested in learing more about measr and diagnostic models,
I taught a workshop in June at [StanCon
2023](https://mc-stan.org/events/stancon2023/). The workshop provided an
overview of diagnostic models and their use cases and then walked
through how to estimate and evaluate different models using measr. All
of the [workshop materials](https://stancon2023.measr.info/) are
available online, including slides, exercises, and solutions.

Finally, if you are interested in attending a workshop in person, I'll
be teaching another workshop in April at the 2024 annual meeting of the
National Conference on Measurement in Education (NCME). The materials
from this workshop will also be made available online for those who are
unable to attend. For more information on the conference and register,
see the [conference
website](https://www.ncme.org/event/annual-meeting/upcoming-meeting2024).
:::

::::::::::::::::: {#quarto-appendix .default}
:::: {#acknowledgments .section .level2 .appendix}
## Acknowledgments {.anchored .quarto-appendix-heading}

::: {.quarto-appendix-contents}
The research reported here was supported by the Institute of Education
Sciences, U.S. Department of Education, through Grant
[R305D210045](https://ies.ed.gov/funding/grantsearch/details.asp?ID=4546)
to the University of Kansas. The opinions expressed are those of the
authors and do not represent the views of the the Institute or the U.S.
Department of Education.

Featured photo by [patricia
serna](https://unsplash.com/@sernarial?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash)
on
[Unsplash](https://unsplash.com/photos/assorted-tape-measures-zPZ9vqqDNBA?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash).
:::
::::

::::::::: {#quarto-bibliography .section .quarto-appendix-contents}
## References {#references .anchored .quarto-appendix-heading}

:::::::: {#refs .references .csl-bib-body .hanging-indent entry-spacing="0" line-spacing="2"}
::: {#ref-liu2016 .csl-entry}
Liu, Y., Tian, W., & Xin, T. (2016). An application of
![](https://latex.codecogs.com/png.latex?M_2) statistic to evaluate the
fit of cognitive diagnostic models. *Journal of Educational and
Behavioral Statistics*, *41*(1), 3--26.
<https://doi.org/10.3102/1076998615621293>
:::

::: {#ref-sinharay2007 .csl-entry}
Sinharay, S., & Almond, R. G. (2007). Assessing fit of cognitive
diagnostic models. *Educational and Psychological Measurement*, *67*(2),
239--257. <https://doi.org/10.1177/0013164406292025>
:::

::: {#ref-templin-emip-2013 .csl-entry}
Templin, J., & Hoffman, L. (2013). Obtaining diagnostic classification
model estimates using Mplus. *Educational Measurement: Issues and
Practice*, *32*(2), 37--50. <https://doi.org/10.1111/emip.12010>
:::

::: {#ref-thompson2019 .csl-entry}
Thompson, W. J. (2019). *Bayesian psychometrics for diagnostic
assessments: A proof of concept* (Research Report 19-01). University of
Kansas; Accessible Teaching, Learning, and Assessment Systems.
<https://doi.org/10.35542/osf.io/jzqs8>
:::

::: {#ref-thompson-joss-2023 .csl-entry}
Thompson, W. J. (2023). [measr]{.nocase}: Bayesian psychometric
measurement using Stan. *Journal of Open Source Software*, *8*(91),
5742. <https://doi.org/10.21105/joss.05742>
:::
::::::::
:::::::::

::::::: {#quarto-citation .section .quarto-appendix-contents}
## Citation {#citation .anchored .quarto-appendix-heading}

<div>

::: {.quarto-appendix-secondary-label}
BibTeX citation:
:::

``` {.sourceCode .code-with-copy .quarto-appendix-bibtex}
@online{thompson2024,
  author = {Thompson, Jake},
  title = {Measr 1.0.0},
  date = {2024-02-05},
  url = {https://r-dcm.org/blog/2024-02-measr-1.0.0/},
  langid = {en}
}
```

::: {.quarto-appendix-secondary-label}
For attribution, please cite this work as:
:::

::: {#ref-thompson2024 .csl-entry .quarto-appendix-citeas}
Thompson, J. (2024, February 5). *measr 1.0.0*.
<https://r-dcm.org/blog/2024-02-measr-1.0.0/>
:::

</div>
:::::::
:::::::::::::::::


I'm stoked to announce the release of [measr](https://measr.info) 0.3.1.
The goal of measr is to provide applied researchers and psychometricians
with a user friendly interface for estimating and evaluating diagnostic
classification models (DCMs). This update is a minor release to
introduce a couple of enhancements to model and prior specifications.

You can install the update version of measr from CRAN with:

:::: {.cell layout-align="center"}
::: {#cb1 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
install.packages("measr")
```
:::
::::

This blog post will highlight the two enhancements included in the
update. For a complete list of changes, check out the
[changelog](https://measr.info/news/index.html#measr-031).

:::: {.cell layout-align="center"}
::: {#cb2 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
library(measr)
```
:::
::::

::: {#model-specifications .section .level2}
## Model Specifications {.anchored anchor-id="model-specifications"}

In this version, support for a new DCM subtype was added. The
compensatory reparameterized unified model (C-RUM) model can now be
specified in `measr_dcm()` with `type = "crum"`. The C-RUM is similar to
the log-linear cognitive diagnostic model (LCDM), except the C-RUM
estimates only item-level intercepts and main effects (i.e., no
interactions when multiple attributes are measured by an item). Because
of this, along with the addition of the C-RUM, the LCDM now has
additional flexibility through the new `max_interaction` argument. When
using `type = "lcdm"`, `max_interaction` specifies the highest level
interaction to be estimated. For example, setting `max_interaction = 2`
would estimate the LCDM with only intercepts, main effects, and two-way
interactions. If an item measures 3 or more attributes, the 3-way
interactions and higher will be excluded. Specifying `type = "lcdm"`
with `max_interaction = 1` is equivalent to the C-RUM, as 1 indicates
that main effects are the highest-level interaction to be estimated.

This version also introduces new options for the structural component of
the DCMs. Currently two attributes structures are possible, and are
defined through the `attribute_structure` argument. The first is
`attribute_structure = "unconstrained"`. This is the default, which
makes no assumptions about the relationships between attributes.
Specifying and unconstrained structural model is equivalent to the
saturated structural model described by [Hu & Templin
(2020)](https://doi.org/10.1080/00273171.2019.1632165) and in Chapter 8
of [Rupp et
al. (2010)](https://www.amazon.com/Diagnostic-Measurement-Applications-Methodology-Sciences/dp/1606235273).
The other option currently supported is
`attribute_structure = "independent"`. When an independent attribute
structure is specified, the proficiency or the presence of one attribute
is independent of other attributes. Future development will include
additional attribute structure specifications such as a reduced
loglinear model (e.g., [Thompson,
2018](https://dissertation.wjakethompson.com)) or a Bayesian Network
(e.g., [Hu & Templin,
2020](https://doi.org/10.1080/00273171.2019.1632165); [Martinez &
Templin, 2023](https://doi.org/10.31234/osf.io/pjc5f)).
:::

::::::::: {#prior-specifications .section .level2}
## Prior Specifications {.anchored anchor-id="prior-specifications"}

There are two main improvements to prior specifications included in this
release. First, custom prior distributions can be specified for the
structural model parameters. We can view the default parameters for each
attribute structure specification with:

:::: {.cell layout-align="center"}
::: {#cb3 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
default_dcm_priors(type = "lcdm", attribute_structure = "unconstrained")
#> # A tibble: 4 × 3
#>   class       coef  prior_def                  
#>   <chr>       <chr> <chr>                      
#> 1 intercept   <NA>  normal(0, 2)               
#> 2 maineffect  <NA>  lognormal(0, 1)            
#> 3 interaction <NA>  normal(0, 2)               
#> 4 structural  Vc    dirichlet(rep_vector(1, C))

default_dcm_priors(type = "lcdm", attribute_structure = "independent")
#> # A tibble: 4 × 3
#>   class       coef  prior_def      
#>   <chr>       <chr> <chr>          
#> 1 intercept   <NA>  normal(0, 2)   
#> 2 maineffect  <NA>  lognormal(0, 1)
#> 3 interaction <NA>  normal(0, 2)   
#> 4 structural  <NA>  beta(1, 1)
```
:::
::::

We can also view the specific parameters available for a specific model
using the `get_parameters()` function. For example, using the ECPE
Q-matrix, we can see the available parameters for each type of model.

:::: {.cell layout-align="center"}
::: {#cb4 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
library(tidyverse)

get_parameters(ecpe_qmatrix, attribute_structure = "unconstrained") |> 
  filter(class == "structural")
#> # A tibble: 1 × 4
#>   item_id class      attributes coef  
#>     <int> <chr>      <chr>      <glue>
#> 1      NA structural <NA>       Vc

get_parameters(ecpe_qmatrix, attribute_structure = "independent") |> 
  filter(class == "structural")
#> # A tibble: 3 × 4
#>   item_id class      attributes coef  
#>     <int> <chr>      <chr>      <glue>
#> 1      NA structural <NA>       eta[1]
#> 2      NA structural <NA>       eta[2]
#> 3      NA structural <NA>       eta[3]
```
:::
::::

The second improvement is additional checking of user-specified priors.
Specifically, `measr_dcm()` will now throw and error if we try to
specify a prior for a class or coefficient that is inconsistent with our
chosen DCM. For example, in the previous example of structural model
priors that there are different parameters depending on whether an
unconstrained or independent structure is specified. If we try to define
a prior for `eta[1]`, which is only relevant for an independent
structure, but an unconstrained structure is specified, we get an error.

:::: {.cell layout-align="center"}
::: {#cb5 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
measr_dcm(data = ecpe_data, qmatrix = ecpe_qmatrix,
          resp_id = "resp_id", item_id = "item_id",
          attribute_structure = "unconstrained",
          prior = prior(beta(5,17), class = "structural", coef = "eta[1]"))
#> Error in `abort_bad_argument()`:
#> ! Prior for parameter `eta[1]` with class `structural` is not relevant for the chosen model or specified Q-matrix. See `?get_parameters()` for a list of relevant parameters.
```
:::
::::

As always, please [open an
issue](https://github.com/wjakethompson/measr/issues) with any bugs or
feature requests for future development!
:::::::::

:::::::: {#quarto-appendix .default}
::::::: {#quarto-citation .section .quarto-appendix-contents}
## Citation {#citation .anchored .quarto-appendix-heading}

<div>

::: {.quarto-appendix-secondary-label}
BibTeX citation:
:::

``` {.sourceCode .code-with-copy .quarto-appendix-bibtex}
@online{thompson2023,
  author = {Thompson, Jake},
  title = {Measr 0.3.1},
  date = {2023-06-06},
  url = {https://r-dcm.org/blog/2023-06-measr-0.3.1/},
  langid = {en}
}
```

::: {.quarto-appendix-secondary-label}
For attribution, please cite this work as:
:::

::: {#ref-thompson2023 .csl-entry .quarto-appendix-citeas}
Thompson, J. (2023, June 6). *measr 0.3.1*.
<https://r-dcm.org/blog/2023-06-measr-0.3.1/>
:::

</div>
:::::::
::::::::


We're excited to announce the release of [measr](https://measr.info)
0.2.1. The goal of measr is to provide applied researchers and
psychometricians with a user friendly interface for estimating and
evaluating diagnostic classification models (DCMs). DCMs are a class of
psychometric models that provide classification of students into
profiles of proficiency on a predefined set of knowledge, skills, or
understandings. This offers many advantages over traditional assessment
models. Because results are reported as proficiency on individual
skills, teachers, students, and parents get actionable feedback about
which areas could use additional attention. However, these models have
not been widely used in applied or operational settings, in part due to
a lack of user friendly software for estimating and evaluating these
models. measr aims to bridge this gap between psychometric theory and
applied practice.

You can install it from CRAN with:

:::: {.cell layout-align="center"}
::: {#cb1 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
install.packages("measr")
```
:::
::::

This blog post will highlight the main features of the package.

:::: {.cell layout-align="center"}
::: {#cb2 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
library(measr)
```
:::
::::

::::::::::: {#estimating-dcms .section .level2}
## Estimating DCMs {.anchored anchor-id="estimating-dcms"}

To illustrate DCMs and their application with measr, we'll use a
simulated data set. In this data set, we have 1,000 respondents who each
answered 15 questions about addition, multiplication, and fractions. For
each item, a 1 represents a correct response to the item, and a 0
represents an incorrect response.

:::: {.cell layout-align="center"}
::: {#cb3 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
library(tidyverse)

dat <- read_csv("data/example-data.csv")
dat
#> # A tibble: 1,000 × 16
#>    resp_id item_01 item_02 item_03 item_04 item_05 item_06 item_07 item_08
#>      <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>   <dbl>
#>  1       1       1       1       1       0       1       1       0       1
#>  2       2       1       1       0       0       0       1       0       1
#>  3       3       1       1       0       0       0       1       0       1
#>  4       4       1       0       1       0       1       0       1       1
#>  5       5       1       1       0       0       0       1       1       0
#>  6       6       1       1       0       0       0       1       0       1
#>  7       7       1       1       1       0       0       1       1       1
#>  8       8       0       0       0       0       0       0       0       1
#>  9       9       0       1       0       0       0       1       0       0
#> 10      10       1       1       0       1       1       1       0       1
#> # ℹ 990 more rows
#> # ℹ 7 more variables: item_09 <dbl>, item_10 <dbl>, item_11 <dbl>,
#> #   item_12 <dbl>, item_13 <dbl>, item_14 <dbl>, item_15 <dbl>
```
:::
::::

When using DCMs, a Q-matrix defines which items measure each attribute,
or skill. In the Q-matrix, a 1 indicates that the item measures the
attributes, and a 0 indicates the item does not measure the attribute.

:::: {.cell layout-align="center"}
::: {#cb4 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
qmatrix <- read_csv("data/example-qmatrix.csv")
qmatrix
#> # A tibble: 15 × 4
#>    item_id addition multiplication fractions
#>    <chr>      <dbl>          <dbl>     <dbl>
#>  1 item_01        1              1         0
#>  2 item_02        1              0         0
#>  3 item_03        0              1         0
#>  4 item_04        0              0         1
#>  5 item_05        0              1         0
#>  6 item_06        1              0         0
#>  7 item_07        0              1         0
#>  8 item_08        0              1         1
#>  9 item_09        0              1         1
#> 10 item_10        0              0         1
#> 11 item_11        1              1         0
#> 12 item_12        1              0         1
#> 13 item_13        0              0         1
#> 14 item_14        0              1         1
#> 15 item_15        1              0         0
```
:::
::::

Our goal is to estimate whether each respondent is proficient in each of
the skills measured by the assessment, and DCMs are our tool. There are
many different types of DCMs, each with different constraints or
assumptions about how the attributes relate to each other. For example,
if an item measures multiple attributes, does a respondent need to be
proficient on all attributes to answer the item correctly? Or can
proficiency in one attribute compensate for the lack of proficiency on
another?

Also several DCM subtypes are supported by measr, we are primarily
focused on the [loglinear cognitive diagnostic
model](https://link.springer.com/article/10.1007/s11336-008-9089-5)
(LCDM), which is a general DCM. This means that it subsumes many of the
other DCM subtypes and allows for the data to determine how much (if at
all) one attribute might compensate for another.

measr works by wrapping the [*Stan*](https://mc-stan.org) language to
estimate DCMs. We can estimate the LCDM using `measr_dcm()`. This
function use the inputs to write a Stan script defining the model and
then estimates the model using either the
[rstan](https://mc-stan.org/rstan/) or
[cmdstanr](https://mc-stan.org/cmdstanr/) packages. To estimate the
model, we just supply our data set and the Q-matrix. Because our data
set and Q-matrix contain respondent and item identifiers, respectively,
we need to specify the names of the identifier columns. We can also
specify prior distributions for each of the model parameters. Finally,
we specify a file for saving the model once it is estimated. For more
information on model estimation, including details on specifying prior
distributions, see the [model estimation
vignette](https://measr.info/articles/model-estimation.html).

:::: {.cell layout-align="center"}
::: {#cb5 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
lcdm <- measr_dcm(data = dat, qmatrix = qmatrix,
                  resp_id = "resp_id", item_id = "item_id", 
                  prior = c(prior(normal(0, 2), class = "intercept"),
                            prior(lognormal(0, 1), class = "maineffect"),
                            prior(normal(0, 2), class = "interaction")),
                  file = "fits/example-lcdm")
```
:::
::::

Once the model has estimated, we can use `measr_extract()` to examine
different aspects of the model, such as proportion of respondents with
each pattern of proficiency across the attributes.

:::: {.cell layout-align="center"}
::: {#cb6 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
measr_extract(lcdm, "strc_param")
#> # A tibble: 8 × 2
#>   class          estimate
#>   <chr>        <rvar[1d]>
#> 1 [0,0,0]  0.165 ± 0.0127
#> 2 [1,0,0]  0.211 ± 0.0153
#> 3 [0,1,0]  0.048 ± 0.0089
#> 4 [0,0,1]  0.072 ± 0.0090
#> 5 [1,1,0]  0.166 ± 0.0164
#> 6 [1,0,1]  0.065 ± 0.0115
#> 7 [0,1,1]  0.084 ± 0.0116
#> 8 [1,1,1]  0.187 ± 0.0164
```
:::
::::
:::::::::::

::::::: {#evaluating-dcms .section .level2}
## Evaluating DCMs {.anchored anchor-id="evaluating-dcms"}

There are several functions included for evaluating the model. For
example we can examine the probability that each respondent is
proficient in each attribute. Here, respondent 1 is likely proficient in
all skills, whereas respondent 2 is likely only proficient in addition.

:::: {.cell layout-align="center"}
::: {#cb7 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
lcdm <- add_respondent_estimates(lcdm)
measr_extract(lcdm, "attribute_prob")
#> # A tibble: 1,000 × 4
#>    resp_id addition multiplication fractions
#>    <fct>      <dbl>          <dbl>     <dbl>
#>  1 1       0.997          0.977      0.920  
#>  2 2       1.00           0.00297    0.0102 
#>  3 3       1.00           0.133      0.0370 
#>  4 4       0.0228         1.00       0.0556 
#>  5 5       0.999          0.0253     0.0223 
#>  6 6       1.00           0.00297    0.00224
#>  7 7       0.998          0.469      0.938  
#>  8 8       0.000520       0.0233     0.0194 
#>  9 9       0.984          0.000116   0.00641
#> 10 10      0.997          0.920      0.957  
#> # ℹ 990 more rows
```
:::
::::

Often, we would dichotomize these probabilities into a 0/1 decision
(e.g., proficient/not proficient). We can also look at the reliability
of those classifications. The `add_reliability()` function will
calculate the classification consistency and accuracy metrics described
by [Johnson & Sinharay (2018)](https://doi.org/10.1111/jedm.12196). Here
we see that all of the attributes have fairly high accuracy and
consistency for classifications.

:::: {.cell layout-align="center"}
::: {#cb8 .sourceCode .cell-code style="background: #f1f3f5;"}
``` {.sourceCode .r .code-with-copy}
lcdm <- add_reliability(lcdm)
measr_extract(lcdm, "classification_reliability")
#> # A tibble: 3 × 3
#>   attribute      accuracy consistency
#>   <chr>             <dbl>       <dbl>
#> 1 addition          0.960       0.927
#> 2 multiplication    0.956       0.917
#> 3 fractions         0.936       0.882
```
:::
::::

As you can see, for each type of model evaluation, we follow the same
process of `add_{metric}` and then we can use `measr_extract()` to view
the results. For a complete list of model evaluation options, see
`?model_evaluation`.
:::::::

::: {#future-development .section .level2}
## Future Development {.anchored anchor-id="future-development"}

We're already in the process of adding features and making improvements
for the next version of measr. Some highlights include:

- Adding support for more DCM subtypes
- More refined prior specifications
- Additional vignettes including a complete description of model
  evaluation and example case studies

If you have a specific feature request, suggestion for improvement, or
run into any problems, please [open an
issue](https://github.com/r-dcm/measr/issues) on the project repository!
:::

:::::::::: {#quarto-appendix .default}
:::: {#acknowledgments .section .level2 .appendix}
## Acknowledgments {.anchored .quarto-appendix-heading}

::: {.quarto-appendix-contents}
The research reported here was supported by the Institute of Education
Sciences, U.S. Department of Education, through Grant
[R305D210045](https://ies.ed.gov/funding/grantsearch/details.asp?ID=4546)
to the University of Kansas. The opinions expressed are those of the
authors and do not represent the views of the the Institute or the U.S.
Department of Education.

Featured photo by
[Fleur](https://unsplash.com/@yer_a_wizard?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash)
on
[Unsplash](https://unsplash.com/photos/drafting-instruments-on-top-of-table-dQf7RZhMOJU?utm_content=creditCopyText&utm_medium=referral&utm_source=unsplash).
:::
::::

::::::: {#quarto-citation .section .quarto-appendix-contents}
## Citation {#citation .anchored .quarto-appendix-heading}

<div>

::: {.quarto-appendix-secondary-label}
BibTeX citation:
:::

``` {.sourceCode .code-with-copy .quarto-appendix-bibtex}
@online{thompson2023,
  author = {Thompson, Jake},
  title = {Measr 0.2.1},
  date = {2023-04-10},
  url = {https://r-dcm.org/blog/2023-04-measr-0.2.1/},
  langid = {en}
}
```

::: {.quarto-appendix-secondary-label}
For attribution, please cite this work as:
:::

::: {#ref-thompson2023 .csl-entry .quarto-appendix-citeas}
Thompson, J. (2023, April 10). *measr 0.2.1*.
<https://r-dcm.org/blog/2023-04-measr-0.2.1/>
:::

</div>
:::::::
::::::::::
