Model Performance and Tests

2020-08-31

library(JWileymisc)

Model Performance

modelPerformance() is a generic function that can be used to calculate performance metrics for a model. JWileymisc implements methods for lm class objects. The output is a named list, with a data table containing results. For linear models, current performance metrics include:

• AIC (Akaike Information Criterion)
• BIC (Bayesian Information Criterion)
• LL (Log Likelihood)
• LLDF (degrees of freedom for log likelihood)
• Sigma (residual standard deviation)
• R2 ($$R^2$$ variance accounted for in the sample)
• F2 (Cohen’s $$f^2$$ effect size, calculated as $$\frac{R^{2}}{1 - R^{2}}$$)
• AdjR2 (Sample size adjusted $$R^2$$, a better estimate of population variance accounted for)
• F (model F test)
• FNumDF (numerator degrees of freedom for model F test)
• FDenDF (denominator degrees of freedom for model F test)
• P (p value for model F test)

mtcars$cyl <- factor(mtcars$cyl)
m <- stats::lm(mpg ~ hp + cyl, data = mtcars)

mp <- modelPerformance(m)
print(mp)
#> $Performance #> Model N_Obs AIC BIC LL LLDF Sigma R2 F2 #> 1: lm 32 169.8964 177.2251 -79.94822 5 3.146243 0.7538578 3.062692 #> AdjR2 F FNumDF FDenDF P #> 1: 0.7274854 28.58513 3 28 1.13969e-08 #> #> attr(,"class") #> [1] "modelPerformance.lm" "modelPerformance" If only certain metrics are desired, these can be found by extracting the “Performance” list element and then the correct column from the data table.  ## Cohen's f^2 effect size mp$Performance[, F2]
#> [1] 3.062692

Another function, modelTest() is a generic providing a comprehensive series of tests for a model. Currently methods are implemented for both lm class models and vglm class models from the VGAM package with a multinomial family.

modelTest()


mt <- modelTest(m)
print(mt)
#> $FixedEffects #> Term Est LL UL Pval #> 1: (Intercept) 28.65011816 25.39768395 31.902552374 5.921199e-17 #> 2: hp -0.02403883 -0.05560048 0.007522814 1.299540e-01 #> 3: cyl6 -5.96765508 -9.32556307 -2.609747083 1.092089e-03 #> 4: cyl8 -8.52085075 -13.28559928 -3.756102224 1.028617e-03 #> #>$RandomEffects
#> [1] NA
#>
#> $EffectSizes #> Term N_Obs AIC BIC LL LLDF Sigma R2 #> 1: hp 0 -0.6675031 0.7982328 1.333752 1 -0.07685536 0.02139775 #> 2: cyl 0 -11.3421811 -8.4107093 7.671091 2 -0.71671885 0.15142046 #> F2 AdjR2 F FNumDF FDenDF P Type #> 1: 0.08693246 0.0134764 2.434109 1 28 0.129954045 Fixed #> 2: 0.61517476 0.1383002 8.612447 2 28 0.001215981 Fixed #> #>$OverallModel
#> \$Performance
#>    Model N_Obs      AIC      BIC        LL LLDF    Sigma        R2       F2
#> 1:    lm    32 169.8964 177.2251 -79.94822    5 3.146243 0.7538578 3.062692
#>        AdjR2        F FNumDF FDenDF           P
#> 1: 0.7274854 28.58513      3     28 1.13969e-08
#>
#> attr(,"class")
#> [1] "modelPerformance.lm" "modelPerformance"
#>
#> attr(,"class")
#> [1] "modelTest.lm" "modelTest"

APAStyler(mt)
#>                 Term                      Est          Type
#>  1:      (Intercept) 28.65*** [ 25.40, 31.90] Fixed Effects
#>  2:               hp    -0.02 [ -0.06,  0.01] Fixed Effects
#>  3:             cyl6  -5.97** [ -9.33, -2.61] Fixed Effects
#>  4:             cyl8  -8.52** [-13.29, -3.76] Fixed Effects
#>  5: N (Observations)                       32 Overall Model
#>  6:        logLik DF                        5 Overall Model
#>  7:           logLik                   -79.95 Overall Model
#>  8:              AIC                   169.90 Overall Model
#>  9:              BIC                   177.23 Overall Model
#> 10:               F2                     3.06 Overall Model
#> 11:               R2                     0.75 Overall Model
#> 12:           Adj R2                     0.73 Overall Model
#> 13:               hp      f2 = 0.09, p = .130  Effect Sizes
#> 14:              cyl      f2 = 0.62, p = .001  Effect Sizes

The model tests can also be used with interactions.


m2 <- stats::lm(mpg ~ hp * cyl, data = mtcars)

APAStyler(modelTest(m2))
#>                 Term                       Est          Type
#>  1:      (Intercept)  35.98*** [ 27.99, 43.98] Fixed Effects
#>  2:               hp    -0.11* [ -0.21, -0.02] Fixed Effects
#>  3:             cyl6   -15.31* [-30.59, -0.03] Fixed Effects
#>  4:             cyl8  -17.90** [-28.71, -7.09] Fixed Effects
#>  5:          hp:cyl6      0.11 [ -0.04,  0.25] Fixed Effects
#>  6:          hp:cyl8      0.10 [  0.00,  0.20] Fixed Effects
#>  7: N (Observations)                        32 Overall Model
#>  8:        logLik DF                         7 Overall Model
#>  9:           logLik                    -77.54 Overall Model
#> 10:              AIC                    169.08 Overall Model
#> 11:              BIC                    179.34 Overall Model
#> 12:               F2                      3.72 Overall Model
#> 13:               R2                      0.79 Overall Model
#> 14:           Adj R2                      0.75 Overall Model
#> 15:               hp       f2 = 0.23, p = .021  Effect Sizes
#> 16:              cyl       f2 = 0.47, p = .007  Effect Sizes
#> 17:           hp:cyl       f2 = 0.16, p = .142  Effect Sizes