Content
Description
These are the power analyses for the intervention study “Using an Intelligent Tutoring System within a Task-Based Learning Approach in English Classes to Foster Motivation and Learning Outcome (Interact4School).” Please see https://doi.org/10.23668/psycharchives.5366 for the pre-registration of, and further information on, this study.
We ran all power analyses under the constraint to achieve a power of 80% and we set the alpha level to 5% (for two-tailed testing). Additional assumptions are described per intervention design, respectively.
Registry 1 (School-level intervention)
Design: Three-level blocked cluster-level random assignment design (students nested in classes, nested in schools, blocked in federal states) with treatment at Level 3
Unit of random assignment: School
Conditions: “FeedBook condition” vs. “Waitlist control condition”
Outcome: Students’ English proficiency
Assumptions:
- for the intraclass correlation at the class level, we assumed a range between \(\rho\) = .15 and .07
- for the intraclass correlation at the school level, we assumed a range between \(\rho\) = .25 and .02
- we assumed that 50% of the variance at both the individual and class levels was explained by a pretest measure used as a covariate
- we assumed that 90% of the variance at the school level was explained by a pretest measure used as a covariate
- we planned to assign schools to the FeedBook condition (vs. the waitlist control condition) with a probability of 0.67
# Conservative
mdes.1.min <- mdes.bcra4f3(power = .80, alpha = .05, two.tailed = TRUE,
rho2 = .15, rho3 = .25,
p = .67, r21 = .50, r22 = .50, r23 = .90, g3 = 1,
n = 24, J = 3, K = 8, L = 2)##
## Minimum detectable effect size:
## ---------------------------------------
## 0.381 95% CI [0.108,0.653]
## ---------------------------------------
## Degrees of freedom: 11
## Standardized standard error: 0.124
## Type I error rate: 0.05
## Type II error rate: 0.2
## Two-tailed test: TRUE
# Optimistic
mdes.1.max <- mdes.bcra4f3(power = .80, alpha = .05, two.tailed = TRUE,
rho2 = .07, rho3 = .02,
p = .67, r21 = .50, r22 = .50, r23 = .90, g3 = 1,
n = 24, J = 3, K = 8, L = 2)##
## Minimum detectable effect size:
## ---------------------------------------
## 0.231 95% CI [0.066,0.397]
## ---------------------------------------
## Degrees of freedom: 11
## Standardized standard error: 0.075
## Type I error rate: 0.05
## Type II error rate: 0.2
## Two-tailed test: TRUE
Registry 2 (Class-level intervention)
Design: Three-level blocked cluster-level random assignment design (students nested in classes, nested in schools, blocked in federal states) with treatment at Level 2
Unit of random assignment: Class
Conditions: “Original FeedBook condition” vs. “Criterial feedback condition” vs. “Criterial feedback & motivational elements condition”
Outcome: Students’ Motivation
Assumptions:
- for the intraclass correlation at the class level, we assumed a value of \(\rho\) = .05
- for the intraclass correlation at the school level, we assumed a value of \(\rho\) = .005
- we assumed a treatment effect heterogeneity of \(\omega\) = .002
- we assumed that 50% of the variance at the individual level was explained by a pretest measure used as a covariate
- we assumed that between 30% and 70% of the variance at the class level was explained by a pretest measure used as a covariate
- we assumed that 90% of the treatment effect variance among schools was explained by a pretest measure used as a covariate
- we planned to assign classes to each one of the three conditions with as probability of 0.33, respectively
Only the schools in the FeedBook condition are considered (i.e., 2/3 of the initial schools).
Additionally, the following analyses are true for the comparison of each two conditions (e.g., Original FeedBook condition vs. Criterial feedback condition); the expected number of classes within schools thus had to be reduced to 2/3 for the purposes of these analyses, and, consequently, the probability of being assigned to one out two conditions was set to 0.50.
# Conservative
mdes.2.min <- mdes.bcra3r2(power = .80, alpha = .05, two.tailed = TRUE,
rho2 = .05, rho3 = .005, omega3 = .002,
p = .50, r21 = .50, r22 = .30, r2t3 = .90, g3 = 1,
n = 24, J = 2, K = 12)##
## Minimum detectable effect size:
## ---------------------------------------
## 0.297 95% CI [0.084,0.509]
## ---------------------------------------
## Degrees of freedom: 10
## Standardized standard error: 0.095
## Type I error rate: 0.05
## Type II error rate: 0.2
## Two-tailed test: TRUE
# Optimistic
mdes.2.max <- mdes.bcra3r2(power = .80, alpha = .05, two.tailed = TRUE,
rho2 = .05, rho3 = .005, omega3 = .002,
p = .50, r21 = .50, r22 = .70, r2t3 = .90, g3 = 1,
n = 24, J = 2, K = 12)##
## Minimum detectable effect size:
## ---------------------------------------
## 0.236 95% CI [0.067,0.406]
## ---------------------------------------
## Degrees of freedom: 10
## Standardized standard error: 0.076
## Type I error rate: 0.05
## Type II error rate: 0.2
## Two-tailed test: TRUE
Registry 3 (Individual-level intervention)
Design: Three-level blocked individual-level random assignment design (students nested in classes, nested in schools, blocked in federal states)
Unit of random assignment: Student
Conditions: “Group A” vs. “Group B”
Outcome: Students’ English proficiency
Assumptions:
- for the intraclass correlation at the class level, we assumed a range between \(\rho\) = .15 and .07
- for the intraclass correlation at the school level, we assumed a range between \(\rho\) = .25 and .02
- at the class level, we assumed a treatment effect heterogeneity of \(\omega\) = .05
- at the school level, we assumed a treatment effect heterogeneity between \(\omega\) = .05 and \(\omega\) = .02
- we assumed that 50% of the variance at the individual level was explained by a pretest measure used as a covariate
- we assumed that 25% of the treatment effect variance among classes was explained by a pretest measure used as a covariate
- we assumed that 25% of the treatment effect variance among schools was explained by a pretest measure used as a covariate
- we planned to assign individual students to Group A (vs. Group B) with a probability of 0.50
Only the schools in the FeedBook condition are considered (i.e., 2/3 of the initial schools).
# conservative
mdes.3.min <- mdes.bira3r1(power = .80, alpha = .05, two.tailed = TRUE,
rho2 = .07, rho3 = .02, omega2 = .05, omega3 = .05,
p = .50, r21 = .50, r2t2 = .25, r2t3 = .25, g3 = 1,
n = 24, J = 2, K = 32)##
## Minimum detectable effect size:
## ---------------------------------------
## 0.102 95% CI [0.03,0.175]
## ---------------------------------------
## Degrees of freedom: 30
## Standardized standard error: 0.035
## Type I error rate: 0.05
## Type II error rate: 0.2
## Two-tailed test: TRUE
# optimistic
mdes.3.max <- mdes.bira3r1(power = .80, alpha = .05, two.tailed = TRUE,
rho2 = .15, rho3 = .25, omega2 = .05, omega3 = .02,
p = .50, r21 = .50, r2t2 = .25, r2t3 = .25, g3 = 1,
n = 24, J = 2, K = 32)##
## Minimum detectable effect size:
## ---------------------------------------
## 0.091 95% CI [0.027,0.155]
## ---------------------------------------
## Degrees of freedom: 30
## Standardized standard error: 0.031
## Type I error rate: 0.05
## Type II error rate: 0.2
## Two-tailed test: TRUE