Bayes Factor Design Analyses ‘Re-Building Trust’

Design

We realize an intervention design with three conditions. Two if which will be randomly assigned to participants (within person design) - presenting all three would make variation too obvious and might create artifacts. As we assume \(\mu 1\)<\(\mu 2\)<\(\mu 3\) with a small to medium effect (\(d= .3-.4\)), we computed power analyses for two t-tests with a power of \(1-\beta = .90\) (\(=.81\) for both tests). Below find power analyses with stopping rule at \(BF= 10\) or \(BF= \frac{1}{10}\) respectively and

• an effect of \(d= .4\)
• an effect of \(d= .3\)
• no effect (\(d= 0\))

Out of financial and institutional reasons we aim for a \(N_{max}= 250\). Due to expected variations in the BF with low \(n\), we begin observing the data (optional stopping) at \(n=150\).

Power Analyses

effect of \(d= .4\)

library(BFDA)

s1.4 <- BFDA.sim(expected.ES=0.4,
                 prior=list("Cauchy", list(prior.location=0, prior.scale=1)), 
                 n.min=20, stepsize=10, n.max=250, 
                 type="t.paired", 
                 design="sequential", 
                 alternative="greater", 
                 B=10000, 
                 cores=2, 
                 verbose=TRUE)

## [1] "Simulation started at 2019-06-07 12:22:38"
## [1] "Simulation finished at 2019-06-07 13:54:16"
## Duration: Time difference of 1.526999 hours

BFDA.analyze(s1.4, design="sequential", boundary=10)

##                                outcome percentage
## 1 Studies terminating at n.max (n=250)         0%
## 2    Studies terminating at a boundary       100%
## 3       --> Terminating at H1 boundary      99.1%
## 4       --> Terminating at H0 boundary       0.9%
## 
## Of 0% of studies terminating at n.max (n=250):
## 0% showed evidence for H1 (BF > 3)
## 0% were inconclusive (3 > BF > 1/3)
## 0% showed evidence for H0 (BF < 1/3)
## 
## Average sample number (ASN) at stopping point (both boundary hits and n.max): n = 59
## 
## Sample number quantiles (50/80/90/95%) at stopping point:
## 50% 80% 90% 95% 
##  50  80 110 130

plot(s1.4, n.min=150, boundary=c(1/10, 10))

                                               # As we plan to do 2 t-tests that supposed to have
SSD(s1.4, power=.90, boundary=c(1/10, 10))     # .8 power together, we aim for .90 power for each test.

## A >= 90% (actual: 91.8%) power achieved at n = 120
## This setting implies long-term rates of:
## 8.2% inconclusive results and
##    0% false-negative results.

effect of \(d= .3\)

s1.3 <- BFDA.sim(expected.ES=0.3,
                 prior=list("Cauchy", list(prior.location=0, prior.scale=1)), 
                 n.min=20, stepsize=10, n.max=250, 
                 type="t.paired", 
                 design="sequential", 
                 alternative="greater", 
                 B=10000, 
                 cores=2, 
                 verbose=TRUE)

## [1] "Simulation started at 2019-06-07 13:54:17"
## [1] "Simulation finished at 2019-06-07 15:24:48"
## Duration: Time difference of 1.508403 hours

BFDA.analyze(s1.3, design="sequential", boundary=10)

##                                outcome percentage
## 1 Studies terminating at n.max (n=250)       2.4%
## 2    Studies terminating at a boundary      97.6%
## 3       --> Terminating at H1 boundary      93.3%
## 4       --> Terminating at H0 boundary       4.3%
## 
## Of 2.4% of studies terminating at n.max (n=250):
## 1.2% showed evidence for H1 (BF > 3)
## 1.2% were inconclusive (3 > BF > 1/3)
## 0% showed evidence for H0 (BF < 1/3)
## 
## Average sample number (ASN) at stopping point (both boundary hits and n.max): n = 93
## 
## Sample number quantiles (50/80/90/95%) at stopping point:
## 50% 80% 90% 95% 
##  80 140 180 220

plot(s1.3, n.min=150, boundary=c(1/10, 10))

SSD(s1.3, power=.90, boundary=c(1/10, 10))

## A >= 90% (actual: 91.5%) power achieved at n = 220
## This setting implies long-term rates of:
## 8.5% inconclusive results and
##    0% false-negative results.

no effect (\(d= 0\))

s0 <- BFDA.sim(expected.ES=0, 
               prior=list("Cauchy", list(prior.location=0, prior.scale=1)), 
               n.min=20, stepsize=10, n.max=250, 
               type="t.paired", 
               design="sequential", 
               alternative="greater", 
               B=10000, 
               cores=2, 
               verbose=TRUE)

## [1] "Simulation started at 2019-06-07 15:24:49"
## [1] "Simulation finished at 2019-06-07 17:18:10"
## Duration: Time difference of 1.889126 hours

BFDA.analyze(s0, design="sequential", boundary=1/10)

##                                outcome percentage
## 1 Studies terminating at n.max (n=250)       7.9%
## 2    Studies terminating at a boundary      92.1%
## 3       --> Terminating at H1 boundary       1.6%
## 4       --> Terminating at H0 boundary      90.6%
## 
## Of 7.9% of studies terminating at n.max (n=250):
## 0.2% showed evidence for H1 (BF > 3)
## 3% were inconclusive (3 > BF > 1/3)
## 4.7% showed evidence for H0 (BF < 1/3)
## 
## Average sample number (ASN) at stopping point (both boundary hits and n.max): n = 79
## 
## Sample number quantiles (50/80/90/95%) at stopping point:
## 50% 80% 90% 95% 
##  50 120 220 250

plot(s0, n.min=0, boundary=c(1/10, 10))

SSD(s0, alpha=.0025, boundary=c(1/10, 10))     # We aim for a .05 alpha, that is .0025 for 2 tests.

## A <= 0.2% (actual: 0.2%) long-term rate of Type-I errors is achieved at n = 40
## This setting implies long-term rates of:
##    61.2% inconclusive results and
##    38.6% true-negative results.