Randomization in clinical trials: stratification or minimization? The HERMES free simulation software

Abstract

Objectives

Operative clinical trials are often small and open-label. Randomization is therefore very important. Stratification and minimization are two randomization options in such trials. The first aim of this study was to compare stratification and minimization in terms of predictability and balance in order to help investigators choose the most appropriate allocation method. Our second aim was to evaluate the influence of various parameters on the performance of these techniques.

Materials and methods

The created software generated patients according to chosen trial parameters (e.g., number of important prognostic factors, number of operators or centers, etc.) and computed predictability and balance indicators for several stratification and minimization methods over a given number of simulations. Block size and proportion of random allocations could be chosen. A reference trial was chosen (50 patients, 1 prognostic factor, and 2 operators) and eight other trials derived from this reference trial were modeled. Predictability and balance indicators were calculated from 10,000 simulations per trial.

Results

Minimization performed better with complex trials (e.g., smaller sample size, increasing number of prognostic factors, and operators); stratification imbalance increased when the number of strata increased. An inverse correlation between imbalance and predictability was observed.

Conclusions

A compromise between predictability and imbalance still has to be found by the investigator but our software (HERMES) gives concrete reasons for choosing between stratification and minimization; it can be downloaded free of charge.

Clinical relevance

This software will help investigators choose the appropriate randomization method in future two-arm trials.

Appendices

Appendix 1

Justification for the number of simulations (10,000) carried out in this study

We aimed to estimate the expected value of a random variable (predictability or imbalance) with n independent realizations (simulations were independent) and identical distribution (patients were simulated in the same proportions). Our indicator was therefore the estimator of the random variable’s expected value. According to the central limit theorem, we know that the standard deviation of our estimator was $\sqrt{{\frac{{Var\,(X)}}{n}}}$, where n is the number of simulations (here 10,000).

Taking Trial 0 as an example, the precision of our imbalance indicator for deterministic minimization (whose estimated expected value was 0.597 %) was 1.42 %/$\sqrt{10,000 }$, which is 0.0142 %.

The 95 % confidence interval of our imbalance indicator was therefore 0.597 ± 1.96 × 0.0142, or [0.569 to 0.625]. This level of precision seemed acceptable to compare these methods, and is the reason why we presented the results with one digit after the decimal point.

Appendix 2

How the allocation method impacts the analysis of resulting data

Type of analysis

Tests of statistical inference are based on the assumption of random assignment to treatment and control groups. Only simple randomization has this property, so that distorted p values and concerns over the validity of the analysis surround not just minimization but all other allocation methods [39]. However, the disadvantage of adaptive methods like minimization is that the correct analysis is complex and not clearly worked out [39, 44]. In fact, minimization achieves balance only among the marginal distribution of the strata [25]. Where the outcome measure is a continuous variable, various authors recommend that adjustment should be made for factors in the minimization using analysis of covariance [28, 32, 45]. Several other authors recommend using permutation tests to analyze trials where minimization has been used [43, 46]; however, because these tests are not straightforward in practice and seem to make little difference to the results obtained, some authors believe permutation tests are unnecessary and that a classical analysis will usually yield satisfactory conclusions provided that minimization factors are used as covariates in the analysis [8, 9, 11, 12]. Some authors consider that the nominal significance level in that case should be adjusted [32, 35, 45], whereas Hagino et al. believe this is unnecessary [38]. For both stratified randomization and minimization, the collapsibility of the data should be checked, and if it depends on the statistic used, the tenuousness of the conclusions should be noted [47].

Inclusion of randomization covariates in the analysis

Although scientists may find the results of simple, unadjusted treatment comparisons with demonstration of good balance of important factors more convincing than the results of a covariate analysis, there is a consensus that the prognostic factors included in the randomization scheme should be taken into account in the analysis, not just for minimization [12, 32, 46, 48, 49] but also for stratification [50]. The p value for a difference between endpoint rates in treatment groups would be otherwise overestimated [18, 38, 51].

Effect on the nominal level and the power of the test

Authors in favor of stratified randomization put forward its value in reducing the risk of type I error [20] and increasing power [52], but this advantage is controversial. Some authors believe that both stratification and minimization procedures produce comparable improvements in reducing type I and II error [35]. Tu et al. found that minimization procedures were inferior to stratified allocation in reducing the two types of error, due to existing interactions between covariates [48]. However, other simulations have given opposite results [32, 35, 38].