Integrative Data Analysis

Integrative data analysis (IDA) is an alternative to meta-analysis that combines participant-level data from multiple studies. Two approaches, fixed effects models (FEM) and multilevel models (MLM), have been used in psychological applications of IDA, but have not been fully evaluated. Because IDA combines data from multiple studies, two different kinds of fixed effects can be studied in IDA: study-level and participant-level effects. Furthermore, between-study differences need to be modeled carefully. For IDA with cross-sectional data, we reviewed three FEMs and two MLMs. We focused on (a) whether and how they can estimate and test participant-level and study-level fixed effects; and (b) whether and how they model between-study differences in study means and participant-level effects. Because IDA is typically conducted with fewer than 30 studies, we evaluated the performance of these models and different MLM estimation methods in a simulation study under realistic IDA conditions. While two of the FEMs accurately estimate the fixed effects, they do not model between-study differences in participant-level effects, leading to incorrect inferences. Only a random-slopes MLM that accounts for differences in both study means and participant-level effects provided accurate inferences and estimates of the fixed effects and between-study differences. We found that MLMs can be feasible for IDA with as few as three to six studies using appropriate estimation methods. We illustrated the application of the five models and how they can provide different estimates and inferences in an empirical example. We conclude with recommendations to guide researchers when planning an IDA.

Cumulative data analysts typically rely on two statistical frameworks: meta-analysis (MA) and integrative data analysis (IDA; Curran et al., 2009). MA analyzes aggregate data (AD, i.e., summary statistics) whereas IDA analyzes individual participant data (IPD).

With the growing availability of open-access data from multiple psychological studies, appropriate statistical methods for synthesizing …