Research Methods
Sampling Design
When surveys are carried out in Ireland and most other countries, individuals are typically sampled within households using a multi-stage design. Within this kind of sampling design, there are compelling reasons for sampling households from a set of geographical areas and for interviewing all adult members of the household. This is the design employed by the Central Statistics Office in Ireland and most other National Statistics Agencies for official surveys. The efficiency of such designs is influenced by the clustering of interviews, and depends on the degree of spatial autocorrelation observed. Nevertheless, sample stratification has the capacity to compensate – at least in part – for the negative effects of clustering on the efficiency of survey-based estimators. The best stratification variable to use when estimating the mean of a variable is one which is closely correlated with the outcome. Given the range of applications of household surveys, stratification criteria must be robust enough to yield efficiency gains across a range of outcomes. We have developed a powerful, efficient and innovative approach which relies on an aggregate measure of socio-economic composition – the HP Deprivation Index– to improve survey efficiency. This approach was developed for the CSO and has been integrated into the sampling design of official surveys in Ireland. Simulation studies and theoretical analysis have confirmed its superiority over alternative approaches to stratification and its potential to yield considerable savings. We have in-depth knowledge and expertise in the field of sampling design and can provide advice to clients in the private and public sectors on optimal designs with a view to improving accuracy and reducing the financial and organisational burden of face-to-face surveys.
Longitudinal Analysis
The representation and measurement of change is a key issue in practically all areas of empirical research. As the advantages of longitudinal research designs have met with greater appreciation, the availability of panel data has also improved. In health research, for example, the analysis of change is of fundamental importance, given the need to understand the dynamics of health, illness and well-being across the life-course. The principal advantage of longitudinal studies, when compared with cross-sectional studies, is that development and change over time can be related to a set of explanatory variables, which may also be changing over time. This poses a number of rather complex challenges and standard statistical methods typically cannot be used. Indeed, no single statistical procedure is suitable for all longitudinal studies, and each research question requires careful consideration in order to determine the most appropriate statistical methods. We have extensive experience in the application of advanced statistical methods to the analysis of longitudinal data using auto-regressive, cross-lagged, random coefficient and latent curve modelling techniques with full-likelihood estimation. These models are generally robust to missing data and irregularly spaced measurement occasions and can handle both time-invariant and time-varying covariates. We work with clients to develop research designs that can answer questions about change that are at the core of their activities, using longitudinal models to provide relevant information for strategic planning, policy-making and service delivery.
Multilevel Modelling
Contextual effects rooted in the neighbourhood, school or centre give rise to a hierarchical social structure comprising distinct levels of variation and dependence. The most commonly-used statistical techniques in the social sciences (such as the Classical Linear Regression Model) generate biased results in the presence of such structures. This bias affects the estimates produced by statistical models due to shared social environments, institutional contexts and other forms of clustering by time, space and milieu. Multilevel modelling techniques are particularly useful in this context, and have experienced rapid development and increasing popularity over recent decades. Not only do they provide more reliable estimates, they also have the potential to shed light on the ways in which the social context shapes and influences individual outcomes. Multilevel statistical techniques enable us to ascertain whether neighbourhood, school or other contextual effects strengthen or weaken the relationship between individual risk and protective factors, on the one hand, and outcomes such as uptake of new technologies, educational attainments or substance use. By treating the aggregate-level units as forming part of a distribution, multi-level models permit efficient and powerful estimates. By treating multiple measurement occasions as nested within individuals, powerful longitudinal models can be estimated using multilevel modelling techniques (also referred to as random coefficient models or hierarchical models). We have been involved in extending the research frontier by applying these techniques to social science data in Ireland, whilst developing and testing new hypotheses regarding the role of neighbourhoods, schools, organisations and time in shaping individual development, skills and well-being.
Structural Equation Modelling
Structural Equation Modelling is a flexible modelling framework which can accommodate a wide variety of models involving observed and latent variables, longitudinal and multilevel structures and arbitrarily complex networks of cause, effect and correlation. Many other models (including Regression Analysis, Exploratory Factor Analysis, Multilevel Analysis, Survival Analysis, Logistic Regression, Loglinear Analysis, Latent Curve Analysis, Path Analysis etc.) can be treated as special cases of the general Structural Equation Model. Researchers who use Structural Equation Modelling techniques often aim to evaluate theoretical hypotheses involving mechanisms and structures. Rather than treating their statistical models as ‘fictions’ or ‘instruments’, these researchers actively seek to construct models that reproduce real processes and structures, including unobserved variables, cross-level influences, reciprocal relationships, mediation, interactions, feedback loops and contextual effects. The flexibility of the statistical theory behind Structural Equation Models means that background assumptions can often be tested explicitly and rigid distributional constraints can be relaxed. Structural Equation Models are also useful tools for developing and testing composite indicators. The dimensions that characterise such indices (of well-being, deprivation, health etc.) can be conceptualised by drawing on theory and prior research, whilst selecting a restricted set of indicator variables to measure each dimension. The scores generated by a Structural Equation Model have the advantage that they measure precisely what the researcher intended, with a measurement structure that can be held constant over time in order to facilitate monitoring. The strength of the relationships depicted in the model is measured by a regression coefficient, which can be given a causal interpretation subject to specific conditions, and can be expressed using a common metric such as standard deviation units.
- TILDA Wave 1 – Direct and Indirect Influences of SEP on the Well-being of Older Adults
- GUI Wave 1 – Well-being and the Family System
- NEYAI – Determinants of Quality and Outcomes of Pre-school Services
- Healthy Ireland – A Conceptual Approach towards a Data and Research Plan
- The 2011 Pobal HP Deprivation Index – Conceptual Underpinning
- PEACE II – Analysis of Community Uptake
Confirmatory Programme Evaluation
Confirmatory Programme Evaluation is a method for conducting theory-driven evaluations. It entails an approach to impact assessment that emphasises the development and testing of a priori programme theories within a broad assessment of programme effectiveness.
In a theory-driven impact evaluation, the explicit theory of the programme is highlighted to establish an a priori model of how the programme is expected to exert its influence. Causal uncertainty is reduced through an examination of the empirical pattern of findings against the expectations inherent in the programme.
Confirmatory Programme Evaluation addresses three key questions: Is programme participation independently and consistently associated with key outcomes? Do the estimated effects vary by background characteristics, such as family attributes, or by programme components? What are the processes or pathways through which participation leads to effectiveness in the short term and over time?
The principal steps involved in Confirmatory Programme Evaluation are as follows (Reynolds, 2005):
- Specify programme theory and processes that are expected to affect outcomes.
- Identify and measure outcomes and their timing over the short, intermediate, and long term.
- Collect or obtain data on hypothesised mediators of the programme theory and key background variables necessary to identify programme impact.
- Estimate main effects of programme participation for the total group and relevant subgroups, quantifying the temporality, size, gradient, specificity, consistency, and coherence of programme effects.
- Where main effects are detected, test hypothesised causal mechanisms of the programme theory to account for estimated effects.
- Interpret the pattern of findings to facilitate generalisation and knowledge transfer.
- Identify formative uses of findings for programme improvement, including modifications to programme theory, programme implementation, and analysis of programme effects.
