Model Selection for Scientists: Theory, Practice, and Alternatives

Theory

• Statistical modelling: putting the “why” before the “how” is essential to finding the right tool
• The need for Model selection: balance complexity vs. goodness of fit, Occam’s razor
• Hypothesis testing: backward, forward and stepwise selection
• Mathematical basis of information criteria: Kullback-Leibler distance (Akaike Information Criterion), Bayesian statistics (Bayesian Information Criterion)
• Use of information criteria to: rank models, define “best” subset, calculate evidence ratios, calculate Akaike weights
• Multimodel inference (averaging across models): conditional averaging vs natural averaging
• Recommendations in practice to avoid common mistakes
• Alternatives beyond information theory: dominance analysis, shrinkage methods.

Practice

• Fit statistical models, each representing a plausible explanation, and use information criteria to: define “best” subset, calculate evidence ratios, calculate Akaike weights, perform multimodel inference
• Assess the impact of common methodological decisions (Which criterion should we use? How do we define the “best” subset?)
• Use simulations to uncover limitations of some methods
• Implement alternatives to information theory (dominance analysis, shrinkage)