This version: June 10 2025 09:47

The investigation of causal relations between economic phenomena presents many problems of peculiar difficulty and offers many opportunities for fallacious conclusions. Since the statistician can seldom or never make experiments for himself, he has to accept the data of daily experience, and discuss as best he can the relations of a whole group of changes; he cannot, like the physicist, narrow down the issue to the effect of one variation at a time. The problems of statistics are in this sense far more complex than the problems of physics.
— Udny Yule

Breaking News

  • The final exam will take place on Thursday, June 19 from 10 am to noon. Room: CB 122

Rotating this figure around a vertical axis (click on it), what plot do you see twice in each full rotation?
Alt text

Calendar (tentative)

  • Classes Tuesdays and Thursdays, May 6 to June 12, 2025 10 am to 1 pm in PSE (Petrie) 321
    • except Thursdays, May 8, May 22, June 12: 11:30 am to 2:30 pm
      • on these days the lecture will start in PSE 321 from 11:30 am to 1 pm and continue in Vari Hall 1158 from 1 pm to 2:30 pm.
  • Office hours and tutorial meetings: Tutorial Friday 4 pm at https://yorku.zoom.us/my/georgesmonette
  • Communications: Messages and questions about the course should be posted publicly on Piazza. You may use private messages on Piazza for confidential messages in which the response is of no benefit to the rest of the class.

Day 1: May 6

Assignment 1: (individual) due Friday, May 9

  • Summary:
    1. Install R and RStudio
    2. Get a free Github account
    3. Install git on your computer
    4. Connect with Piazza, create a LOG post and introduce yourself
  • 1. Install R and RStudio following these instructions. If you already have R and RStudio, update them to the latest versions.
  • 2. Get a free Github account: If you don’t have one, follow this excellent source of advice from Jenny Bryan on using git and Github. with R and RStudio.
  • 3. Install git on your computer using these instructions.
    • If you are curious about how git is used have a look at this tutorial!
    • As a final step: In the RStudio menu click on Tools | Global Options ... | Terminal. Then click on the box in Shell paragraph to the right of New terminals open with:
      • On a PC, select Git Bash
      • On a Mac, select Bash
    • You don’t need to do anything else at this time. We will see how to set up SSH keys to connect to Github and to blackwell through the RStudio terminal in a few lectures.
  • 4. Connect with Piazza and post about your experiences installing the above:
    • Join the MATH 6642 Piazza site by going to this URL: piazza.com/yorku.ca/spring2025/math6642
      • Use the access code ‘blackwell’ when prompted.
    • Create a post on Piazza to introduce yourself to your colleagues: - The title should read: Introduction: followed by the name you use socially, e.g. Jon Smith - The first line should show the name of your Github account, e.g. Github: jsmith - Follow this with a discussion of your goals in taking this course? - Then information on what computing languages you are familiar with? Which ones are you proficient with?
      • Specifically with respect to R and RStudio:
        • Where and how did you learn R?
        • Do you use Rmarkdown?
        • Do you use the ‘hadleyverse’? e.g. tidyr, ggplot2
        • Do you often write functions to solve data analysis problems?
        • Have you written any packages, private or public?
        • Share some interests if you wish: hobbies, favourite musicians, movies, restaurants, etc.
        • Click on the assn1 folder when you submit your post.
      • Create yet another post entitled ‘Getting started’.
        • Describe any problems you encountered installing R, RStudio, and git
        • Describe any problems registering with Github
        • How could the instructions be improved to make the process easier?
        • If you couldn’t complete the installation, describe the problem or error message(s) you encountered.
        • Click on the assn1 folder before submitting your post.

Assignment 2: (individual) due Sunday, May 11

The goals of this assignment are

  • to explore the implications of Lord’s Paradox for causal inference with the simplest of longitudinal data, that is, a pre-test/post-test design with various forms of treatment assignment or selection
  • to develop and encourage the practice of simulation to explore the properties of statistical methods
  • to practice your programming skills

You may use AI to help with this assignment provided you describe the prompts you used and credit AI for the portions of the assignment it helped to create. We all need to learn how to use AI effectively, to leverage its strengths and recognize its weaknesses.

Your work should be submitted privately to the instuctor on Piazza using the folder assn2. I might make some of your work public after the due date. When I do so, Piazza protects your identity by replacing your name with a pseudonym. If you don’t wish your identity to be revealed, avoid including any identifying information in your submission. I can tell who submitted the work from your identity in Piazza.

In this assignment, you will simulate data similar to that in Lord’s Paradox using four specific scenarios.

Suppose we want to study the effect of a special calculus tutorial for incoming students in fall term of 2025. Suppose that all incoming students must take a calculus pre-test on August 1. Call the score on this test Y_1. They have the option of taking a one-week calculus refresher course in mid-August. The variable Course is equal to 1 if they took the course and 0 if they did not. All students take the same test again on September 1. Call this result Y_2.

We assume that each student has a latent ability level, L, that has a normal distribution with a variance of \(\tau^2\) (it’s up to you to choose a suitable value). L cannot be measured directly but each student has some awareness of their own value of L.

We further suppose that, for each student, Y_1 = L + eps_1 and Y_2 = L + eps_2, with eps_1 and eps_2 normally distributed and independent, each with some variance \(\sigma^2\).

This is not required for the assignment but you might be interested in considering the correlation between Y_1 and Y_2 as a function of \(\tau^2\) and \(\sigma^2\).

You simulate data in which you suppose that the course is of no benefit to see whether your analyses provide a ‘correct’ estimate of that fact, or whether they estimate a spurious positive or negative effect of the course.

Choose any three of the following scenarios for ‘treatment’ assignment or selection – assuming unrealistically that all students comply with the result of their assignment:

  1. Half the students are assigned randomly to take the course independently of Y_1.
  2. Students are assigned to take the course if their grade on Y_1 is below a pre-determined cutoff such that approximately half the students would be expected to take the course.
  3. Students choose themselves whether to take the course based on their ‘internal’ value of L. Suppose they choose to take the course if L is below a cutoff. Again, suppose that this results in roughly half the students taking the course.
  4. Students are influenced by both L and Y_1 so that they choose to take the course if the value of L + Y_1 is below a cutoff.

Consider the following methods for the analysis of the data to determine whether the course is beneficial or detrimental – we know it has no effect because we simulated it that way:

  1. Simple comparison of outcomes: Y_2 ~ Course.
  2. Analysis of change scores. Let D = Y_2 - Y_1 and test whether there is an improvement using the linear model: D ~ Course
  3. Using the post-test Y_2 as an outcome and the pre-test Y_1 as a covariate with the model: Y_2 ~ Course + Y_1.
  4. Using the change score as an outcome and the pre-test as a covariate: D ~ Course + Y_1

For each ‘assignment/selection’ create a plot showing your data (with Y_2 on the vertical axis and Y_1 on the horizontal axis) and lines showing the predicted outcome from each model.

Write a brief summary of the implications of your results.

Note that your choices for the relative variances of L and eps_i will have a considerable effect on your results. If you structure your data simulation through a function in R that uses these two parameters as input, you can easily explore their impact. However, you shouldn’t worry about doing this in a ‘sophisticated’ manner. Just give it an honest try. Don’t forget to credit AI, or other sources, where credit is due. You may help each other over Piazza or otherwise but your submission should be your own. Both asking for and providing help count as course contributions.

Day 2: May 8

  • Continue Day 1
  • New:
  • Something that came up in the last class
    • Here are references to three articles by John Ionannidis on what has come to be known as the crisis of reproducibility:
      • “Why Most Published Research Findings Are False.” John P. A. Ioannidis (2005). This is the paper that brought the issue to wide awareness.
      • “What Have We (Not) Learnt from Millions of Scientific Papers with P Values?” John P. A. Ioannidis (2019): a followup, and
      • “When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment,” Szucs and Ioannidis (2017).
  • Things before the next class:
  • Continue Day 1
  • Exercises:
    1. Let \(Y\) and \(X\) be numerical variables and let \(G\) be a factor. Consider the following models. All but one of these models will produce the same regression coefficient for X or Xr but they will produce different standard errors. Identify the model that produces a different coefficient. Rank the others where you can according to the standard error of the estimated coefficient for X (or Xr) stating which would be equal if any (assume a very large \(n\) and ignore the effect of slight differences in degrees of freedom for the error term). Explain your reasoning briefly.
      1. Y ~ X + G
      2. Y ~ X
      3. Yr ~ Xr where Yr is the residual of Y regressed on G and similarly for Xr
      4. Y ~ Xr
      5. Y ~ X + Xh where Xh is the least-squares predictor of X based on G
      6. Y ~ X + Xh + Zg where Zg is a G-level numerical variable, i.e. it has the same value for all observations with a common value of G.
    2. Let \(Y\) and \(X\) be numerical variables and let \(Z_1, Z_2, ..., Z_k\) be numerical or factor variables. Consider the following models. All but one of these models will produce the same regression coefficient for X or Xr but they will produce different standard errors. Identify the model that produces a different coefficient. Rank the others where you can according to the standard error of the estimated coefficient for X (or Xr) stating which would be equal if any (assume a very large \(n\) and ignore the effect of slight differences in degrees of freedom for the error term). Explain your reasoning briefly.
      1. Y ~ X + Z1 + Z2 + ... +Zk
      2. Y ~ X
      3. Yr ~ Xr where Yr is the residual of Y regressed on Z1, Z2, ..., Zk and similarly for Xr
      4. Y ~ Xr
      5. Y ~ X + Xh where Xh is the least-squares predictor of X based on Z1, Z2, ..., Zk
      6. Y ~ X + Xh + Zg where Zg is a linear combination of Z1, Z2, ..., Zk.

Day 3: May 13

Assignment 3: (teams) (see due dates below)

  • Do the exercises below following these directions:
    • Each member of the team does every fourth (note that the largest team now has 4 members) question by the first deadline. Post each solution as a separate post on Piazza and make it private to your team so members of other teams can work independently. Usee the folder ‘assn1’ when posting.
    • Don’t forget to repeat the question so your post can be read and understood without have to use an external reference.
    • Between the first and second deadline, all members of the team help each other out to correct or improve their solutions.
    • Between the second and third deadline, the original author combines contributions by other members to create a polished answer.
    • Only after the third deadline, edit each solution to make it public to the whole class.
    • Have a look at the course description for more explanation.
    • Important note: It’s excellent to use online materials provided you provide links to them and that you assess the quality of the information. Searching online is a very important part of research. But a lot of material is posted online by people who have a limited or very superficial understanding of the topic they post about. It’s important for you to develop the critical ability to distinguish well-informed material from superficial or erroneous material. It’s also important to assess the context in which material is posted.
  • The two exercises posted on the last day. These make use of the ‘three basic theorems of regression’. They show how many different models can yield the same estimate of the ‘effect of X’. However, perhaps surprisingly, one model does not. Use the title ‘AVP 1’ and ‘AVP 2’ for these exercises.
  • Exercises corresponding to numbers circled in green in Outline of Linear Algebra for Regression. Use the titles ‘LA X’ where X is the number of the exercise.
  • Deadlines: See the course description for the meaning of these deadlines.
    1. Friday, May 16 at noon
    2. Sunday, May 18 at noon
    3. Monday, May 19 at noon
  • Random sequence: 1 4 2 3

Some Explanations: The exercises posted on the last day consist of 12 questions: 6 in the first part and 6 in the second part; and there are 12 numbered exercises in the ‘Outline of Linear Algebra for Regression’ – consider 6a and 6b as one exercise and skip the unnumbered one … unless you can’t resist. So there are 24 exercises altogether.

The random sequence (I apologize for not having explained it in class) is intended to counter the effect of the typical difficulty gradient in the usual sequence of exercises. If the team member whose has number 1 on the team always did question whose order is 1 mod 4, and the fourth member always did questions whose order is 4 mod 4, then person number 1 would be cheated from the experience of wrangling the more difficult questions. The random numbers are intended to vary the order for each assignment.

The way it works is that person number 1 takes the first random number, call it \(n_1\) and questions whose order is \(n_1\) mod 4, person number 2 on the team takes the second random number, \(n_2\) and does questions whose order is \(n_2\) mod 4, etc.

Use mod 4 even if your team has fewer than 4 members. Otherwise smaller teams would have more work. The random numbers will be different for each team exercise.

The exercises assigned on the last day involve comparisons so you can’t really do each question on its own. What you should do therefore is, if you have model number \(n\), pick any other model other than \(n\) and compare your model with that one, explaining whether \(\hat{\beta}_X\) would be the same and, if so, what can you say about \(SE(\hat{\beta}_X)\).

Re the Spectral Decomposition Theorem (Eigenvalue-Eigenvector decomposition of a symmetric matrix): You don’t have to prove this theorem, you can take it as known.

Day 4: May 15

Day 5: May 20

Assignment 4: (teams)

  • There are numbered ‘QUESTIONS’ in Lab_1.R and in Lab_2.R, as well as some numbered EXERCISES in Lab_2.R.
    • Team member #1 should do QUESTIONS and EXERCISES that numbered 2 mod 4,
    • team member #2, 4 mod 4
    • team member #3, 1 mod 4
    • team member #4, 3 mod 4
  • Deadlines: See the course description for the meaning of these deadlines.
    1. Saturday, May 24 at noon
    2. Sunday, May 25 at noon
    3. Monday, May 26 at noon

Day 7: May 27

Day 10: June 5

  • continuation

Day 11: June 10

Overview of Bayesian methods using Stan:

  • MCMC and HMC
  • Some examples in Stan / Rscript
    • Discounting outliers with thick-tailed error distributions
    • Flexible multi-level modelling
  • Stan for shortitudinal data
    • The estimate of the compositional effect with data with few observations per cluster is biased due to the relatively large error variance in the measure of the contextual variable as a measure of the ‘true’ cluster mean. Modelling with Stan makes it easy to separately specify a structural model: the model based on the ‘true’ but unknown variables and the measurement model: the model that specifies how observed variables depend on the variables in the structural model. As a result we obtain a consistent estimate of the compositional effect.

Day 12: June 12

  • Project presentations starting at 11:30 am.

Figure: Divergent chains in Stan

1. Ideas in Regression: Why Models Matter

Predictive versus Causal Inference

Regression: Correlation, Data and Beta Ellipses

2. Linear Models for Nested Data with Normal Response

Hierarchical and Mixed Models for Clustered Data

  • Slides: Hierarchical Models and Mixed Models / annotated

    • equivalent models with complementary insights

  • R script: Lab 1 - Mixed Models / html / pdf

    • Exploring data
    • Data: Levels and structure
    • Selecting a random subset of clusters
    • First look at variables
    • Looking at Level 2 variables (invariant within schools)
    • Creating additional Level 2 ( and Level 1 ) variables with ‘capply’
    • Tranformations of Level 1 variables within groups
    • Looking at data in 3D
    • Looking at Level 1 and Level 2 data using Lattice graphics
    • Visualizing fitted lines in beta space
    • Looking at between group effect
    • Fitting a mixed model
    • Convergence problems
    • Handling NAs
    • Hausman Test: Is the between effect different from the within effect?
    • Fitting a model with a contextual mean
    • Role of contextual variable for ses
    • Interpreting the model with contextual effect
    • Estimating the compositional effect (between effect)
    • Visualizing the fitted model
    • Plotting error bars
    • Using CWG instead of raw SES
    • CWG vs raw in RE model
    • Notes on testing: ML vs REML
    • Diagnostics
      • Diagnostics with Level 1 residuals
      • Scale - location plot:
      • Diagnostics with Level 2 residuals
      • Influence diagnostics – drop one row or one cluster at a time
    • Looking at the model
    • Predicted response
    • Plotting effects with confidence bounds
    • Building and testing the RE model
    • Using simulation to calibrate p-values
    • Simplifying the FE model
    • Some Dos, don’ts and whys
    • Wald or Likelihood Ratio Test
    • Some Level 2 diagnostics
    • Visualizing the model
    • More effect plots
    • Refining the model
    • Multilevel R squared
    • Visualizing the model and asking sharper questions —-> <!—-

  • Short note: Contextual, compositional and within effects (cf line 1200 of Lab 1.R)

  • John’s files on the effects package:

Hierarchical and Mixed Models for Longitudinal Data

  • Slides: Longitudinal Models
  • R script: Lab 2 - Longitudinal Models / html / pdf
    • Detailed example showing how different longitudinal assumptions affect major estimates.
    • LME model
    • Hausman test:
    • Adjusting for time
    • Diagnostics: Level 1
      • Diagnostics for heteroskedasticity
      • Diagnostics for autocorrelation
    • Diagnostics: Level 2
    • Dropping observations
    • Modeling autocorrelation
    • Modeling heteroskedasticity
    • Interpreting different kinds of residual plots
    • Visualizing the impact of model selection
    • Displaying data and fitted values together
  • References:
    • Singer and Willett (2003)
    • Snijders, A. B., and Bosker (2012)
    • Fox (2015, pp 699–741)
    • Mixed Effects in R, an online appendix to Fox and Weisberg (2019a) provides a very clear presentation and reference for much of the material presented here.

3. Introduction to Bayesian Ideas and Modern Bayesian Methods

4. Introduction to Stan

Diverging chains

5. Non-Linear Models for Normal Responses: Asymptotic Functions of Time

  • Slides: Non-Linear Mixed Models
  • R script: Recovery after TBI
    • Asymptotic recovery curves With Stan
      • there is no intrinsic difference between linear and non-linear models
    • Problems with convergence – possible remedies: reparametrization, more informative prior, sensitivity analysis
    • Multivariate response model

6. Models for Non-Normal Responses: GLMMs

7. Missing Data with Bayesian Methods

  • Synopsis:
    • If the model missingness is MAR and ignorable in the longitudinal ‘Model of Analysis’ (MA) then you can analyse the observations you have.
    • Sometimes missingness is not MAR in the MA but would be MAR if you could include other ‘auxiliary variables’, e.g. mediators, that you need to exclude from the MA.
    • Multiple Imputation provides a solution and Bayesian Modeling provides another.
  • Slides: Missing Data
  • R script: Lab 4 a: Missing Data with Multiple Imputation
  • Bayesian approach: Multiple imputation involves performing a small number of analyses, each with a different set of sampled values for the missing data, and then combining these analyses, averaging them and taking into account the variability between them.
  • With a Bayesian analysis we can do the equivalent of and infinite number of analyses each with a different set of sampled values for the missing data. That is, we use the whole distribution of possible value for the missing data. And then we marginalize over the missing data. Which happens to be easy with MCMC because we marginalize by ignoring the random values generated by MCMC.
  • With a distribution represented as a density, marginalizing is extremely difficult and conditioning is relatively easier. With a distribution reprented by a sample, it’s the opposite. Marginalizing is trivial, conditioning is challenging.

The following lab uses Bayesian imputation for the data with missingness determined by the mediator Weness:

8. Shorter topics

8a Functions of time: Splines

8b Shortitudinal Data

Adjusting for measurement error in computed contextual variables.

8c Things That Can Go Wrong with Bayes

Other references

  • Andrew Gelman’s blog
  • R Bloggers
  • Andrew Gelman and Jennifer Hill are working on a new edition of Gelman (2007), which was published before Stan flourished. Gelman says that the new edition will make extensive use of Stan using the rstanarm package.
  • Mixtures in Stan
  • Gelman, Hill, and Yajima (2012) Multiple comparisons in Bayesian approaches to inference. #’

Postscript

John Fox and Tanya Murphy sent some very interesting references:

Issues and Errata

Acknowledgments

A great many collaborators, students and friends have contributed to many of the ideas in this course. I’d like to acknowledge a few with deep apologies to the many I’m missing.

  • Heather Krause
  • John Fox, Michael Friendly, Hugh McCague, Mirka Ondrack, Bryn Greer-Wootten, Jolynn Peck, Robert Cribbie, David Flora, Alyssa Counsell, Jessica Flake
  • Phil Chalmers, Carrie Smith, Ernest Kwan, Andrew Hunter
  • Jane Heffernan, Angie Raad
  • Yifaht Korman, Tammy Kostecki-Dillon, Pauline Wong, E. Manolo Romero Escobar
  • Andy Koh, Jordan Collins

References

Arnqvist, Göran. 2020a. “Mixed Models Offer No Freedom from Degrees of Freedom.” Trends in Ecology & Evolution 35 (4): 329–35. https://doi.org/10.1016/j.tree.2019.12.004.
———. 2020b. “Mixed Models Offer No Freedom from Degrees of Freedom.” Trends in Ecology & Evolution 35 (4): 329–35. https://doi.org/10.1016/j.tree.2019.12.004.
Bates, Douglas, Martin Mächler, Ben Bolker, and Steve Walker. 2015. “Fitting Linear Mixed-Effects Models Using Lme4.” Journal of Statistical Software 67 (1). https://doi.org/10.18637/jss.v067.i01.
Bergland, Christopher. 2019. “Rethinking P-Values: Is "Statistical Significance" Useless?” Psychology Today. https://www.psychologytoday.com/blog/the-athletes-way/201903/rethinking-p-values-is-statistical-significance-useless.
Best, Nicky, and Alexina Mason. 2012. “Bayesian Approaches to Handling Missing Data.”
Blackwell, Matthew. 2013. “Observational Studies and Confounding,” 6.
Buuren, Stef van, and Karin Groothuis-Oudshoorn. 2011. “Mice : Multivariate Imputation by Chained Equations in R.” Journal of Statistical Software 45 (3). https://doi.org/10.18637/jss.v045.i03.
Buuren, Stef Van, and Karin Groothuis-Oudshoorn. 2011. “Mice : Multivariate Imputation by Chained Equations in R.” Journal of Statistical Software 45 (3). https://doi.org/10.18637/jss.v045.i03.
Byeon, Sangmin, and Woojoo Lee. 2023. “Directed Acyclic Graphs for Clinical Research: A Tutorial.” Journal of Minimally Invasive Surgery 26 (3): 97–107. https://doi.org/10.7602/jmis.2023.26.3.97.
Chambers, J., T. Hastie, and D. Pregibon. 1990. “Statistical Models in S.” In Compstat, edited by Konstantin Momirović and Vesna Mildner, 317–21. Heidelberg: Physica-Verlag HD. https://doi.org/10.1007/978-3-642-50096-1_48.
Christensen, Rune Haubo B. n.d. “A Tutorial on Fitting Cumulative Link Mixed Models with Clmm2 from the Ordinal Package,” 10.
Cox, D. R., and N. Wermuth. 1999. “Derived Variables for Longitudinal Studies.” Proceedings of the National Academy of Sciences 96 (22): 12273.
Curran, Patrick J., and Daniel J. Bauer. 2011. “The Disaggregation of Within-Person and Between-Person Effects in Longitudinal Models of Change.” Annual Review of Psychology 62 (1): 583–619. https://doi.org/10.1146/annurev.psych.093008.100356.
Curran, Patrick J., Khawla Obeidat, and Diane Losardo. 2010. “Twelve Frequently Asked Questions About Growth Curve Modeling.” Journal of Cognition and Development 11 (2): 121–36. https://doi.org/10.1080/15248371003699969.
Damelang, Andreas, and Sabine Ebensperger. 2020. “Gender Composition of Occupations and Occupational Characteristics: Explaining Their True Relationship by Using Longitudinal Data.” Social Science Research 86 (February): 102394. https://doi.org/10.1016/j.ssresearch.2019.102394.
Dang, Kevin, Megan A. Kirk, Georges Monette, Joel Katz, and Paul Ritvo. 2021. “Meaning in Life and Vagally-Mediated Heart Rate Variability: Evidence of a Quadratic Relationship at Baseline and Vagal Reactivity Differences.” International Journal of Psychophysiology, March. https://doi.org/10.1016/j.ijpsycho.2021.03.001.
Daniels, Michael J., and Joseph W. Hogan. 2008. Missing Data in Longitudinal Studies: Strategies for Bayesian Modeling and Sensitivity Analysis. Chapman and Hall/CRC.
Daniels, Michael J., and Yan D. Zhao. 2003. “Modelling the Random Effects Covariance Matrix in Longitudinal Data.” Statistics in Medicine 22 (10): 1631–47. https://doi.org/10.1002/sim.1470.
Dean, C. B., and Jason D. Nielsen. 2007a. “Generalized Linear Mixed Models: A Review and Some Extensions.” Lifetime Data Analysis 13 (4): 497–512. https://doi.org/10.1007/s10985-007-9065-x.
———. 2007b. “Generalized Linear Mixed Models: A Review and Some Extensions.” Lifetime Data Analysis 13 (4): 497–512. https://doi.org/10.1007/s10985-007-9065-x.
Diggle, Peter J., and David Taylor-Robinson. 2019. “Longitudinal Data Analysis.” In Handbook of Epidemiology, edited by Wolfgang Ahrens and Iris Pigeot, 1–34. New York, NY: Springer. https://doi.org/10.1007/978-1-4614-6625-3_75-1.
Dong, Yiran, and Chao-Ying Joanne Peng. 2013. “Principled Missing Data Methods for Researchers.” SpringerPlus 2 (1): 222. https://doi.org/10.1186/2193-1801-2-222.
Fitzmaurice, Garrett, and Geert Molenberghs. 2008. “Advances in Longitudinal Data Analysis: An Historical Perspective.” In Longitudinal Data Analysis, edited by Geert Verbeke, Marie Davidian, Garrett Fitzmaurice, and Geert Molenberghs, 20085746:3–27. Chapman and Hall/CRC. https://doi.org/10.1201/9781420011579.pt1.
Fox, John. 2015. Applied Regression Analysis and Generalized Linear Models. Sage Publications.
———. 2019a. “Bayesian Estimation of Regression Models.” In, 31.
———. 2019b. “Bayesian Estimation of Regression Models.” In, 31.
Fox, John, and Sanford Weisberg. 2019a. An R Companion to Applied Regression. 3rd ed. Thousand Oaks CA: Sage. https://socialsciences.mcmaster.ca/jfox/Books/Companion/.
———. 2019b. “Nonlinear Regression, Nonlinear Least Squares, and Nonlinear Mixed Models in R.” Population 150: 200.
Fox, John, Sanford Weisberg, Brad Price, Michael Friendly, Jangman Hong, Robert Andersen, David Firth, Steve Taylor, and R Core Team. 2022. “Effects: Effect Displays for Linear, Generalized Linear, and Other Models.” https://cran.r-project.org/web/packages/effects/index.html.
Friendly, Michael, Georges Monette, and John Fox. 2013. “Elliptical Insights: Understanding Statistical Methods Through Elliptical Geometry.” Statistical Science 28 (1): 1–39. https://doi.org/10.1214/12-STS402.
Fu, Liya, You-Gan Wang, and Jinran Wu. 2024a. “Chapter 6 - Recent Advances in Longitudinal Data Analysis.” In Handbook of Statistics, edited by Donald E. K. Martin, Arni S. R. Srinivasa Rao, and C. R. Rao, 50:173–221. Modeling and Analysis of Longitudinal Data. Elsevier. https://doi.org/10.1016/bs.host.2023.10.007.
———. 2024b. Recent Advances in Longitudinal Data Analysis. Vol. 50. Elsevier. https://doi.org/10.1016/bs.host.2023.10.007.
Gelman, Andrew. 2004. “Treatment Effects in Before-After Data.” In Wiley Series in Probability and Statistics, edited by Andrew Gelman and Xiao-Li Meng, 1st ed., 195–202. Wiley. https://doi.org/10.1002/0470090456.ch18.
———. 2007. “Rejoinder: Struggles with Survey Weighting and Regression Modeling.” Statistical Science 22 (2): 184–88. http://www.jstor.org/stable/27645819.
———. 2024a. “Hopes and Limitations of Reproducible Statistics and Machine Learning.” Harvard Data Science Review 6 (1). https://doi.org/10.1162/99608f92.040d5523.
———. 2024b. “A Chain as Strong as Its Strongest Link? Understanding the Causes and Consequences of Biases Arising from Selective Analysis and Reporting of Research Results.” Journal of Research on Educational Effectiveness 17 (3): 459–61. https://doi.org/10.1080/19345747.2023.2219673.
Gelman, Andrew, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2013. Bayesian Data Analysis. Chapman and Hall/CRC.
Gelman, Andrew, John B. Clarlin., Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2025. “Bayesian Data Analysis, 3rd Ed. Home Page.” Bayesian Data Analysis, 3rd Ed. Home Page. https://sites.stat.columbia.edu/gelman/book/.
Gelman, Andrew, Jennifer Hill, and Masanao Yajima. 2012. “Why We (Usually) Don’t Have to Worry About Multiple Comparisons.” Journal of Research on Educational Effectiveness 5 (2): 189–211. https://doi.org/10.1080/19345747.2011.618213.
Gelman, Andrew, and Iain Pardoe. 2006. “Bayesian Measures of Explained Variance and Pooling in Multilevel (Hierarchical) Models.” Technometrics 48 (2): 241–51. https://doi.org/10.1198/004017005000000517.
Gigerenzer, Gerd, Wolfgang Gaissmaier, Elke Kurz-Milcke, Lisa M. Schwartz, and Steven Woloshin. 2007. “Helping Doctors and Patients Make Sense of Health Statistics.” Psychological Science in the Public Interest 8 (2): 53–96. https://doi.org/10.1111/j.1539-6053.2008.00033.x.
Gillespie, Colin, and Robin Lovelace. n.d.a. Efficient R Programming. Accessed April 22, 2025. https://csgillespie.github.io/efficientR/.
———. n.d.b. Efficient R Programming. Accessed December 26, 2019. https://csgillespie.github.io/efficientR/.
Hartig, Florian. 2022. DHARMa: Residual Diagnostics for Hierarchical (Multi-Level / Mixed) Regression Models. Manual. https://CRAN.R-project.org/package=DHARMa.
Hernán, Miguel A, and James M Robins. 2020. Causal Inference: What If. Boca Raton: Chapman & Hall/CRC: Chapman & Hall/CRC. https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/.
Imbens, Guido W. n.d. “Causal Inference in the Social Sciences.”
Ioannidis, John P A. 2005. “Why Most Published Research Findings Are False.” PLoS Medicine 2 (8): 6. https://journals.plos.org/plosmedicine/article/file?id=10.1371%2Fjournal.pmed.0020124&type=printable.
Ioannidis, John P. A. 2019. “What Have We (Not) Learnt from Millions of Scientific Papers with P Values?” The American Statistician 73 (sup1): 20–25. https://doi.org/10.1080/00031305.2018.1447512.
Jahangiri, Mina, Anoshirvan Kazemnejad, Keith S. Goldfeld, Maryam S. Daneshpour, Shayan Mostafaei, Davood Khalili, Mohammad Reza Moghadas, and Mahdi Akbarzadeh. 2023. “A Wide Range of Missing Imputation Approaches in Longitudinal Data: A Simulation Study and Real Data Analysis.” BMC Medical Research Methodology 23 (1): 161. https://doi.org/10.1186/s12874-023-01968-8.
Johnson, Paul, and John Gruber. n.d. “R Markdown Basics,” 19.
Kass, Robert E., Brian S. Caffo, Marie Davidian, Xiao-Li Meng, Bin Yu, and Nancy Reid. 2016a. “Ten Simple Rules for Effective Statistical Practice.” PLOS Computational Biology 12 (6): e1004961. https://doi.org/10.1371/journal.pcbi.1004961.
———. 2016b. “Ten Simple Rules for Effective Statistical Practice.” Edited by Fran Lewitter. PLOS Computational Biology 12 (6): e1004961. https://doi.org/10.1371/journal.pcbi.1004961.
Kim, Yongnam. 2019. “The Causal Structure of Suppressor Variables.” Journal of Educational and Behavioral Statistics 44 (4): 367–89. https://doi.org/10.3102/1076998619825679.
Kuh, Swen, Lauren Kennedy, Qixuan Chen, and Andrew Gelman. 2022. “Using Leave-One-Out Cross-Validation (LOO) in a Multilevel Regression and Poststratification (MRP) Workflow: A Cautionary Tale.” arXiv. https://doi.org/10.48550/arXiv.2209.01773.
M. Gad, Ahmed, and Rasha B. El Kholy. 2012. “Generalized Linear Mixed Models for Longitudinal Data.” International Journal of Probability and Statistics 1 (3): 41–47. https://doi.org/10.5923/j.ijps.20120103.03.
Matuschek, Hannes, Reinhold Kliegl, Shravan Vasishth, Harald Baayen, and Douglas Bates. 2017. “Balancing Type I Error and Power in Linear Mixed Models.” Journal of Memory and Language 94 (June): 305–15. https://doi.org/10.1016/j.jml.2017.01.001.
McElreath, Richard. 2018. Statistical Rethinking: A Bayesian Course with Examples in R and Stan. 1st ed. Chapman and Hall/CRC. https://doi.org/10.1201/9781315372495.
McGillycuddy, Maeve, Gordana Popovic, Benjamin M. Bolker, and David I. Warton. 2025. “Parsimoniously Fitting Large Multivariate Random Effects in glmmTMB.” Journal of Statistical Software 112 (April): 1–19. https://doi.org/10.18637/jss.v112.i01.
Monette, Georges. 2025. “MATH 6642: Applied Longitudinal Data Analysis.” MATH 6642: Applied Longitudinal Data Analysis. http://blackwell.math.yorku.ca/MATH6642/.
Morgan, Stephen L., ed. 2013. Handbook of Causal Analysis for Social Research. Handbooks of Sociology and Social Research. Dordrecht: Springer Netherlands. https://doi.org/10.1007/978-94-007-6094-3.
Oliver, John. 2016. “Last Week Tonight with John Oliver: Scientific Studies.” HBO. https://www.youtube.com/watch?v=0Rnq1NpHdmw.
Pearl, Judea, and Dana Mackenzie. 2018. The Book of Why: The New Science of Cause and Effect. Basic Books.
Peugh, James L., Sarah J. Beal, Meghan E. McGrady, Michael D. Toland, and Constance Mara. 2021. “Analyzing Discontinuities in Longitudinal Count Data: A Multilevel Generalized Linear Mixed Model.” Psychological Methods 26 (4): 375–97. https://doi.org/10.1037/met0000347.
Pinheiro, José C., and Douglas M. Bates. 2000. Mixed-Effects Models in S and S-PLUS. Statistics and Computing. New York: Springer.
“R Interface to CmdStan.” n.d. Accessed March 25, 2025. https://mc-stan.org/cmdstanr/.
Raudenbush, Stephen W., and Daniel Schwartz. 2020. “Randomized Experiments in Education, with Implications for Multilevel Causal Inference.” Annual Review of Statistics and Its Application 7 (1): 177–208. https://doi.org/10.1146/annurev-statistics-031219-041205.
Rietbergen, Charlotte, Thomas P. A. Debray, Irene Klugkist, Kristel J. M. Janssen, and Karel G. M. Moons. 2017. “Reporting of Bayesian Analysis in Epidemiologic Research Should Become More Transparent.” Journal of Clinical Epidemiology 86 (June): 51–58.e2. https://doi.org/10.1016/j.jclinepi.2017.04.008.
Rosa, G. J. M., D. Gianola, and C. R. Padovani. 2004. “Bayesian Longitudinal Data Analysis with Mixed Models and Thick-tailed Distributions Using MCMC.” Journal of Applied Statistics 31 (7): 855–73. https://doi.org/10.1080/0266476042000214538.
Rosset, Saharon, and Ryan J. Tibshirani. 2017. “From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation.” https://doi.org/10.48550/arxiv.1704.08160.
———. 2020a. “From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation: Rejoinder.” Journal of the American Statistical Association 115 (529): 161–62. https://doi.org/10.1080/01621459.2020.1727236.
———. 2020b. “From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation: Rejoinder.” Journal of the American Statistical Association 115 (529): 161–62. https://doi.org/10.1080/01621459.2020.1727236.
———. 2022. “From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation: Correction.” Journal of the American Statistical Association 117 (537): 529–29. https://doi.org/10.1080/01621459.2021.2016420.
Schervish, Mark J. 1996. “P Values: What They Are and What They Are Not.” The American Statistician 30 (3): 203–6.
Sheetal, Abhishek, Zhou Jiang, and Lee Di Milia. 2023. “Using Machine Learning to Analyze Longitudinal Data: A Tutorial Guide and Best-Practice Recommendations for Social Science Researchers.” Applied Psychology 72 (3): 1339–64. https://doi.org/10.1111/apps.12435.
Shen, Xiaotong, and Hsin-Cheng Huang. 2020. “Discussion of "From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation".” Journal of the American Statistical Association 115 (529): 152–56. https://doi.org/10.1080/01621459.2018.1543597.
Singer, Judith D., and John B. Willett. 2003. Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. Oxford University Press, USA.
Snijders, Tom A. B., and Roel J. Bosker. 2012. Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling, Second Edition. Sage.
“Stan.” n.d. Stan. Accessed April 22, 2025. https://mc-stan.org/.
Stolwijk, A. M., H. Straatman, and G. A. Zielhuis. 1999. “Studying Seasonality by Using Sine and Cosine Functions in Regression Analysis.” Journal of Epidemiology and Community Health (1979-) 53 (4): 235–38. https://www.jstor.org/stable/25568956.
Stram, Daniel O., and Jae Won Lee. 1994. “Variance Components Testing in the Longitudinal Mixed Effects Model.” Biometrics 50 (4): 1171. https://doi.org/10.2307/2533455.
Štrumbelj, Erik, Alexandre Bouchard-Côté, Jukka Corander, Andrew Gelman, Håvard Rue, Lawrence Murray, Henri Pesonen, Martyn Plummer, and Aki Vehtari. 2024. “Past, Present and Future of Software for Bayesian Inference.” Statistical Science 39 (1). https://doi.org/10.1214/23-STS907.
Sung, Lillian, Jill Hayden, Mark L. Greenberg, Gideon Koren, Brian M. Feldman, and George A. Tomlinson. 2005. “Seven Items Were Identified for Inclusion When Reporting a Bayesian Analysis of a Clinical Study.” Journal of Clinical Epidemiology 58 (3): 261–68. https://doi.org/10.1016/j.jclinepi.2004.08.010.
Sutradhar, Brajendra C. 2003. “An Overview on Regression Models for Discrete Longitudinal Responses.” Statistical Science 18 (3): 377–93. https://doi.org/10.1214/ss/1076102426.
Szucs, Denes, and John P. A. Ioannidis. 2017. “When Null Hypothesis Significance Testing Is Unsuitable for Research: A Reassessment.” Frontiers in Human Neuroscience 11. https://www.frontiersin.org/articles/10.3389/fnhum.2017.00390.
Tan, Xianming, Mariya P. Shiyko, Runze Li, Yuelin Li, and Lisa Dierker. 2012. “A Time-Varying Effect Model for Intensive Longitudinal Data.” Psychological Methods 17 (1): 61–77. https://doi.org/10.1037/a0025814.
Taylor, Jonathan, and Robert J. Tibshirani. 2015a. “Statistical Learning and Selective Inference.” Proceedings of the National Academy of Sciences 112 (25): 7629–34. https://doi.org/10.1073/pnas.1507583112.
———. 2015b. “Statistical Learning and Selective Inference.” Proceedings of the National Academy of Sciences 112 (25): 7629–34. https://doi.org/10.1073/pnas.1507583112.
van Zwet, Erik, Andrew Gelman, Sander Greenland, Guido Imbens, Simon Schwab, and Steven N Goodman. n.d. “A New Look at p-Values for Randomized Clinical Trials.”
Vanderweele, Tyler J., Hong, Jones, and Joshua L. and Brown. 2013. “Mediation and Spillover Effects in Group-Randomized Trials: A Case Study of the 4Rs Educational Intervention.” Journal of the American Statistical Association 108 (502): 469–82. https://doi.org/10.1080/01621459.2013.779832.
Vehtari, Aki, Daniel Simpson, Andrew Gelman, Yuling Yao, and Jonah Gabry. n.d. “Pareto Smoothed Importance Sampling.”
Wager, Stefan. 2020. “Cross-Validation, Risk Estimation, and Model Selection: Comment on a Paper by Rosset and Tibshirani.” Journal of the American Statistical Association 115 (529): 157–60. https://doi.org/10.1080/01621459.2020.1727235.
Wainer, Howard, and Lisa M. Brown. 2004. “Two Statistical Paradoxes in the Interpretation of Group Differences: Illustrated with Medical School Admission and Licensing Data.” The American Statistician 58 (2): 117–23. http://www.jstor.org/stable/27643519.
Wang, Wei. 2013. “Identifiability of Linear Mixed Effects Models.” Electronic Journal of Statistics 7 (0): 244–63. https://doi.org/10.1214/13-EJS770.
Wong, Pauline P., Georges Monette, and Neil I. Weiner. 2001. “Mathematical Models of Cognitive Recovery.” Brain Injury 15 (6): 519–30. https://doi.org/10.1080/02699050010005995.
Wu, Lang, Wei Liu, Grace Y. Yi, and Yangxin Huang. 2012. “Analysis of Longitudinal and Survival Data: Joint Modeling, Inference Methods, and Issues.” Journal of Probability and Statistics 2012 (1): 640153. https://doi.org/10.1155/2012/640153.
Yuan, Ying, and Roderick J. A. Little. 2009. “Mixed-Effect Hybrid Models for Longitudinal Data with Nonignorable Dropout.” Biometrics 65 (2): 478–86. http://www.jstor.org/stable/25502309.