ZHUOJUN LYU

Bootstrap and Monte Carlo Simulation

2021-08-12T00:00:00-07:00

Methods: Bootstrap, Monte Carlo Simulation

I found myself perplexed by the difference and relationship between Bootstrap and Monte Carlo Simulation. Then, I read Comparing Groups: Randomization and Bootstrap Methods Using R and it clearly explains these two methods. This book uses simple language to explain intricate statistical concepts,offering concrete examples with orgnized code. It also introduced about effectively presenting statistical findings in research papers. This book is highly beneficial for individuals inclined toward statistical analysis within the realm of social sciences. The primary content of this blog post draws heavily from insights gleaned from this book.

The bootstrap methodology uses Monte Carlo simulation to resample many replicate data sets from a probability model assumed to underlie the population, or from a model that can be estimated from the data. (p.140)

I think both Bootstrap and Monte Carlo Simulation are resampling methods. Monte Carlo Simulation resamples in a random way, while Bootstrap resamples according to the empirical distribution of the data.

Example

The dataset Latino has 150 observations with two columns. Column 1 is “Mex” which suggests whether the person is from Mexico, and Column 2 is “Achieve” which is the level of the person’s fluency in English. In the dataset, 116 people of them are from Mexico and the other 34 are not.

H0 Hypothesis: People from and not from Mexico have the same level of English. In other words, the average difference of their English level is not statistically significant

Monte Carlo Simulation：

Step 1: Resample 5000 times

permuted <- replicate(n = 4999, expr = sample(latino$Achieve))

Step 2: For the resampled dataset, calculate the difference in means of the two groups

mean.diff <- function(data) {
  mean(data [1:34]) - mean(data[35:150]) 
  }
diffs <- apply(X = permuted, MARGIN = 2, FUN = mean.diff)

Step3：Calculate p-value

(length(diffs[abs(diffs) >= 0.39])+1) /5000 
# 0.39 is the group difference in mean in the original dataset.

Bootstrap：

Step 1: Resample 5000 times under empirical distribution

Step 2: For the resampled dataset, calculate the difference in means of the two groups

Step3：Calculate p-value

library(boot)
mean.diff.np <- function(data, indices) {
  d <- data[indices, ]
  mean(d$Achieve[1:34]) - mean(d$Achieve[35:150])
 }  
nonpar.boot <- boot(data = latino, statistic = mean.diff.np, R = 4999)
(length(par.boot$t[abs(par.boot$t) >= 0.39])+1)/5000

References

Zieffler, Harring, Long (2011). Comparing Groups: Randomization and Bootstrap Methods Using R. Wiley.

Paper Reading 1: Analysis

2021-06-13T00:00:00-07:00

Concepts: Clustered Standard Errors
Methods: Hendonic Regression, Test for Randomization

Gender Peer Effects on Students’ Academic and Noncognitive Outcomes: Evidence and Mechanisms

Jie Gong, Yi Lu and Hong Song, 2019, Journal of Human Resources

Empirical analysis:

$Y_{ics}=\alpha+\beta_{1}Peerfem_{-ics}+\beta_{2}Female_{ics}+\phi X_{ics}+ \tau W_{cs}+\lambda_{sg}+\epsilon_{ics}$

$Y_{ics}$: Measures of academic and noncognitive outcomes for student i in class c of school s
$Peerfem_{ics}$: Proportion of females in i’s class, excluding i;
$Female_{ics}$: Binary. whether i is female;
$X_{ics}$: i’s predetermined characteristics and teacher controls;
$W_{cs}$: Peers’ ability controls, including baseline academic ability for male and female peer.
$\lambda_{sg}$: school-grade fixed effect;

We cluster standard errors at the class level, accounting for correlation in outcomes for students in the same class.

Clustered standard errors: Estimate the standard error of a regression parameter in settings where observations may be subdivided into smaller-sized groups (“clusters”) and where the treatment assignment is correlated within each group. It’s useful when treatment is assigned at the level of a cluster instead of at the individual level.
- E.g., we want to discover whether a teaching technique improves student test scores. We assign teachers in “treated” classrooms to try new technique, while leaving “control” classrooms unaffected. When analyzing results, we want to keep data at the student level (However, classical SE are inappropriate because student test scores within each class are not independently distributed. Instead, students in classes with better teachers have high test scores regardless of whether they receive the experimental treatment). Thus we cluster SE at classroom level to account for this aspect of the experiment.

Main Results

A. Gender Peer Effects on Academic Performance

Examine the gender peer effect on students’ academic outcomes using regression model
Interpretation: All regressions include subject and school-grade fixed effects. The coefficient for the proportion of female peers is positive and statistically significant, which suggests that on average, when a student has more female peers in the class, he or she tends to achieve higher grades. After controlling for predetermined characteristics of the focal student, the teachers, and the academic ability of female and male peers, we find that the effect is consistently positive and statistically significant at the 1% level. The coefficient, 1.019, suggests that a 10-percentage-point (approximately 1.25 standard deviation) increase in the proportion of female classmates raises a student’s test score by 10.19% of a standard deviation.

B. Gender Peer Effects on Noncognitive Outcomes
Interpretation: the estimated impact on mental stress is small in magnitude and statistically insignificant, which suggests that having more female peers does not appear to influence students’ mental stress levels. Table 4, columns 4 to 6, report the estimated effects on students’ social acclimation and general satisfaction in school. Overall, we find a positive effect of having more female classmates on students’ outcomes along this dimension. The effect remains robust after controlling for student and teacher characteristics, as well as for peers’ ability.

Robustness Checks

1.Effects from female students’ ability spillover.
Concern: Effects may come from the spillover of female students’ academic ability and performance, given that the literature has established girls’ advantage in test scores during primary and middle school.

There are some gender differences between female and male characteristics and baseline academic ability, but the magnitudes are small. The pattern of academic performance before middle school is mixed: while male students are more likely to repeat grades, they are also more likely to skip grades.
when we control for the academic ability of female and male peers, the main results remain similar.

2.Teacher assignment, differential teaching and grading.
Concern: it may not reflect better academic achievement, but rather differential teaching and grading by teachers.

Conduct a balancing test on teachers characteristics(i.e., regressing teacher pre-determined characteristics (gender, education, etc.) on female peer proportion and controlling for school-grade fixed effect.)
Results: Most estimates are statistically insignificant, suggesting no strong correlation between teachers characteristics and the percentage of female students.

Paper Reading 1: Overview

2021-06-11T00:00:00-07:00

Concepts: Heterogeneity, Endogeneity, Exogeneity.
Methods: Balancing Test, Cross-cohort Design, Permutation Test with Resampling;

Gender Peer Effects on Students’ Academic and Noncognitive Outcomes: Evidence and Mechanisms

Jie Gong, Yi Lu and Hong Song, 2019, Journal of Human Resources

Data: China Education Panel Survey 2014 (CEPS 2014)

Challenge: Nonrandom grouping of students

if there are unobserved characteristics of students that are associated with both gender composition in the classroom and students’ outcomes, the estimation of gender peer effects would be biased.
To address this identification problem, researchers often exploit cross-cohort variation or use random assignment\

Solutions:
a) Cross-cohort Design : samples a cohort (a group of people who share a defining characteristic, typically those who experienced a common event in a selected period, such as birth or graduation) e.g., In order to research on the relationship between smoking & lung cancer, we match obs in terms of variables such as economic status and other health status so that the variable being assessed, the smoking(independent) can be isolated as the cause of the lung cancer(dependent).

Test for Randomization
b) Permutation test with a resampling approach

for classroom i in grade j, school k:
- for all student characteristics:
  - randomly draw 10,000 synthetic classrooms of the same size from the sample of all students from grade j, school k.
  - Calculate the average value for each characteristic within classroom i
  - Obtain an empirical p-value (proportion of the 10,000 resampled classrooms with lower statistics for the corresponding characteristic within the observed classrooms.)

c) Randomly drop observations and see whether regression results change dramatically.

Data Processing:

Noncognitive outcomes are obtained from students’ responses to eight survey items⇒use component analysis to classify the eight survey items into two categories: (1) the level of mental stress, and (2) the level of social acclimation and satisfaction in school.
Students’ academic performance is measured by test scores, supplement with test scores with students’ self-assessed performance scores.
Normalize each index to have a mean of zero and a standard deviation of one.
Conduct balancing test
- Balancing test: female proportion (dependent)~ predetermined characteristics (independent). test the factor one-by-one. Null hypothesis: insignificant t-statistics, which means predetermined characteristics do not related to the proportion of female students.
Endogenous school choice. While random class assignment is conducted, students’ school choices may not be random⇒ introduce school-grade fixed effects $\lambda_{sg}$ to control.
- Endogeneity: situations in which an explanatory variable is correlated with the error term.
  - When the independent and dependent variables are mutually causal, it leads to endogeneity. In a model, the variable whose value is determined by its relationship with other variables within the model, are endogenous; In contrast, exogenous variables are independent, which have no formulaic relationship.
  - In a model, the dependent variable should be endogenous, and independent variables should be exogenous, whose values not determined by the model. Endogeneity refers to the situation where explanatory variables are not entirely exogenous but exhibit endogeneity.

Main takeaway:

This paper examines gender peer effects on students’ academic and non-cognitive outcomes

Higher proportion of female peers in class improves students’ test scores and noncognitive outcomes, which include their social acclimation and general satisfaction in school. ⇒10% increase in the proportion of female classmates raises students’ test scores by 10.2% of agi standard deviation and improves their social acclimation and satisfaction in school by 7.7% of a standard deviation.
Heterogeneity of gender peer effects. ⇒ Positive effect on test score is stronger among male students, or when the teacher is male.
- Heterogeneity: differences across the units being studied

Channels:

More interactive teaching style, more time allocated to teaching-related tasks ⇒ when there are more female students in class, teachers tend to introduce more discussions with and among students, allocate more time to teaching and grading, and be more patient with and responsible for their students.
Improved classroom environment ⇒Students also report that the environment is friendlier and more satisfying, and that they devote more hours to homework and tutorials.

Clarification: noncognitive outcome: mental stress, social acclimation and general satisfaction in school.
Caveats: No strong support for ability-based spillover from female students.