Data, Methods, Debate
Sara Goldrick-Rab of UW-Madison started an interesting late night tweaching (thanks to @akilbello for term) session on methodological rigor. It continued with an interesting tete-a-tete with Stuart Buck. I'm refraining from drawing parallels between tools of oppression and quantitative analysis
@AmyAlex63 Seriously, if you guys are covering this study, I do hope you've asked someone to review it. Many issues.- Looks like @BrookingsEd plans to release that study this morning. I like Matt Chingos a lot, but the statements are overblown.
- Here's a paper explaining issues w this type of subgroup analysis. 1.usa.gov/R5GMRN Also note sizable crossovers, baseline non-equiv
- Larger tx effects on college among blacks, where T group had more BA-educated parents than C's (p=.09). Not random. bit.ly/PBgqul
- See Table 1, by subgroup, then notice the teeny-tiny attention 2 those serious issues on next page. bit.ly/PBgqul
- Seems like @EdWeek only interviewed researchers w an agenda aimed at supporting vouchers... bit.ly/PBj4Ao
@saragoldrickrab I am slower than you! It takes me a minute. But I was wondering about home effects. That's growth. :D- .@chingos @rweingarten Matt have u read Howard Bloom's piece on effect heterogeneity? These subgroup diffs are shaky at best.
- .@chingos @paul_e_peterson You are really stretching beyond ur results w statements like that. One subgroup, unbalanced, no avg effects...
- .@chingos @paul_e_peterson discussing small point estimates like these in percentages usually leads to overblown conclusions too
- .@chingos @stuartbuck1 Disparity btw table 1 results & text on following pg is stunning.Gliding past key treatment disparities at baseline?
- .@shankerblog @nealmccluskey @stuartbuck1 @chingos I'd like 2 see this study try to be submitted as an NBER paper. Attempt it?
- @DianeRavitch No serious statistician would publish that report. It's grasping at straws to find something nice to say about vouchers.
- @saragoldrickrab grad students don't get this kind of clear breakdown of methods in current research often. this is seriously awesome.
@saragoldrickrab The main conclusion of the study was that vouchers made no difference, but they teased out a way to spin "success" for WSJ
@saragoldrickrab Perhaps we should discuss this a bit more in paper, but I'm not that worried about it given joint sig test.- .@saragoldrickrab @rweingarten Will take a look at Bloom piece tomm, but Af-Am effects robust across outcomes, consistent w/ test scores.
- @saragoldrickrab @shankerblog @nealmccluskey @stuartbuck1 I'd be glad to but I'm pretty sure NBER papers have to be by NBER members.
