Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Confounding variables don't invalidate research, they merely inform us on how to weigh the evidence, and shape future research.

Studies can have interesting conclusions even in the presence of confounding variables. Sometimes the state of the art in a certain hypothesis space hasn't progressed past of of these yet, but that's still better than gut feeling.

In some areas like public policy research you will never completely eliminate all of these factors, but that's no reason not to attempt to create knowledge.



Sure, you will never completely eliminate all of these factors. But this is far, far short of "not completely eliminated". This is, "the confounding variables are almost certainly much larger effects than what you are trying to measure". The difference between python, ruby, and php in make a web framework, is much much smaller than the difference between web frameworks, statistical analysis packages, machine learning, and devops scripts. Python is used for the last three, ruby and php much less so. The difference between python and ruby and php is therefore going to be dominated by the kinds of tasks they are used on, and the difference between the languages is small compared to that.

Smokers are routinely excluded from analysis of cancer rates, for a similar reason; the effect is so large that if (for example) you're looking at diet, and the difference in diet is correlated with a difference in smoking rates, any findings are invalid.


I don't think GitHub mining is state of the art, and there are opportunities, as the article says, to separate the signal from the noise. I agree in the general sense, but there should be some lines drawn at which point we can say research is invalid so people don't run away with the results.


Yeah, the original study is clearly so methodological unsound that it's conclusions shouldn't be repeated.

But fankly most of the posts here are people almost calling the original authors heretical for even trying to data mine commit messages.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: