MLB umpires show discrimination against non-white players

The author of the analysis (who did this as a senior thesis) explains his work in the following thread:

Everything else aside, this is not a true statement. In very large samples we can see that ice cream sales have a positive correlation with drownings. Ice cream doesn’t cause drownings

Also, just because correlation does not equal causation, doesn't mean it is not the source of the cause.  You can say hitting yourself in the head with a hammer leads to head injuries and someone can say that correlation does not equal causation.  But sometimes it is.

Now do the Yankees.

Disclaimer I haven’t read this paper. But jaffinator is right. At high enough sample size everything becomes significant. So you have to consider the effect size as well, which is the size of the difference between two groups. Here that’s the difference between white and non white umps for some endpoint (maybe called strikes or safe calls or something - I should read the paper).

To be precise with language, there is no experimental design because this is not an experiment. I would say that I don't agree with the interpretation/presentation of the statistical modelling.

In general, when writing/reading social science research (which is what this essentially is) there's a lot of ground in between "this is significant so I believe it" and "this is insignificant so I don't." The results are significant, but there are various reasons that I'm at something less than "this is significant so I believe it." First is the inconsistency with the relationships presented in the paper. If bias is driving these results, why are effects not consistent across some breakdowns (white umps are reported as discriminating against both Black and Hispanic pitchers, but only against Hispanic batters for instance)? Then yes, as others have mentioned the combination of the small effect sizes and gigantic sample sizes (tenths of a percent effect size, millions of observations) makes it extremely easy to detect a relationship, when in reality there may not be one.

I'm not saying that bias doesn't affect strikes/balls but instead that this paper is not strongly convincing evidence that it does.

That's kind of in the paper in the form of controls where you can compare the average marginal effects. Just straight up comparing the AME to the AME of some other controls -

1) 1 magnitude greater than batter All-Star appearances and score effects.
2) Same magnitude as it being any inning other than the first, it being the top of the inning,  (I guess this is home field advantage?), there being a right-handed batter, there being a right handed pitcher, pitcher WAR, and pitcher All-Star appearances.
3) 1 magnitude less than the effects of there being one strike (compared to 0), there being more than 0 balls, there being two outs, there being a runner stealing, there being men on base, there being men in scoring position, batter WAR, and some of the years.
4) 2 magnitudes less than the effects of some years, the bases being loaded.
5) 3 magnitudes less than how far off the actual call was.

All the effects I listed above are statistically significant according to the paper.

An aside, but please don't just report average marginal effects. I understand that the paper is meant to be readable to a wider audience but AME depends strongly on the distribution of the data and it would be nice to see log-odds as well.

I appreciate that people ask questions, whether wokie or otherwise, because the alternative is... not asking questions.  Seriously, how is that better?  I would just like them to be asked in ways that result in defensible conclusions, whichever direction those go.

Statistical significance is driven by sample size and effect size among other things. You can achieve statistical significance between between two groups at very low sample sizes if the effect size is big enough. For example, I wouldn't need many at bats vs major league pitching before a statistical test (and anyone watching) declared I am a significantly different hitter than than Ted Williams.

If you want to detect very small differences between two groups you need increasingly large sample sizes. Say you have a clinical trial with 1000 patients that shows no statistical significance between two drugs. Adding another 1000 patients can absolutely help achieve statistical significance - though the difference between the groups may be small and meaningless for all practical purposes.

In 2013 Jose Iglesias got 9 hits in his first 20 at bats (.450). Based on those ABs alone he was not statistically different than Ted Williams. As his number of at bats increased we could say with statistical significance that Jose Iglesias's batting average is significantly worse than Ted Williams'.

In your example of 2014 vs 2015 mechanical batter, I would suggest that as you approach infinity with your sample size you would start to detect subtle differences between the years. But you are right for two inherently equal processes the effect size is 0 so you need a sample size of infinity to achieve statistical significance.

For this study on ump racial bias (oops still haven't read it), I think jaffinator and others have asked questions about the effect size despite it claiming statistical significance.