Three Methodology Questions: Pew Research Center’s “Jewish Americans in 2020”

While the Jewish institutional world begins to debate implications of the latest Pew report, I want to ask (sincerely, not rhetorically!) three questions about methodology — and, by implication, about how much and in what ways we should trust the findings.

Disclaimer 1: I’m not a social scientist of any kind. I officially Do Not Know What I Am Talking About when it comes to survey methodology, so you should read me with a lot skepticism on this. I may simply be misunderstanding things that would be obvious to experts, and if so, I hope someone will let me know. I’m going ahead and writing this despite my methodological ignorance because I figure if I’m confused by things, other non-experts might be confused as well.

Disclaimer 2: I’m an employee of a Jewish organization, but I ask this set of questions purely in my personal capacity as an individual American Jew, not representing the views of anyone else.

Disclaimer 3: Please believe me that I’m not here to cast aspersions or hurl accusations. Or even draw conclusions! I recognize that social science is hard, valuable, and much easier to criticize than to do. In a spirit of curiosity and respect I am literally (and at the risk of being That Guy) “Just Asking Questions.” If and when someone provides some legible expertise to answer these questions in ways I find useful, I’ll gladly update this post to quote and/or link it.

All throats having been cleared, let us proceed.

1. Does this study tell us about “Jewish Americans,” or “Jewish Americans who trust institutions”?

Given all the problems pollsters had with political polls in 2020 regarding low-trust voters not responding to polls, how do we know these findings didn’t have significant biases related to trust?

What if some significant number of Jews got screeners from an organization they saw as tied to the American establishment (in whatever way might be meaningful for their perceptions) and thought, “This isn’t for me”? And what if that kind of lack of social trust in mainstream institutions isn’t distributed evenly across various sub-categories of the Jewish population? In particular I wonder about this regarding Haredim and Jews of Color, but I don’t say that with any certainty, and trust issues could theoretically affect any number of other findings beyond those demographics.

If there’s anything about this possibility in the Methodology, I’m not seeing it. (Though it does note that “the response rate to the screening interview was lower than projected.”)

2. What about Yiddish-speaking Hasidim?

The screener questionnaire, from which all else flowed, was in English, Spanish, and Russian. (Cf. the Methodology.) What about Yiddish? [Cue some guy in the back to shout, “WHAT ABOUT LADINO?”]

Without Yiddish, isn’t there a significant chance of missing responses from a significant number of Hasidim for whom Yiddish is their first language and an English screener might have been either a barrier or enough of an inconvenience that this population was undersampled?

3. What about the pandemic?

This survey was conducted between Nov 19, 2019 and June 3, 2020. Pew Research Center, of course, could not have predicted this timing because they do surveys, not extispicy. Nonetheless, I think they might set some minds at ease by more specifically addressing how the study might have been affected by illness, grief, quarantine, social distancing, and an atmosphere of general panic and upheaval during a goodly portion of their survey timeline.

It’s not that they leave this completely unaddressed. The report says that “most of the interviewing was completed before the coronavirus pandemic hit the United States,” and that, therefore, “the results by themselves do not paint a clear picture of the pandemic’s impact on Jewish Americans.”

I wish I knew, though, what “most” means in that sentence. Were, say, 99.9% of interviews done before the pandemic hit? Were 75% done? Was it just 51%? No timeline of rolling survey completion figures is given, and for this particular time range, mightn’t one help? Since COVID-19 did not hit all parts of the country, or all parts of the Jewish world, with the same intensity, how that timing shook out could theoretically have a meaningful impact on what may have been going on among those who did and (especially) didn’t respond.

Again, this is for clarity.

I don’t want to tear down Pew or anyone else. I love knowledge, I am grateful to this study’s authors for contributing to American Jewry’s collective self-knowledge, and I’m simply curious about the extent of some potential limitations. If all the concerns above are legitimate concerns, then the value of this report will not nearly drop to zero, it will just suggest a need for some new interpretive work along those lines. And if, in fact, someone shows me that all the concerns above are non-concerns, then I will be glad to have been wrong, as well as glad to have prompted clarifications on these matters and provided new reasons to bolster the report’s credibility.