Challenges in the analysis of cross-cultural data

Abstract

I am going to begin with shortly discussing potential problems in the analysis of (particularly but not only survey) cross-cultural or longitudinal data, addressing in particular the risk of comparing 'apples with oranges' and possible sources for such problems. Next I will mention traditional statistical techniques to test for comparability of data across groups and expand them to newer methods suggested in the methodological literature to address potential (non-)comparability problems. These methods either identify or try to explain non-invariance. I will question whether some of them may eventually even solve non-comparability problems. Finally, I will relate this methodological discussion to recent suggestions especially in the political science literature to ignore problems of non-comparability, its background and the dangers associated with such a strategy when trying to conduct a meaningful analysis of comparative data.

Another look at data quality in cross-national comparative research

Abstract

The quality of survey data is a function of the three levels of actors involved in survey projects: the respondents, the interviewers, and the employees of the survey research organisations. I argue that task simplification dynamics can occur at each of these levels and the effect of such task simplifications is to reduce the quality of the data. The precise form of task simplification differs at the three levels. For respondents it might take the form of utilizing only specific parts of the available response options; for interviewers it can take the form of asking primarily the demographic questions and fabricating plausible responses for the remainder; for employees of research institutes it can take the form of near-duplication of entire questionnaires. I analyse different cross-national surveys to document various task simplification (and time saving) techniques at each of these levels. The analyses also produce results that run counter the established assumptions, for example, the lower the cognitive achievements of the respondents, the higher the reliability of their responses. This finding is, however, consistent within our task simplification framework.

Keywords: data fabrication, interviewer effects, cross-national comparisons, response styles, task simplifications.