Unequal participation rates in surveys or false responses can distort data about voting behaviour. This can create a problematic bias that should be addressed with complementary methods.
Diesen Artikel in deutscher Sprache lesen.
fter every election, it’s the same story: political experts analyse, lead candidates explain, and journalists report. Voter migration, successful issue framing, and coalition options are discussed. For some, the days following the election determine the course of their careers, while for others, it determines whether they will soon have to adapt to faster deportations or a further increase in air pollution. To achieve a comprehensive contextualization of election results and a well-founded discussion, „good“ data on the development of voter turnout, approval ratings of current policies, or the evaluation of candidates by voters is needed.
„No response“ as an essential part of surveys
During the collection of this data, it should be clear to respondents which response options they can choose from. Even the mere act of reading the different options could influence respondents by triggering certain memories (Diekmann, 2007, p. 446 ff.). This is particularly true for quantitative surveys where respondents are asked closed-ended questions, such as: „Which party did you vote for in the 2017 federal election?“. In such cases, respondents should be provided with a list of all the parties that ran in the 2017 federal election as answer options. Additionally, to account for unforeseen response possibilities, it may be useful to provide an „I prefer not to answer”-option for those who do not want or cannot provide a response (e.g., because they do not remember).
The author
Clara Weißenfels is PhD student at the program „Political Economy of Inequality“. Her research focus: political participation and power relations.
Systematic Withholding or Just Bad Luck?
Dealing appropriately with methodological challenges in capturing political behaviour or attitudes has been a focus of many dedicated researchers for decades. Many studies analyse survey data in which respondents answer questions either online, over the phone, or in person. The primary interest lies in capturing as many individuals as possible to apply quantitative methods. However, two particular problems have emerged. Firstly, people who are politically engaged are much more likely to participate in surveys than those who are less politically integrated („oversampling“). Secondly, actual behaviour does not always align with respondents’ reported answers. This often occurs due to perceived social expectations (Philipps and Clancy, 1972), whereby voting is seen as socially acceptable and not voting is socially sanctioned. Respondents tend to claim they voted even if they did not, which distorts the results („overreporting“ of voter turnout). Both problems have been documented in various social and national contexts (Selb and Munzert 2013; Sciarini and Goldberg 2016) and can be mitigated, at least in countries with accessible voter registers, through measures such as statistical weighting and comparing reported and actual behaviour.
However, even if respondents participate in the survey but refuse to answer certain questions („item non-response“), a problematic bias can emerge in the data. In this case, the goal of collecting comparable information from all respondents and analysing it is not achieved. Alvarez and Li (2022) investigate whether it is possible to identify respondents who refuse to answer out of boredom or inattentiveness in online surveys. Berinsky (2017) argues for appropriate question formulation to avoid overwhelming respondents. He makes a surprising observation for many researchers and/or politically interested individuals: „most of the people, most of the time, do not pay attention to politics“ (p. 317).
In recent years, research has increasingly focused on the phenomenon of respondents consciously withholding or providing false information about their sympathies towards, for example, right-wing populist parties. These individuals may feel that their political opinions are socially undesirable or they may reject participation in scientific surveys a priori, perceiving scientific studies – perhaps even in line with their own political beliefs – as untrustworthy. When there is a systematic relationship between individuals’ characteristics or attitudes and non-response, it results in a problematic distortion of the results. The goal of analysing such data is to identify precisely these systematic patterns, and if entire groups refuse to participate in the survey, it becomes impossible to do so. However, there might also be no connection to their specific political orientation, and the lack of response may simply be arbitrary.
Series on Inequality and Power
Growing inequality is one of the most significant problems of our time. At the same time, scientific interest is increasing and providing new insights with a view to the most pressing questions and answers on various dimensions of inequality and its underlying power structures.
For the debate series „Inequality and Power“, doctoral students from the doctoral programme „Political Economy of Inequality“ at the Institute for Socio-Economics at the University of Duisburg-Essen have written down these new findings. In the articles, the doctoral students, who are funded by the Hans Böckler Foundation, present partial results of their research and discuss related societal challenges as well as political pathways. With a focus on inequality dimensions and underlying power relations, the thematic arc ranges from poverty and taxation to labour market, gender equality or climate policy. Through the thematic breadth and diversity of the methods used, the authors seek to initiate a wider societal debate on how to counter rising inequality.
The series has been published in German language in the Makronom magazine between April and June 2023. We also document the series here at ifsoblog.
Who Provides „No Response“?
In longitudinal studies like the German Socio-Economic Panel (SOEP), which has been conducting comprehensive surveys with households across Germany since 1984, participants can choose to provide „no response“ to all questions (data set: SOEP-Core, v37, EU Edition). In the following, we examine the composition of those who chose „no response“ when asked about their voting decision in the 2017 federal election in the SOEP. The respondents had various options for indicating how they distributed their votes among the participating parties. In 2017, only 7.7% of the respondents did not provide an answer, making it difficult to derive statistically significant correlations. Cramér’s V measure showed no statistical significance for the observed irregularities in the calculation. However, examining the composition of the group still yields interesting insights.
For both years, respondents could indicate whether they did not vote (if eligible), which party or parties they voted for, or whether they wanted to provide „no response.“ Invalid votes or invalid responses within the survey were excluded from this analysis.
In the 2017 federal election, the group that chose „no response“ consisted of 1,859 individuals, while 19,853 respondents indicated their voting decision. This means that the overall ratio was 92.3% responded, 7.7% did not. I was particularly interested in how these numbers differed when we look at gender, education level, interest in politics, poverty status, and income. The following analysis takes into account the size of the groups, i.e., whether, for example, more men or women were surveyed, as it proportionally represents the response behaviour.
Women (gender is measured as binary in the SOEP) more often chose „no response“ (8.3%) than men (7.1%). Respondents with a secondary school degree (Realschulabschluss), considered as „intermediate education level,“ particularly frequently refused to provide an answer. Interestingly, respondents with a lower secondary school degree (Hauptschulabschluss) or no degree disclosed their voting participation at the same rate as the highly educated group.
6.9% of those living below the poverty line refused to answer, while the proportion was 7.8% among those who were not poor. This result shows that the economically disadvantaged in this analysis were more likely to provide answers compared to those with household incomes (weighted) above the poverty line.
The higher the level of political interest, the higher the proportion of respondents who answered. While „no“ or „not much“ interest leads to refusal rates of 9.4% and 9.1%, respectively, it decreases to 6.1% for strong interest and 4.8% for very strong interest – the lowest refusal rate observed here. Thus, political interest appears to play a significant role in the decision of whether respondents provide their voting decision.
To represent the income distribution of all respondents and divided into provided or withheld responses, I first calculated income deciles. This divides the group of respondents into ten equal-sized groups. The lowest 10% represents those with the lowest income, the second decile includes respondents who have between 10% and 20% of the income distribution, and so on. This makes the data more robust against outliers due to particularly low or high incomes. The following graphic shows how incomes are distributed within each group proportionally.
The graphic shows the „kernel density estimators,“ i.e., the probability that a respondent has the displayed income if they belong to the respective group. Individuals who provided „no response“ (yellow line) have a higher probability of having incomes in the lower range of up to 1,500 euros per month.
The median income for the entire group of respondents is 1,666.67 euros, for those who provided responses it is 1,736.67 euros, and for those who withheld responses, it is 1,700 euros.
Politically interested individuals have the lowest dropout rate
When analysing voting behaviour using data from the SOEP, it becomes apparent that we obtain responses primarily from respondents who have a higher interest in politics, which raises concerns about data collection. This contribution by no means questions the quality of the SOEP, whose staff ensures a valuable data source for Germany through constant quality control and elaborate weighting procedures. Rather, it aims to question whose opinions we are actually capturing with questionnaires and whose opinions are „falling by the wayside.“
One’s own political behaviour is closely linked to emotions and learned behaviours (e.g., Marx 2019). So why do people refuse to provide insight into their voting behaviour? Is it simply a lack of interest in the question, a lack of recollection, or distrust towards the interviewer? Or are there entirely different reasons?
These questions could be specifically examined in personal interviews or focus groups to achieve an overall improvement in data collection. The appeal of the extensive and elaborate data collection in the SOEP lies precisely in the comparability of data across various contexts. The prerequisite for this is the uniform capture of the target population, i.e., the group about which researchers aim to make statements. In the case of the SOEP (and other large panel datasets), this target population is none other than the entire population of Germany. However, when certain groups fail to answer the questions for unknown reasons, it is crucial that we address this phenomenon of „non-response“ to specific questions more closely.
The Political Economy of Inequality
The PhD program „Political Economy of Inequality” investigates extent, causes, and consequences of rising socio-economic inequality. While economic aspects of inequality are a key focus of the research, they are always contextualized with non-economic dimensions of inequality. Political, social, and ecological causes and consequences of material inequality are systematically integrated into the analysis. The research at the Institute for Socio-Economics is characterized by an interdisciplinary and applied socio-economic approach. See all blog posts by members of the program.
This article has first been published at makronom.de.
Brief summary
There can be various methodological problems with survey data on voting behaviour, such as the oversampling of e.g. politically interested respondents, false accounts of respondents or item non-response. The analysis of item non-response in German panel data confirms that respondents with lower political interest and lower income are more likely to not provide an answer on who they voted for in the last national elections. This indicates that information for the more affluent and politically interested citizens is more abundant for research and public discussion, possibly skewing the debate. This adds to the discussion about the appropriateness of survey data for research on political preferences. One possible improvement would be to word questions more carefully, while complementary methods like personal interviews or focus groups might also be required to circumvent these limitations.
