Eurovision judging - it's noisy! (and biased) 1/2

By Mark House

Recently I had the opportunity to observe from the inside the deliberations and machinations of a panel of experts asked to judge competition entries. Following the Eurovision model, the organisers had elected for a combination of a public vote and an expert panel.

The correlation between the rank order in Eurovision given by the expert panel process and the public vote process has been a subject of analysis and debate. Studies and observations indicate that there is often a noticeable disparity between the two voting processes.

In this 2-part blog series we will be investigating how noise and bias influence human judgement, how each is effected by the judgement process, and what we might do about it.

Eurovision: Key Points on Correlation:

Disparity in Rankings

There is a significant difference in how expert panels and the public rank the performances. For instance, in some years, the public's favorite performances have received lower scores from the expert juries, and vice versa. This disparity was particularly evident in the 2023 contest, where Finland received the second-highest televote in ESC history but was significantly lower in the jury rankings

Historical Trends

Historically, the jury and public votes have not always aligned. Between 2009 and 2014, the jury and public disagreed on the winner only once. However, from 2015 to 2023, they disagreed on the winner seven out of eight times.

Understanding Noise and Bias in Eurovision Voting

The Eurovision Song Contest employs a dual voting system, combining the judgments of professional juries and public televoting. While this approach aims to balance expert opinions and popular appeal, both voting methods are susceptible to noise and bias, which can influence the outcomes. Let's examine the potential sources of noise and bias in each voting approach.

Jury Voting

Noise

  • Individual differences among jury members, such as personal preferences, musical backgrounds, and cultural perspectives, can lead to variability in their assessments of the same performance
  • Situational factors, like the order of performances or the surrounding acts, may influence how juries perceive and score a particular entry

Bias

  • Cultural biases can arise from shared linguistic, ethnic, or religious ties between jury members and the performers, potentially skewing their evaluations.
  • Geographical proximity and historical connections between countries may also introduce biases in jury voting patterns.
  • The professional backgrounds of jury members, such as classical musicians or radio presenters, could shape their preferences and biases towards certain musical genres or styles

Public tele-voting

Noise

  • Individual differences in musical tastes, cultural backgrounds, and personal preferences among the general public can lead to significant variability in televoting patterns.
  • External factors like the order of performances, stage presence, or visual elements may influence public perceptions and voting decisions, introducing noise

Bias

  • Cultural biases stemming from shared linguistic, ethnic, or religious ties between the public and the performers can sway televoting patterns.
  • Geographical proximity and historical connections between countries may also introduce biases in public voting

It sounds noisy.......and biased!

Both the jury and public voting systems have inherent strengths and weaknesses. While juries aim to provide expert assessments based on technical merits, they are not immune to cultural, geographical, and professional biases. On the other hand, public televoting reflects popular appeal but can be swayed by factors beyond the musical performance itself, such as visual elements, familiarity biases, and political influences.

To mitigate the impact of noise and bias, the Eurovision Song Contest employs a combination of jury and public voting, with the hope that their respective biases and noise will balance each other out. However, as evidenced by the research, both voting methods are susceptible to various forms of bias and noise, which can potentially influence the final outcomes.

What can we do top reduce Noise and Bias in human judgement?

These concepts, extensively explored in Daniel Kahneman's book Noise: A Flaw in Human Judgment, highlight the inherent flaws in human judgment and offer insights into how we can mitigate these errors.

In the second half of this blog series, we will explore how Adaptive Comparative Judgement might be used to reduce both noise and bias.

Photo by Elyas Pasban on Unsplash