Lies, damned lies and survey results

The Scottish Government has recently released the results from the Scottish Household Survey: Satisfaction with Schools 2024. The survey’s reporting merely presents headline results without any contextualisation or analysis. This does a major disservice to local authorities that rely on the Survey for key performance information.

The headline results are:

In 2024, 69% of adults were satisfied with local schools, the same as in 2023 (69%). Satisfaction with schools has generally decreased over time since reaching a peak of 85% in 2011, with a slight increase between 2017 and 2019, followed by a sharp drop to 69% in 2022, which has remained at this level since then. 

Satisfaction with schools in 2024 varied by local authority, with the lowest levels of satisfaction in East Lothian (50%), Midlothian (53%) and Dumfries and Galloway (58%). The highest levels of satisfaction were in East Renfrewshire (88%), North Ayrshire (84%) and East Dunbartonshire (83%). 

(Note: satisfaction is the combined total of very satisfied and satisfied responses.)

The problem with this simple reporting of the survey results is that it does not provide any context or analysis.

Before going any further, a declaration of interest. I am a survey and opinion polling geek and read results of surveys and opinion polls with a critical eye.  I was, until last year, the Corporate Policy and Performance Manager at East Lothian Council, where I was responsible for three Residents Surveys carried out for the Council.

I identify four issues with how the 2024 Household Survey results have been reported.

1. The question asked in the survey, “Overall, how satisfied are you with… Local Schools”, is open to interpretation depending on what aspects of school provision and education services respondents base their answer on.

Various aspects of education and schooling could determine how respondents reply to this question: the quality of education provided, the latest exam results, the state of school buildings, whether the council is proposing to close a school, concerns about anti-social behaviour and bullying in schools, and whether children with Additional Support Needs are being supported.

Although the results of the question provide some indication of overall satisfaction or dissatisfaction with schools, they do not offer any insight into why respondents responded in the way they did.  The only way to get that insight is to ask more detailed questions in the survey or in follow-up ‘focus groups’.

2. The survey results, as reported, do not account for people who had no view – those who responded ‘Don’t Know’.

The options given to respondents were Very Satisfied, Satisfied, Neither Satisfied nor Dissatisfied, Dissatisfied and Very Dissatisfied. This means that even if the respondent had no current experience of schools, they would have been forced to provide an answer.  Many people with no experience may have ticked the ‘Neither Satisfied nor Dissatisfied’ box, but this can be interpreted as indicating a lack of strong feelings about the service (i.e., it is okay, could be better/ could be worse), as well as a lack of opinion (i.e., ‘Don’t Know’).

3. Given the uncertainty about the Neither Satisfied nor Dissatisfied option, a much better and more reliable gauge of satisfaction with the service can be found from analysing the result of the question from people with ‘skin in the game’; i.e. service users, in this case, parents of school-aged children.

For all the ‘satisfaction’ questions in the Scottish Household Survey, responses of all respondents, irrespective of whether they use the service in question, will be clouded and influenced by personal bias, media coverage (e.g. banner headlines berating a council service) and anecdotal conversations with family and friends.

In the satisfaction with schools question in the Scottish Household Survey, there was a significantly more positive view of schools amongst service users (where there is a school-aged person in the household) – 81% compared to the 63% satisfaction recorded for respondents who were not service users.

When reporting on satisfaction levels, the responses of service users should always be considered alongside those of all respondents.  But the Scottish Household Survey samples at the local authority level are too small to provide a breakdown of results for ‘service users’.

4. The biggest concern wth how the Scottish Household Survey is reported is in relation to the detail provided for responses at a local authority level.

The survey is based on a large-scale sample of approximately 5,500 people across Scotland, ensuring statistical reliability with a 95% confidence interval. However, the local authority results are based in some cases on samples of as low as 120.  The reliability of results from such small samples is very dubious.

One result of the small local authority sample sizes is that there are much greater variations between years for local authority results than for the national results.

The Scottish satisfaction level remained at 69% through 2022, 2023 and 2024. However, for East Lothian, which was quoted as having the lowest satisfaction level in 2024, the surveys from the last three years showed that satisfaction levels rose from 75% in 2022 to 88% in 2023 (the highest amongst 32 local authorities) and then fell dramatically to 50% in 2024 (the lowest).

As an East Lothian Council resident and former senior officer, I can attest that there were no significant policy, funding, or quality of service changes between 2023 and 2024. What could have been the cause of East Lothian’s shift from being the local authority with the highest satisfaction levels in schools in 2023 to the lowest in 2024?

The low sample sizes – 140 in 2023 and 150 in 2024 – will almost certainly be a significant factor in explaining this variation.  Another statistical factor (see section 2 above) is that the percentage of respondents in East Lothian who said they were Neither Satisfied nor Dissatisfied rose from only 6% in 2023 to 36% in 2024 (the Scottish average was only 22%).  The percentage of respondents who were dissatisfied varied from 4% in 2022 to 6% in 2023 and 14% in 2024.

Apart from my geeky need to ensure that data from surveys is understood and analysed properly and rigorously, why is this important?

Firstly, there is the issue of how the media – traditional and social – can misuse survey results, and in some cases, use them to attack or denigrate a council.  So the local East Lothian paper carried a story with the headline, East Lothian schools have lowest satisfaction in Scotland. Whilst the article did include comments from the council regarding the small sample size, there was no analysis of the survey results or explanation as to why East Lothian schools’ satisfaction levels fell from the highest to the lowest in one year.

Secondly, the satisfaction questions from the Scottish Household Survey are used as the basis for seven indicators in the Improvement Service’s Local Government Benchmarking Framework (LGBF). If the statistical reliability of the local authority results is in question, then how much reliance should we give to the results as reported in the LGBF?

I am the last person to suggest that the Scottish Government or local Authorities should stop carrying out residents’ surveys.  They can provide useful data on public attitudes and perceptions. However, I would strongly urge that when the results of these surveys are reported, more attention be given to interpreting and analysing the methodology used and what lies behind the headline results, rather than just taking them at face value.

As the old adage goes: Beware of Lies, Damned Lies and Survey Results.