Procurement and AI: big opportunities and big challenges

Anna Studman and Mavis Machirori from the Ada Lovelace Institute discuss the potential and challenges of integrating AI technologies in local government, highlighting the need for effective procurement processes to ensure societal benefits. They emphasise the risks of damaging public trust through misuse and the confusion stemming from inconsistent guidance and lack of clear definitions in AI governance.

Background

There are high hopes for the transformative potential of AI for the public sector, and for local government in particular. This was demonstrated at the Local Government Association’s Technology Innovation Showcase in London this November. At the event, groups of councils presented ‘challenge statements’ (covering complex areas like social care and customer service) to which vendors presented data and AI solutions.

There is a clear need – and desire – for councils to embrace technology for efficiency and productivity gains, and ultimately to deliver better services. Councils cited the inordinate amount of time social workers spend on admin, the difficulty communities and some frontline professionals have accessing information about council services, and the poor allocation of resources that they hope technology can start to address.

But use of AI in local government is high stakes. There is a significant risk of damaging public trust and causing harm to vulnerable people if things go wrong. We’ve already seen examples of simpler algorithmic systems failing with profound consequences. Newer AI technologies present additional complex challenges, such as malfunctioning in unexpected ways.

OpenAI’s Whisper transcription tool has been found to ‘hallucinate’, by introducing phrases or sentences that never appeared in the underlying audio. While OpenAI  warns against using Whisper in ‘high risk domains’, more than 30,000 clinicians in the US are using a tool based on Whisper’s technology for transcribing appointments.

The New York City government implemented a Microsoft-powered AI chatbot to help business owners and landlords with queries about local laws and policies. Despite being trained only on official NYC.gov websites, the chatbot was found to be providing false information that went against the law, such as recommending that business owners take a portion of their workers’ tips, which is illegal in New York City.

Both transcription services and chatbots may offer targeted solutions to local government challenges, but the examples above show that caution is required. Procurement is one mechanism local government should use to guard against these errors.

In local government, AI technologies are more likely to be purchased from external suppliers than developed in-house, so procurement is an important process through which the public sector can interrogate the quality and impacts of technologies and hold suppliers to account.

Our research

At the Ada Lovelace Institute, we set out to explore how well procurement processes are functioning as a lever for ensuring data and AI technologies deliver societal benefits. Throughout our research on AI and data-driven systems in public services – ranging from digital healthcare in the NHS to local government use of data analytics – procurement has emerged as an important process for scrutinising technology.

We spoke to experts involved in different stages of procurement – from data governance and engineering to those buying and supplying AI technologies – and found that for this to be achieved, procurement processes and infrastructure need to change.

To understand the governance landscape in which local government procurers operate, we analysed 16 pieces of official guidance and legislation and found that local government in England does not have the infrastructure, support and guidance it needs when it comes to procuring AI.

Our analysis revealed that there are more than 50 terms used across the guidance and legislation to refer to societal benefit, including 14 terms related to fairness and eight related to transparency. There was also no cohesive definition of ‘AI’.

This creates a confusing foundation, which is further complicated by a lack of clarity about how and where to operationalise these concepts. People we spoke to in local government said this makes it difficult for buyers to engage suppliers in conversations about the broader social impacts of their products.

To further understand the experiences of buying AI in local government, we held discussions and a workshop with stakeholders from across local and central government and industry.

Challenges

There are multiple issues facing procurers in local government, which minimises the ability to ensure positive social outcomes from the procurement and use of AI. The first is that, as above, disparate guidance and narrow legislation limits what is possible in practice.

Another challenge is related to data and information systems held within councils and used by vendors (and in effect, by developers or designers). There are multiple layers to data usage when it comes to AI technologies. These go beyond data quality and accuracy but extend to data uses. We heard that it was not always clear if data held by councils was appropriate for modelling and training technologies – i.e. is it too sensitive to share? Is there adequate information to fully understand and critique what suppliers are demanding? Are data protection impact assessments adequately conducted for how complex and black-boxed AI technologies work? Do data subjects consent to, or know about, those uses of their data?

Local government stakeholders told us that it is not always clear what AI can or cannot do, which makes it difficult for them to cut through the hope and hype presented by AI vendors and decision-makers in the UK government. This echoes our previous work on the use of predictive analytics within a local authority in London, where we heard from frontline workers that they needed to understand how AI or predictive tools were generating insights in order to trust the technology.

There is also a recurring issue with the lack of shared knowledge of what AI is being used and where, whether that AI works and in general, what its impacts are. This speaks to a fear of being seen as failing to innovate if a technology does not produce expected results. This can scupper the ability of councils to jointly interrogate AI technologies, rather than work in silos.

This knowledge gap worsens the existing knowledge and expertise imbalances between local government and the private sector. The current skills gap between public and private sectors is well documented and we’ve seen large tech companies stepping in to guide local government on the implementation of AI. This situation might stifle competition in the sector and limit choice for procurers.

This is indicative of market failures that lead to monopolies in the AI market. In our research we heard how big tech companies priced out or outbid SMEs, sometimes supported through the use of favoured procurement frameworks. Some of the practitioners we spoke to told us how these monopolies or market capture played out: local government was unable to hold suppliers to account, as they did not have the right contracting templates and regulatory/legislative support to pursue redress if promises were not realised.

The way forward

There are clear priorities for better supporting local government in the procurement of AI, including streamlining guidance documents and clarifying definitions of key terms, creating practical tools like contract templates and assessment frameworks, identifying metrics for success when using AI in local government, and further upskilling for council staff on assessing or auditing AI systems.

The challenges can seem daunting but many solutions are already within reach. Across local government, several organisations are working at the coalface to support better procurement, adoption, use and scrutiny of AI. But what we’ve found is that while there are multiple groups doing good work, they are not always well-resourced and sometimes work in silos from each other, especially in early stages. To achieve the priorities set out above, a more joined-up approach is required.

We are therefore calling for a National Taskforce for Procurement of AI in Local Government to continue and elevate the work that is already being undertaken in a more cohesive and supported way. Though we would like to see local government more empowered, we note that central government support will be key to any change to the overall procurement ecosystem.

Any solution to the challenges set out above should prioritise collaboration. This can be achieved by involving expertise on the ground from local government, by jointly setting clear metrics for success, and by working transparently and in constant dialogue between local and central government from the outset. 

We believe that a taskforce presents an exciting opportunity to change and support local government procurement of AI, and to ensure it benefits people. We see a real opportunity for central and local stakeholders to come together to bridge the gap between the aspirations of technology in the public sector and the reality on the ground in local government.

Conclusion

Procurement is a vitally important process for ensuring local government can uphold its responsibilities to the public – especially in regard to transparency, integrity and trust – when it comes to using AI. But this aim can only be realised if some important changes are made.

The time to act is now. The public sector is in the relatively early stages of AI adoption –  teams are starting to do good work but are often siloed, and the pressure to make high-stakes decisions is heaped on individuals.

While enabling change requires working with central government, there are actions that councils can take to be in a better position for adopting and deploying AI.

• Be transparent. Transparency across individual organisations is important, not only for staff but for residents. Guidance from DSIT suggests keeping a central record of where AI is in use, what it is being used for, which teams are involved and how it is assessed or checked. We also encourage local government to engage with the Algorithmic Transparency Recording Standard, which helps public sector organisations provide clear information about the algorithmic tools they use, and why they are using them.
• Involve and support diverse council voices. This includes working with cross-council groups to address AI challenges and to help facilitate knowledge sharing from the ground up, instead of being driven by central government or the private sector.
• Rethink social value. Individual councils can also consider how social value – already a core part of procurement processes – may need to adapt when it comes to buying AI. Thinking about social value as an expected outcome and aim of the procured technology could help councils track the impact of these technologies on issues like inequalities and accessibility.
• Be proactive about sharing insights. Councils can share insights among themselves to better understand what is and what isn’t working. Collective knowledge can lead to better definitions of metrics of success for the use of AI in local government. Shared learning can help identify pockets of expertise in local government that others can draw from in their own decision making.

There are many remaining questions, such as how to involve the public in these decisions, how to scale up and down between local and central government, and how to achieve change without an extra burden on local government. But it’s clear that a collaborative approach and a commitment to societal benefit will be the way forward.

We call for a more collaborative working relationship between local and central government and we believe a national taskforce for the procurement of AI is a positive and necessary next step. This will ensure local government are confident and supported in the adoption of AI amid a rapidly changing technology and governance landscape.

About the authors: Anna Studman and Mavis Machirori are part of the Ada Lovelace Institute – an independent research institute with a mission to ensure that data and AI work for people and society. Ada promotes informed public understanding of the impact of AI and data-driven technologies on different groups in society. Through research, policy and practice, Ada guides the development and deployment of these technologies in a way that centres social wellbeing.