AI legislation at least as impactful as EU privacy rules

Walters’ extensive analysis of 15 large and small companies, and a supplementary literature review, show that in many organisations preparations for the AI Act are still far from ideal. Training on the risks of bias in data collection and AI models is still often lacking. Also, while the AI Act identifies specific requirements for technical documentation, there are often no guidelines yet for such documentation.

MUCH STILL UNCLEAR

Walters notes that it is not surprising that companies are not yet adequately prepared, as much is still unclear regarding the European regulations, which are worded in rather general terms. “Companies have been working on this for two years, because the basis for the AI Act was already established in 2021. The Act is very specific on certain matters, particularly when it comes to very high-risk AI, such as models that determine what benefits someone is entitled to. But the Act is not very clear on other matters, such as the high risk involved in incorporating AI into your ‘critical infrastructure’. What the Act will mean in practice on those matters is still ambiguous.”

MORE IMPACT THAN THE GDPR

Nevertheless, preparation is essential, because the regulations are going to have far-reaching consequences, Walters believes. “This is going to have just as much impact on companies as the GDPR, if not more. Most companies are already using AI in one form or another, but sometimes they don’t even know where or how it is used. If you run your own webshop, you might not even realise that AI is used in a recommendation system, for example.”

GREATER CHALLENGE

What also makes the impact of this European legislation so great is the fact that privacy is something companies can add later on. “What’s tricky about the AI Act is that you have to carry out risk assessments on rights and discrimination, and you have to be able to demonstrate that you’ve thought it all through properly. You also have to use certain methodologies, and companies do not know which ones yet. The GDPR is much more concrete because it deals mostly with anonymisation or pseudonymisation of personal data. So this is going to be a much greater challenge for organisations.”

BOTTLENECKS

For her research, Walters highlighted relevant sections of the AI Act and translated them into 90 survey questions. A variety of organisations participated, after which Walters assigned scores for how often a company or organisation performs a particular required action (sometimes, regularly, rarely or always). “The average score was 58 percent, so there is definitely room for improvement.”

Scores were particularly low for technical documentation. “Companies do not yet have protocols for writing technical documentation, whereas the AI Act calls for very specific things in this area,” according to Jacintha.

Knowledge of the risks of ‘internal AI models and data’ is also still lacking. “Most of the companies surveyed stated that they had not encountered any risks in the use of their datasets in the last two years. This is unlikely because all datasets and models involve risks. Therefore, more training is needed in this area.” The companies did, however, score well on keeping their AI models up to date.

SURPRISED

“What really surprised me during this Master’s programme is that there are so many risks involved in the use of AI,” Jacintha says. “Bias is a major issue; there are so many groups to consider. But the more variations you add, the less effective a model becomes. Therefore, as an organisation, you have to keep asking yourself whether it’s better to automate things, or whether the risks involved are so great that it’s better to refrain from doing so. There will be many more dilemmas about this in the near future, also for municipalities in the Netherlands that are increasingly using AI.”

MORE INFORMATION

Jacintha graduated from the Master’s programme in Applied AI through the Centre for Market Insights, under the supervision of researcher Diptish Dey and professor Jesse Weltevreden. Jacintha collaborated with AUAS researchers Diptish Dey, Debarati Bhaumik and Sophie Horsman on the scientific paper ‘Complying with the EU AI ACT’, which is set to be published soon.

As a result of her insights, Jacintha decided to start her own consultancy: Babelfish, to assist organisations in the responsible use of AI models and support them in preparing for AI regulations.