From AI Policy to AI Management System What General Counsel Need to Know

Introduction

[00:00:03] Sacha Kirk: Hello, and welcome to another episode of In-House Matters by Lawcadia, where I talk with legal leaders about what matters to them. I’m Sacha Kirk, Co-Founder and Chief Strategy Officer at Lawcadia. Today, we’re talking about AI governance, and in particular, what happens in practice.

[00:00:18] AI is being used across most organisations in some form, but having an AI policy does not mean that you have effective AI governance. And the question we’re talking about today is really: how confident are you that the governance around AI is actually working? Joining me is Jonathan Hunter, an experienced General Counsel, AI governance practitioner, and founder of Laero.

[00:00:39] Jonathan has more than 25 years of experience as a General Counsel, and through his consulting business, Laero, is now working with organisations to develop AI management systems aligned with ISO 42001. I’ve known Jonathan for a number of years, and he has recently completed an engagement with Lawcadia in assisting us setting up our own AI Management System, which is currently undergoing the external audit for ISO 42001.

[00:01:05] I asked Jonathan to be a guest for this podcast series as AI governance is a topic that he is incredibly passionate about, and this is a great opportunity to share his knowledge and experience with our broader community. Jonathan, welcome.

[00:01:18] Jonathan Hunter: Thanks, Sacha. As you said, I’m always happy to talk on this subject.

[00:01:21] Sacha Kirk: I’m really keen to explore the gaps in AI governance.

Jonathan’s Journey into AI Governance

[00:01:25] Sacha Kirk: Jonathan, you’ve spent more than 25 years in General Counsel roles. What led you to move more deeply into AI governance, and why has this become such an important focus for you?

[00:01:35] Jonathan Hunter: Thanks, Sacha. So I’ve been working in technology companies for a number of years and was heavily involved in the AI boom a few years back when everyone started experimenting with ChatGPT and later Anthropic and Copilot.

[00:01:48] Most businesses were and still are encouraging their staff to experiment with AI. I jumped onto that bandwagon and started playing with it myself with various productivity tools and agents and did a little bit of vibe coding of apps and those sorts of things. I quickly realised three things as part of that process.

[00:02:06] One is that people were being given a green light to experiment, but very few people actually really understood what they were doing. It was new to everyone, right?

[00:02:16] Two, many businesses didn’t really know all the AI systems that were being used by their staff or more importantly, how they were being used.

[00:02:24] And thirdly, only the largest players in regulated industries are really thinking through controls and risks before or at least around the same time as they were giving a green light to this experimentation. My background in tech companies, the business and the board tend to have a high-risk threshold.

[00:02:39] So discussions at board level were more about how the company was moving forward with AI and how quickly it could adopt it, what productivity gains was it getting from it, which was very little, and not really what controls were being put in place around it. So, once I started looking into it, I realised that putting governance around AI is quite different to other controls.

[00:03:01] It’s such a new area that it really requires some creative thinking and new approaches, and that’s what really attracted me. It requires both an intellectual and a strategic approach, as well as being pragmatic and practical, and having all of those things or trying to hit them all on the head can provide some interesting challenges.

[00:03:19]

Common AI Governance Gaps

[00:03:19] Sacha Kirk: For you, personally, with what you’re seeing in the market? What are the common governance gaps?

[00:03:24] Jonathan Hunter: I mean, that’s an easy one. The most common gap is that there’s practically no governance. So most organisations understandably got caught up in the excitement and potential of AI without really even thinking about governance, which, to be frank, has traditionally been less interesting and lower priority issues for businesses.

[00:03:42] A lot of businesses quickly put a relatively simple AI usage policy in place, and that was it. I mean, that was all we understood. That was all we knew. With time, we’ve developed a far greater understanding of both the potential gains from AI, although we’re still exploring that obviously, but also the risks.

[00:04:00] We’re now seeing that play out in the media. Every week we’re seeing or we’re reading about some previously unknown or unexpected adverse outcome from AI. So, it’s important to recognise that AI is, to use a coined phrase, a socio-technical system. That phrase was coined by people who are far more intellectual than me.

[00:04:20] But what it really means is an AI system’s behaviour and impact come from the interaction between the technology on the one hand and the people, the organisations, the rules and contexts around it, right? Not from the model or the technology alone. The way it impacts so many aspects of our lives in so many ways, it’s actually dangerous to consider it as simply another form of technology.

[00:04:44] So the governance around it needs to also cover off on all those aspects. It needs to take a holistic approach.

[00:04:51] Sacha Kirk: What really stands out for me is understanding the impacts on the people in your team, the impacts on the community, impacts on your clients, impacts on other social aspects. And that’s a different way of thinking about it, isn’t it?

[00:05:04] Jonathan Hunter: Yeah, it is. It’s so different from how we’ve approached, these sorts of governance structures in the past, and to be completely frank, governance was never a really attractive area for me. It was AI that was attractive, and the governance then grew around that and proved to be something quite different from anything we’ve had to do before.

What is an AI Management System?

[00:05:22] Jonathan Hunter: I now want to explore, the AI Management System framework, and for someone hearing the term for the first time, it’d be good if we can start with an understanding about what is an AI Management System, or AIMS, as it’s sometimes called.

[00:05:35] Yeah, sure. So an AIMS is, I guess most simply put, as a set of policies, objectives, roles, processes, and controls that an organisation uses to govern the development, the provision, or the use of AI systems in a repeatable and auditable way.

[00:05:54] The name was coined by ISO, the International Organization for Standardization, for its international standard, ISO 42001. Most people will be familiar with ISO 27001. It’s the international standard for information security management systems, or ISMS. If a business already has an ISMS, an AIMS can build onto that with specific AI controls.

[00:06:19] If it doesn’t have an ISMS, an AIMS can be created as its own standalone management system. Either way, an AIMS requires a very different approach, when compared to an ISMS. The international standards are very different. They work in very different ways and, without going into all the details, I’d say, an AIMS requires a more thoughtful and strategic approach for the reasons mentioned earlier.

[00:06:43] It’s not just a sort of a tick the box exercise. So, for example, you downloaded a set of policies from the internet or AI, I’m pretty confident you would soon find it to be pretty ineffective. And by ineffective, I mean the policies would sit in your policy manual and probably wouldn’t be used or applied.

[00:07:01] So there’s no real governance that would come from that, right? An effective AIMS covers, and this is the intention of the international standard as well in the way that it’s framed, an effective AIMS covers not just policies, but also planning, clear and practical processes and controls, and I would say, most importantly, it covers people.

[00:07:18] That’s your trickiest element.

[00:07:20]

Adopting the AIMS Framework

[00:07:20] Sacha Kirk: You’re absolutely spot on with that. For me personally, there’s been immense value in the framework, and I think from a people perspective, it gives non-technical people like me a structure that I can understand. That is valuable even if you are in an organisation that isn’t intending to ever get certified for ISO 42001.

[00:07:38] What is your view in what is relevant and what is of value for organisations approaching AI governance through that framework?

[00:07:46] Jonathan Hunter: I come from a technology company background, but I am not previously, intensely tech-savvy, But, you don’t need to be entirely tech-savvy to put something like this in place, right? I think the starting point is if you’re using AI, you need to put governance around it.

[00:08:01] So let’s start with that, right? It needs to be a little bit more than a usage policy. We’ve all heard about agents going rogue, confidential data being inadvertently fed into a large language model, people relying on AI hallucinations and getting themselves into all sorts of strife.

[00:08:17] There are plenty of examples of this, particularly in the legal industry. And there are biases that are inherent in our society that are being amplified by using AI without proper controls. So the AIMS framework, as I mentioned, is not a tick the box exercise.

[00:08:30] It provides guidance on how to think about governing AI. So if you’re not planning to get certification, as Sacha was saying, you can adopt as much or as little as seems appropriate to your business. I think that’s a really important perspective. It needs to be, specialised for you and the way that you work.

Who Should Own AI Governance?

[00:08:47] Jonathan Hunter: So the important thing I would say to start off with is it’s not just a CTO job. I’ve seen many organisations put responsibility for an AIMS with the tech team because it involves technology, and to be fair, that’s normally where Information Security Management Systems sit. So it’s natural to put the AIMS there too

[00:09:05] In fact, I would say it requires representatives from across the organisation to come together and to consider the risks and to debate and discuss the issues. And so bringing together technology, risk, operations, sales, HR to create a common language and a common understanding of how you use AI in your business and how to manage it.

[00:09:29] This exercise, if you will, forces you to work through the important questions, which is more than just policies and procedures, right? It’s everyone across the business needs to understand their responsibilities, needs to understand risk assessment, controls, oversight and accountability.

Assessing if Governance is Working

[00:09:47] Sacha Kirk: So Jonathan, when it comes to governance, the critical issue is what happens in real life. So once you have those governance structures in place, how do you assess whether they are actually working in practice?

[00:09:59] Jonathan Hunter: Yeah. So, let’s assume you’ve already got a great AIMS framework set up or some sort of AIMS framework set up. You’ll see a couple of basic essential elements of an AIMS. One, having an AI governance committee, two, having an AI system register. So the governance committee oversees AI usage across the business, and then all uses are recorded in the AI system register.

[00:10:22] It’s a source of truth. So how do you know if it’s working? The starting point, I would say, should be how many queries and escalations are you getting from staff? Really, you should see that number increase, particularly while people are experimenting with AI, while we’re in this experimental phase.

[00:10:38] People may be wanting to create and test new agents, put some of them into production. They’ll be using APIs and MCP connectors between systems. These are examples of activities that should be known to the organisation, recorded in some way The framework should also help you to quickly assess whether a proposed activity is relatively innocuous and can just be waved through, or whether it demands closer inspection and a risk assessment.

[00:11:05] If it does require closer inspection, a good governance structure should make it easy and transparent for staff to follow the assessment process and make decisions or escalate as required. I would say most importantly, it needs to work in with the way you and your staff operate. There’s absolutely no point creating policies and processes that sit in your intranet or HR portal that aren’t practical to use, right?

[00:11:30] To go back to your question about how do you know if it’s working, the way AI is being used these days so prolifically, you would really need to test the effectiveness of your framework every now and then. And what you’re testing is, are just different decisions being made because of the framework?

[00:11:46] Are processes being followed or are they overly burdensome? In which case you probably need to make some changes, right? Are the right things being escalated to the AI governance committee? Does their agenda have the right issues coming through it? There’s just a few examples.

[00:12:00] Sacha Kirk: From our own experience, some of the best conversations were around a lot of those key decision points.

[00:12:06] So, exploring, if we do this, what is that friction, if that is there, what is that then going to create? Because, we want to avoid things like, shadow AI or avoid things where, people are inadvertently putting information of a certain classification into certain AI tools.

[00:12:23] And it’s really understanding what are some of the behaviours, and how can we make sure, put processes, protocols, and technical things in place, so then how can you guide people to do things the way you need to do it? Having it in a policy is not sufficient.

[00:12:38] Jonathan Hunter: No. And look, what can we control? What can we cannot control? You can’t control everything- right? So with the things that you can control, sure, you can put some processes and policies around it. But there are things that you can’t control. How do we manage those issues? How do we mitigate?

Slowing Down to Speed Up

[00:12:52] Sacha Kirk: One of the challenges is also this idea of slowing down.

[00:12:55] Is the governance process going to slow things down? I’ve always taken the view, having gone through the AIMS process and the framework, that actually by slowing things down, we can then speed things up. What’s your perspective on that?

[00:13:08] Jonathan Hunter: I would say a good governance structure should enhance adoption, not slow it down. So yes, you’ll go through a bit of a process, but everyone is already using AI, and most people have not got a governance structure around that. So let’s start with the reality. We’re already behind the eight ball on that a little bit.

[00:13:27] But if you clearly understand and set out what AI uses are already approved, how they can be used, and then the clear process for approving new use cases, what circumstances require reassessment, and then you’re training your staff across all of it, then you’re facilitating effective and safe rollout of AI.

[00:13:45] An off-the-shelf check the box approach, which may have worked in the past for ISO 27001 and they’re being offered by the certification bodies, they won’t necessarily get you there. An AI governance framework should work with your way of doing business and just fit into it and become part of it, right?

[00:14:01] Doesn’t need to be overly onerous. The level of governance should be proportionate to your organisation’s risk appetite and the types of AI usage. So for example, an SME using ChatGPT to create presentations and track internal sales data, that would take a very different governance approach from, say, a health technology company with AI built into their customer facing platform.

[00:14:23] Completely different scenarios, very different governance. But I think it really comes back to AI governance now becomes just a part of your doing business, right? And you’re using AI in every part of your business. So the two things go hand in hand. It shouldn’t be seen as a slowing down of any sort of process.

Accountability in AI Governance

[00:14:38] Sacha Kirk: Now, one of the complexities that is common because AI responsibility can sit across technology, security, privacy, risk, legal, procurement, and then individual business units. Yeah. How do you avoid a situation where everybody is responsible, so nobody is responsible?

[00:14:53] Jonathan Hunter: Yeah. So as I mentioned earlier, you’ve got an AI governance committee, which ideally involves representatives from across the business. But that doesn’t mean that everybody on the governance committee is responsible for everything AI. The new buzzword, we’ve probably all heard it, is accountability.

[00:15:09] See that term cropping up everywhere as one of the key answers to governing AI, and that’s absolutely right when you’re talking about the people aspect. One aspect, I guess, is accountability . So if I get a little bit legal now, we’ve all accepted that the large language models like Anthropic and OpenAI, they generally won’t take responsibility or liability for AI outputs and impacts, except where they’ve created their own agents and they do harm like hacking Medicare, but let’s put that to the side.

[00:15:34] So that means an organisation and its staff using AI, they’re responsible for their usage, not the LLMs. So that doesn’t mean that everyone is responsible for everything, and it doesn’t mean the AI governance committee is responsible for AI use by the entire organisation. A good AI governance framework will help determine the accountability of individuals for particular AI uses and use cases. So for example, if you create an agent that you’re testing, like in a sandbox environment, you are responsible for ensuring that the agent isn’t connected to unsanctioned systems outside the sandbox.

[00:16:07] It doesn’t move into production without appropriate approvals and guardrails. Another example, you use AI to create a customer presentation, you are verifying the output, right? I mean, your manager also has a certain level of responsibility if you’re a more junior member of staff, you get your work approved by your manager as you normally would.

[00:16:26] That’s just the normal processes. AI obviously doesn’t change that. So where that gets you is that an important part of AI governance is staff training, right? You need to teach your people not just how to use AI, but how to verify its outputs, when to escalate issues, when to avoid inputting personal information or client confidential information, when they need to be transparent about having used AI.

[00:16:52] Transparency is something that’s now really coming to the fore, in thinking, in AI governance. Those are all covered off by a good governance framework.

The Role of the General Counsel

[00:17:01] Sacha Kirk: And what about the role of the General Counsel? They may not always be leading AI governance, but they certainly have a significant influence.

[00:17:08] What’s your perspective about where the GC should be involved?

[00:17:11] Jonathan Hunter: Yeah, the GC is the person who’s usually ultimately responsible for managing the organisation’s risks, right, at the highest level. So you may be fortunate to have a CRO, but often they’ll also report to the GC. I would say, if we stick to this context, the GC is a critical member of any AI governance committee.

[00:17:30] Whether or not they’re tech-savvy, GCs have a good instinct for risk management, right? I mean, that’s ultimately what we do as GCs, we manage risks. So there are a few things I’d say about that in the context of AI. Probably I’d start with there are very few experts out there. Everyone is learning as we go.

[00:17:46] So the rule that there are no stupid questions, that applies here more than ever before, so don’t be intimidated by it. Another thing I would say is AI is already so prolific and touches so many aspects of our lives. The human element in AI governance is huge. I’ve already said that a bit, but I’ll probably rant on about it a little bit more.

[00:18:07] Like, bias, mistakes, ignorance, assumptions, they’re all going to be tested and magnified by AI. So that’s why I say there should be a diverse group of people in the room when assessing a significant AI use case in your organisation. It’s really common knowledge these days that a group of people with diverse backgrounds and experience add the greatest value to an outcome.

[00:18:34] AI is where this really rings true, and I’ve seen it myself. It’s amazing how many different opinions, approaches, issues, and most importantly, solutions come from discussion about AI with a diverse group of people Then I’d say getting a bit more practical, data control and leakage. Everyone who’s watching would know that’s a huge issue in this space.

[00:18:57] So it’s not just a jurisdiction issue, but it’s the way that data flows through AI systems. It’s different to how traditional systems behaved. So I would suggest you learn about inference processing, data processing locations and retentions with LLMs, obviously data storage locations, but also like MCP and API connectors, because those sorts of things can magnify the issues at hand.

[00:19:20] I recommend GCs get involved in these things because it’s bloody interesting, you know? It’s new, it’s changing every week. It will add a new dimension to your day-to-day work as a GC if you’re not already in this. So find the areas that you’re particularly interested in and insert yourself.

Questions GCs Should Ask

[00:19:35] Sacha Kirk: I think that’s great advice. And I know from personal experience, I’ve learned and grown so much in understanding and getting my head around a lot of this as well. Specifically for General Counsels who are listening to this, what could be some questions that they could be asking?

[00:19:50] Jonathan Hunter: So to start with, I would find out, what are the AI use cases throughout the organisation, sanctioned and unsanctioned, if you can. Then, what data is being used in AI? So does it include confidential customer data or personal data? And then you bring those two things together.

[00:20:07] Which AI systems are being used with which data? I think that’s your starting point. That’s the critical area, right? I would also say, I would ask, has the AI usage policy recently been updated? Does it properly reflect the current uses? I’ve seen a lot of these policies that were put in place when AI first became widespread, and are now quite simplistic and out of date.

[00:20:28] So revisit those. And another thing to consider would be, you may or may not conduct privacy impact assessments, but I’d also consider AI impact assessments for the main use cases. I’ve seen them be a really valuable exercise to get the relevant stakeholders thinking about the possible issues.

[00:20:46] Sacha Kirk: I think that last one is a great one because it also puts the accountability on the person who wants to be using it to really be thinking through the impacts. Yeah. And they are often the best people to be thinking through it.

[00:20:56] Jonathan Hunter: Absolutely.

[00:20:56] And then it forces them to bring the other people into the room, to discuss it and get those different diverse perspectives as I mentioned earlier. And it was very intentionally created. I’ve spoken to one of the writers of the standard 42005, which covers AI impact assessments.

[00:21:11] She had a very clear perspective on bringing that diverse group of people into a room for the AI impact assessment to get the best outcomes.

[00:21:19] Sacha Kirk: That’s fascinating that it was just so deeply considered and purposeful. So for anyone on the call, if they already know that their AI adoption in the organisation has moved ahead of its governance, and I think there’d be a lot of situations like that, what advice would you give them?

Get Started with AI Governance

[00:21:34] Sacha Kirk: What should they be doing first?

[00:21:36] Jonathan Hunter: I mean, firstly, I’d say you’re part of a majority of organisations. AI governance is only just starting to actually become a thing. So I’d start by finding out how AI is currently being used throughout your organisation, and also the intended uses in the near future. It’s changing so quickly. You can’t create a workable governance structure without first knowing what you’re governing. The governance structure, it needs to fit around how your organisation is using AI. It shouldn’t work the other way around. I can’t see how that would lead to a working governance model.

[00:22:06] So that’s what I’d do first.

[00:22:08] Sacha Kirk: If you could leave General Counsel with one piece of advice about their role in AI governance, what would it be?

[00:22:13] Jonathan Hunter: I’m repeating myself a little bit here, so I apologise for that. Something I mentioned earlier: don’t leave it with your CTO. GCs play just as crucial a role in AI governance as a CTO, a COO, a CPO.

[00:22:26] Your C-suite and your senior representatives from across the organisation should all play a role given how widely AI is used and how significantly AI impacts an organisation and almost everyone in it.

Wrap-up

[00:22:39] Jonathan Hunter: Jonathan, thank you. I think what comes through strongly from this discussion is that general counsel and leaders do not need to become or be AI technologists to play an important role in AI governance, and that diversity of thought and having those diverse opinions and thinking in the room is really important, and that really shone through .

[00:22:56] I think An AI Management System really can help provide a way of creating that structure and moving towards governance that can be embedded in how the organisation operates. For everyone on the call, as a follow-up resource, I’m going to be sharing a link to a case study that Jonathan prepared for the Lawcadia AI Management System project.

[00:23:15] So this case study outlines, the challenges that we faced, the approach that we took, as well as highlighting some of the valuable support and leadership from Jonathan. Thank you so much for joining us for this episode of In-House Matters by Lawcadia.

[00:23:27] I encourage you to reach out to Jonathan on LinkedIn, and if you’d like to find out more about Lawcadia, please visit our website at lawcadia.com.