On-Demand Webinar: “Managing AI Tools: Oversight, Risk, and Compliance in Practice”

HCRR #114: Managing AI Tools: Oversight, Risk, and Compliance in Practice

Presenters: Valerie Rock, Miriam Murray, and Erin Walker

This transcript was generated with AI and may contain errors or omissions. Please refer to the webinar recording for the most accurate version.

00:08 – Welcome and introduction

PYA Moderator [00:08]: Good morning, everyone. Welcome to today’s episode of PYA’s Healthcare Regulatory Roundup Webinar Series. Today’s topic is managing AI tools, oversight, risk, and compliance in practice. PYA is happy to present today’s webinar on this important topic. With that, I would like to introduce our presenters: Valerie Rock, Miriam Murray, and Erin Walker.

00:35 – Why healthcare AI oversight matters

Valerie Rock [00:35]: Thank you, Jen. Thank you all for joining us today for this presentation, whether you’re listening to it live or after the fact, we appreciate you joining. I’m Valerie Rock. I’m a principal with PYA, and we want to share with you today, you know, really the concepts of the infrastructure behind AI and how to implement that, but we don’t want to scare you from using AI. We want to make sure that you’re actually using it and being functional with using it, but there is a way to use it that is safer for you, and if you create the infrastructure in place and have that, you know, as kind of your due diligence on the way you’re implementing AI, that will show an external party that comes in that looks at you that says you’ve created an error or something like that, and you can say, well, we’ve done everything that was expected to be done, and so we’re, you know, we’re showing you that we did, we tried, right, we tried to make sure that we were mitigating this risk, so if we show that we’re doing our due diligence, that just as you would implement a compliance program to mitigate risk and show an effectiveness of that compliance program. We want to show an effectiveness of AI oversight. So this is a discussion of that, and I have here today with me both Erin and Miriam, who are going to give their insights on this, both from an operational and kind of an IT and security perspective, so we each come from a different perspective. I oversee all of our revenue integrity services and kind of assist from that perspective, both from coding diligence, from operational oversight, and relate in relation to coding, and also reactive reviews related to any type of external party, so we’re seeing AI from a whole, you know, lots of different facets that are impacting that realm. So, Erin, you want to describe a bit about your background as well?

Erin Walker [02:37]: Sure, thanks, Valerie. Good morning, everyone. My name is Erin Walker, and I’m a manager at PYA, and I live in the regulatory world, whether it’s from compliance programs, whether it’s HIPAA privacy and security, helping organizations make sure that their programs are effective, are functional, and are operationalized appropriately, and with that, as Valerie mentioned, in her world with AI in my world under the regulatory lens, we are seeing a lot of AI come into play, and just its management and being knowledgeable about appropriate ways to onboard and bring on these tools, and then operationalize them in a way that allows you to continue to monitor and make sure that you keep appropriate oversight is really becoming an important component of these programs of all the programs, all the compliance programs, whether you’re looking at it through a privacy lens, a security lens, or just a seven elements compliance program lens. So excited to be here and speaking with you all today, and looking forward to our discussion. Miriam,

03:41 – AI use cases and operational and regulatory risks

Miriam Murray [03:41]: thank you, Erin and Valerie. I’m Miriam Murray. I’m a senior manager with PYA, and I’m a utility player. I am both on the regulatory compliance side as well as the revenue integrity side. My purview is both from a regulatory and compliance side, but also operational, and how it impacts our clients, and so today we’re going to talk a little bit about all of those pieces and parts, so we’ll let’s get started. So today we’re going to focus on how you do AI in practice, so looking at the operational and regulatory risks, and then from there walking through how organizations are managing AI tools, the technology, the vendor relationships, and how oversight is being integrated into existing compliance structures, we’re also going to spend some time on what effective human oversight looks like in an AI environment, not just in theory, but in day-to-day operations, and finally we’re going touch on how vendor due diligence is evolving, particularly as AI capabilities are embedded into third-party solutions. So, let’s get started. So many organizations are already using AI, even if they don’t call it AI. It’s embedded in our EHRs. It’s being utilized in revenue cycle tools, HR platforms, vendor solutions often without formal review or transparency from the vendor. So, the risk starts with not even knowing where AI exists. In addition, AI adoption is moving faster than governance, both externally with our federal government and state governments, but also internally within our own governance structure. Teams are turning on features and adopting tools before compliance, privacy, or legal has any visibility or knowledge, creating gaps in oversight. AI is a life cycle. Models are changing, vendors are doing updates, outputs are evolving. So risk isn’t static, it’s constant and requiring ongoing monitoring and revalidation. The issue isn’t using AI, it’s using it without oversight. That’s where we see the real exposure, privacy risks, biased outputs, failure to validate results, unsupported billing decisions, and potentially audit issues down the road, if you don’t know what AI is being used, where it’s being used, how it’s being used, you can’t govern it. If you can’t explain it, you can’t defend it. So we know it’s embedded in all of these areas already. Many are already using it from a clinical operation standpoint, looking using tools like ambient listening, note summarization, scribe services. They’re helping our providers be more efficient in their documentation, and honestly I’ve heard feedback that it’s better patient care, patients are more satisfied, they’re actually talking to the provider versus talking to the back of the head of the provider who’s trying to key in the information in the computer, so utilizing these tools has been a benefit, it’s helping them be more efficient and provide better patient care, but it’s also creating risk, especially if they’re not reviewing what’s coming out of the tool, and they have that responsibility to review the information and validate its accuracy. Same thing for coding and charge capture tools. These tools are flagging codes, suggesting codes, suggesting charges, which can increase efficiency, but it can also increase audit risk for organizations, especially if you don’t catch that there is an error coming out, it can multiply exponentially from a coding and billing standpoint. Providers are using diagnostic tools, especially in imaging. I know mammograms, I know they’re using it in the GI world to identify anomalies, tumors, but again, providers have to be looking at this information and reviewing it, validating the accuracy, and then patient triage and symptom checking, you know, methodologies they’re using digital healthcare apps and portals, which could influence care, and it can also introduce risk for inaccurate or biased results. So, appropriate oversight is critical in the clinical and operations section. When you get to compliance and privacy and legal chat bots, Copilot, Gemini, Claude, those are all being used, implemented in companies, in organizations now as a tool to help these functions provide services where the risk is not validating the results that come out of those queries, so it’s extremely important to make sure that if they are using them that they validate the results for accuracy, and then you’re also looking at auditing and monitoring tools, looking for anomalies in billing and coding, these are good things to use, but again, there is risk because you do need to validate the results. They’re also using them from an administrative function.

Miriam Murray [10:07]: I don’t know how many of you have utilized them for screening resumes or candidate rankings. Boy, it’s helpful. I sure can tell you, when you’re getting hit with, like, a couple hundred resumes, it’s hard to vet those tools quickly, so there is a benefit, but you have to make sure that the tool you’re using isn’t creating bias or potentially unfair hiring, so you’ve got to review those, as well as performance analytics and employee surveys, they are being utilized for those to monitor potentially employee productivity, or you know, performance. So, looking at these tools, utilizing these tools, there is also risk involved. So, understanding what that risk is. So, just be mindful of what the risk could be, so the risk is real, it’s real, it’s there for each of these sections, you know, risk of exposure of PHI, PII, how we’re monitoring the rules and the regulations related to HIPAA or regulatory, the state governments are already starting to come out with rules or laws to address the use of AI. You’ve got to stay on top of those clinical and operational safety, making sure that you have good patient care, and you’re looking at validating the results that come out, data quality and bias. You’ve got the risk of potentially incomplete data. Where is your data source? It could lead to unreliable results, or unreasonable results, or inaccurate results. You’ve got to pay attention. There’s your risk right there, and then you’ve got your third-party risk. Are they being transparent in using AI, embedding AI in their processes? Are they letting you know that, and then governance and oversight, and having a lack of structure or lack of processes in monitoring these services, it’s all there, and it’s hitting all of these areas. Ladies, anything to add to this section? All right. Well, we’re going to move on.

12:36 – Governance, AI inventory, and data handling

Erin Walker [12:36]: Thanks, Miriam. This next section, we really want to take some time to put some structure around AI use, but we all know adding structure, adding more requirements, more policies, more processes. We want to do it in a way that doesn’t slow us down more than it needs to, that allows us to have the appropriate oversight mechanisms in place, but still allows us to provide patient care, do our jobs, and do it in a way that actually makes it worthwhile, right? So I want to focus on a few core things, and Miriam has already touched on them. There’s three really quick questions to ask yourself initially. Who owns the AI? Who’s going to be not only who owns it, if we’re using a third-party vendor, but who’s responsible for it internally. It’s not just IT. What’s being used? So, is it PHI, is it administrative data, is it financial data? What’s being used and going into the tool, and what regulations must we pay attention to protect that information. So, with that said, I really want to talk about some foundational elements when it comes to management of AI. If you take one thing away from my section, I want you to hear that these are really the fundamental components to determining, you know, do we have an appropriate structure in place to effectively manage these AI tools and technologies. It starts with governance. It always does. It’s not just, do we have a policy? I’m seeing a lot of, there are policies, there are even AI subcommittees that have been implemented, but nobody really knows what AI is in place. Nobody can really get their fingers around who’s approving the use of AI. They don’t really have AI use cases. It might just be somebody found the cool new toy that really does great things, but there’s never been a use case done for it. And who owns the risk of use of the tool? Because, as Miriam mentioned, the risk is there, it is real, and we really have to have a governance component that we start with when we’re looking at these tools. I mentioned. And inventory, it is so important to have an inventory of all of the AI in your environment, and that can be shadow AI, that can be, you know, other forms, the listening that’s being utilized, or just all your vendors that are using the AI, but even, or that are helping you implement the AI, but it goes further than that, and it goes back to that administrative part that Miriam was talking about, Copilot and Claude. I was actually just talking with someone the other day, and we were talking about putting stuff together, and why? Just throw it in Claude, okay? But you know, are we verifying, are we making sure that what Claude is giving us is accurate and appropriate, because again, these tools are great. They really help provide efficiencies. I think it’s great for patient care, but again, we have to make sure that we’re still keeping that human-in-the-loop, and that we’re still verifying what is coming out of it. Data use and handling is another foundational element, and this is where HIPAA comes in, and other state privacy laws. Now they’re expanding to personally identifiable information, or PII. This is where we really have to think about the minimum necessary to effectively use the tool. What needs to go in it versus what are we just throwing in it? Do we have guardrails around what can be put in it, what cannot be put in it? Have we trained on those guardrails? Do people truly understand the importance of only putting the minimum necessary information into those tools that is needed to actually perform the work that we’re trying to get done, once you define that, though, then it comes down to ensuring the tools are configured appropriately. So, we may have decided what our minimum necessary standard is going to be, what information can go in there, and what information should not go in there. But is the tool configured appropriately to make sure that is what’s happening, Now that we’ve talked about initial setup, initial foundation.

17:07 – Risk assessment, vendor risk, and security safeguards

Erin Walker [17:07]: How do we do this ongoing oversight and management of these AI tools? First, we start with a risk assessment, and those in compliance, and IT, and enterprise risk, and quality, and patient safety risk assessments are abundant, right. We’re doing them in lots of capacities within our organizations, but AI needs to be part of those assessments. We need to integrate it into those assessments instead of doing it separately, and the reason is because it can affect all those areas. If we’re using it for patient care, it can affect quality rev cycle, coding, documentation, patient safety. It can affect our IT environment. It can affect our compliance landscape. And so we really need to include it within our risk assessment processes. One area that I have seen over time is that AI is kind of being siloed out. We’re keeping it separate. We’re not embedding it into our processes like we do with ERM, and it really needs to be. That’s how you’re going to get the most effective picture of what your risk looks like with use of these tools. Vendor risk, this is where I’m seeing one of the biggest gaps. Miriam touched on it. Most organizations we’re seeing are still utilizing traditional vendor due diligence when they’re looking at an AI vendor. It’s got to go further than that, because those traditional due diligence processes, they’re not designed to catch some of the areas that need to be looked at when implementing AI tools and technologies with these tools and technologies, it’s just not enough to ask, are you secure, is it secure, are they compliant. You really need to understand how the tool works, what data it touches, and not only that, but what happens to that data over time. Some of these tools are using that data separately for other reasons, and you know some tools can touch data that you don’t intend for it touch, and so that’s why it goes back to that configuration component being so important, but from a compliance perspective, transparency and audit rights when it comes to these AI tools and technologies, are critical. You need to be able to have audit rights as it relates to these tools, especially for the third-party tools that you don’t own and that you don’t manage, that are managed by a third-party. You need to be able to go in and show that we have audited these vendors and we have been able to document that we are staying within the guardrails that are required, not by not just by our own policies and procedures, but by regulatory requirements. If a vendor tells you they don’t want you to ask questions, or they are, they don’t want you to verify what they’re doing, you don’t have any oversight into it, you just have a tool that you’re relying on a third-party to tell you is effective, and finally, security safeguards. The way I think about this is, are we treating AI tools differently than any other application software that we might add into our environment? It can’t sit outside of your existing security controls. You have set security controls, security safeguards for those reasons, and AI needs to be built into those as well. At a minimum, you need baseline requirements. What about encryption? Are we thinking about multifactor authentication? What about access controls? How are we? How are we managing who can access the actual tools and technologies, and what types of vulnerabilities do these introduce into our environments? Are we looking at them from a do we have to open, you know, certain parts of our network in order to utilize those, and what risk does that pose, and is that an acceptable risk, just by very nature of use of the tool, what vulnerability might we wind up with that we hadn’t yet thought of, because it wasn’t something that was needed to be thought about until these tools came into play. I see a lot with organizations, they’re creating exceptions for AI use cases, it’s the well, it’s an AI tool, so we’re going to handle this separately, but they really can’t be.

21:26 – Human review of AI-generated clinical documentation

Erin Walker [21:26]: They really have to be integrated within your own security infrastructure, and if that means updating or enhancing the security infrastructure, I know that can be a lot of work, and I know that can also be kind of stressful, so you know, but it needs to be where everything is treated the same, and there aren’t exceptions for AI use cases, because if something does happen, that’s going to be a hard question to answer, and you’re going to have a hard time proving that it was an acceptable risk to keep the AI out of your regular security landscape and your security infrastructure, so what we wanted to do, we’ve talked a lot about AI, and here’s the foundational elements, and here’s the risks, but let’s look at it from how does this even affect us? Where are we? Where are we seeing this? Miriam touched on this already, but you know, AI-generated notes, AI suggesting orders that can be so helpful. It can allow that you’re not looking at the physician’s back of the head, you’re actually having a conversation with your provider. You know, you’re feeling seen, you’re feeling heard, but the risk, they’re not always complete and accurate. It’s not that clinicians are using the tool wrong, it’s the opposite, it’s they’re relying on it more than they should, you know. Questions to think about, if your staff are accepting generated notes without reviewing them, because what’s happening is they may review for the first couple months and everything’s right, and so this tool is great, and we, you know, this is amazing. So they stop reviewing because AI has a tendency to make you trust more than you should, and there should never be a case with AI where you’re not using the trust but verify method. You can lose that clinical oversight into what’s being documented, and then you can find yourself down the road with some issues. The medical record, as we all know, still has to represent the clinician’s professional judgment, not the tools. So, even if the tool is right, and we don’t have to make any changes, we really need to make sure that clinicians are reviewing and they’re staying on top of that, and that’s again, I’ve mentioned it. Val is going to talk about it in a little bit, but this is where human-in-the-loop comes in. They can’t just rely on this tool, and we need to be monitoring that, you know, monitoring to make sure they are actually reviewing human-in-the-loop. I know we’ve probably heard it a lot, but it’s not a buzzword, it’s a thing, and it needs to be an integral component of our AI management, especially of these tools and technologies, and so I would say just remember that just because we have these tools that allow us to be more efficient, it doesn’t mean that we can fall back on accuracy, accountability, and clinical judgment. We can’t lose those with use of these tools, so we wanted to give some questions to ask yourself in this situation. Starts with governance, like I mentioned in the beginning of this section. Was it approved? Do we have a use case? Have we defined what we’re using it for? Do we have a business associate agreement? Does our agreement address the AI-specific risks we talked about identifying as part of your diligence processes and your risk assessments? What if it, what if the data going into the tool is being used to train other models? What’s happening with that data? Can we delete it? Do we retain. Oversight and ownership of it. What’s happening to the data once it’s going into these tools? Can we evidence that we’re keeping the human-in-the-loop? Can we show evidence of that answering these types of questions and figuring out a way to say yes and find the defensible answer is how you’re going to be able to evidence that you’re using AI appropriately? Miriam or Valerie, do you have anything on this one? I know you guys have seen this with coding and documentation, so before I move on

Valerie Rock [25:32]: now, I’ll be covering some additional components of this later, so carry on.

25:38 – AI embedded in EHR and clinical workflows

Erin Walker [25:38]: Carrying on, next scenario that I wanted to cover. This is a different type of AI. It’s the AI that’s already embedded in your clinical workflows, and this gets tricky because sometimes organizations I’m seeing, they will assume that it’s not their responsibility, it’s already being handled because it’s in their EHR, so it’s their EHR vendor that has to handle the AI, it’s that third-party that has to handle it, but these tools influence care decisions, and that can raise patient safety and liability questions. So, it’s influencing care decisions within your organization, so therefore there still needs to be some internal management of the tool and what’s happening with it, we’ve noticed alert fatigue, workflow disruption, because of that. Just because when people don’t know how to use it, they’re getting all these alerts, they’re not sure who’s responsible for what, and do I need to be monitoring or somebody else. People are just getting tired of all the alerts, and you know, as we’ve seen technology advance, not just with the use of AI, but in other areas, alerts are a thing, alert fatigue is a thing, and so it’s really important to have that defined governance and be providing appropriate training and education. At the end of the day, regulators are expecting organizations to explain how they’re overseeing these tools, and the answer cannot be it’s our EHR vendor’s responsibility. It cannot be just like with any other outsourced service or outsourced application. You’re, we’re still responsible for that oversight, and we have to be able to show that we’re doing it. So, just like my last example, the question with this specific scenario, what would we need to be able to answer to demonstrate that we actually have oversight of these tools, and I included some questions here for you to think about and talk with your teams on who owns the tool, who is responsible for approving, but also who’s responsible for when the tool is updated and there’s changes made, or what if someone is starting to see it’s learned, the AI has learned wrong, something has gone wrong, it’s doing this wrong. How do we escalate that? Who do we escalate it to? And how do we put pause on it in the interim while we’re working through that? Is the tool, if the tool is advisory, do your teams truly understand that? Have they truly grasped that this tool is giving them suggestions, and it’s still their responsibility to make the ultimate determination, and that comes down to training and education more than anything, but also periodic refreshers and reminders, because people are going to start to trust these tools more and more over time, especially as they see that they’re giving them the right outputs, and so we really can’t lose sight of the importance of making sure that it’s, if it’s advisory, these are suggestions that our teams understand that, and as Miriam mentioned, these tools aren’t static, they’re going to constantly change, and they can even change if there’s not been a vendor update made, because, as I mentioned, the AI can learn wrong. There can be information introduced that it’s not necessarily learning wrong, but it’s not applicable to what the tool was intended to be used for, and so therefore the outputs are skewed. So you have to have a way to monitor the accuracy of these tools and what they’re providing, especially AI bias. They can, these tools can develop their own bias, and once they have, it can take you down a wrong road. Miriam mentioned from an administrative perspective, using tools to look at resumes, we’ve seen tools learn wrong and reject resumes in a technically discriminatory way, and so it’s really important that you’re monitoring there. And I would just say, finally, can you explain how the tool works, what its limitations are, and how it plays a role in ultimate decision making in your organization, and I would just advise that the answer shouldn’t be it is the ultimate decision maker, because you want to make sure that you’re staying on top of everything as you move through with that. I will give it back to Miriam.

30:01 – Integrating AI oversight into compliance infrastructure

Miriam Murray [30:01]: Thank you, Erin. So, how do you integrate the oversight into your compliance structure? So, from a regulatory standpoint, it’s important to understand that AI oversight is an extension of your existing compliance quality and patient safety frameworks, Regulators expect organizations to demonstrate their documented oversight, just like Aaron just said, being able to document your oversight, your decision making, especially where AI is influencing a clinical or an operational process, Where AI impacts care decisions, you definitely want to document how these expectations are being addressed, but you have to make sure you’re applying the same level of quality and patient care that the organization, and honestly, that the government expects. We’re also seeing clear expectations about AI risk. They want you to start embedding it just like you would any other high-risk area in your organization, so doing your enterprise risk assessment. What audits are you doing to address the risk? How are you monitoring the risk? How are you responding to incidents, and those are part of what is expected already, it was in the OIG guidance that they actually already included the fact that you need to address AI. The focus is on how the organization controls, monitors, audits, and responds to the risk, both before at implementation and over time. It’s not going to stop. You’ve got to constantly pay attention to the risk. That’s what the regulators are expecting. They don’t expect you to be AI experts. I will say that they don’t expect you to be AI experts, or at least that’s what they say, but they do expect you to be good at managing the risk and documenting your risk and applying that discipline consistently over AI, so while they don’t expect you to be experts, they do expect you to look at all of these areas, and many of these areas, if you look at them, are part of the elements of a compliance program, so that’s what they’re looking for when they start to look at this AI services. So we’re going to get into the weeds a little bit of each of those categories, so clear accountability. Erin said this already, but understanding who’s accountable, having an inventory, but who’s the owner, not who the vendor is. Who is the operational clinical IT owner of the AI tool that you’re going to use? Is there a defined governance process? Have you documented the risk? Have you documented the approval of the implementation, as well as any changes that come about over time? You’ve got to document the intended use. This is the who, what, where, when, how impact process. Literally, it is. It is understanding. understanding who’s going to use it. How are they going to use it? How is it going to impact the organization? But it also has to be in alignment with the organization’s expectations and objectives. You got to integrate it into policies. If you haven’t already, you really need to look at your policies and start integrating your AI use into your policies. HIPAA privacy and security is one big one. Your compliance program, it’s how it’s going to be used, your user access, and your vendor management. These are just a few of the policy categories that you need to be looking at, but you should have a defined process for vetting and approving all AI tools, ongoing auditing and monitoring. Yes, definitely, you’ve got to have periodic reviews, you need to do risk assessments. Maybe you need to do risk assessments before, during, and after any change, just to make sure that the tool is operating as expected. Looking for errors, looking for bias, looking for drift. You’ve got to revalidate the tools, especially if there are changes and updates. Transparency, and this is transparency across the organization, not only leaders, your leadership, your board need to understand how your, how the organization is using AI tools, that there’s clear guidance on how those tools are being used. Not only the leadership and the board, but your employees need to understand what AI tools are out there. How are they expected to use them? That goes into training, of course. So, you’ve got to make sure that your staff have been trained prior to going full board into the AI tool.

36:16 – Training, issue reporting, and corrective action

Miriam Murray [36:16]: Obviously, sometimes you may have to wait until the tool comes into play, but they need to be educated and trained on how to use the tool. What’s acceptable use? What’s prohibited use? Are there limitations? What data are they allowed to use? But they also need to be aware of the output. What’s expected to come out of the tool? Does it make sense from their professional judgment and knowledge and experience? If it doesn’t make sense, obviously that goes into reporting. Do they have reporting channels? How to report issues, errors, anomalies, things that are not making sense, that should make sense. AI is not to replace your common knowledge, it’s to assist with your common knowledge. So, being aware of what’s expected, but also being aware when it doesn’t work the way it’s supposed to, or as expected, making sure you have an ability to raise those issues, making sure there’s a process in place when those issues are raised to document what the issues are. What did we do about it? How did we review it? Did we do an audit? Did it result in corrective actions? All of that needs to be documented as part of the process of integrating your AI tool into operations.

Valerie Rock [37:55]: Miriam, can we sit on that for just a minute? I think it’s important to think about that reporting channel, we have a reporting channel for through a hotline or a compliance hotline at this point. So, is it best to utilize that compliance hotline for all of these types of reporting, or it should it be a different channel? Because my thought is if you use a different channel, you’re going to get less reporting up of what’s going on, so what are you guys’ thoughts on that?

Miriam Murray [38:25]: You know, as I’m, as I’ve thought about this, and I’ve worked with, you know, our clients on these. Yes, it could be used through the hotline. Sure, it could, and it’s a way to capture the issues. It’s a way for compliance to be made aware of it. Maybe they have a robust AI governance structure that they have communicated out to the employees that you know any kind of AI anomaly or error can be forwarded up through maybe AI governance, at a minimum, it needs to go to the owner of the AI tool to ensure that the owner is aware that there are some things going on, but there’s got to be that collaboration within all of the departments to ensure that when it is reported that there is appropriate action to review what the issue is. How are we going to address it? What corrective action needs to be taken? How are you going to communicate that with the AI vendor as well? There’s that communication channel as well to ensure that there they can address the issue, especially if they’re in control of the tool itself, so that’s some thoughts as well. Aaron,

Valerie Rock [39:47]: yeah, just to add a little bit to that, because I’m thinking from an end user perspective, there’s going to be thousands of tools or thousands of places where AI is embedded, so as an end user, I’m not going to know the owner of every tool, right, so it may just be faster and easier for the end user across the system to be able just to report it up through the hotline and at least get it recorded, and then it can be distributed from there. Additional thoughts, Erin.

Erin Walker [40:16]: My only additional thought, I agree with either the hotline, or if you choose to use, like, if your information technology department has, like, a help desk or a ticketing system that you could utilize, that would be another avenue, but also having a way to escalate, like, I know you know we have a way to immediately escalate, like a phone number to call to, you know, if something bad were to happen that we don’t want to wait on, you know, someone to get an email and read or check the next morning. Having trained on that, I’m not sure that you know how much that would come into play, but having an immediate escalation as well. But I think the hotline is a good idea, as is maybe a help desk or a ticketing system like that.

Miriam Murray [40:57]: Yeah, I think it depends on the size of the organization as well. I mean, your smaller organizations may not be as robust as a full-blown health system, so I think it can be scalable, but I like both of these ideas of either the hotline or the IT help desk, or some sort of a governance mechanism. They may even set up a separate governance email reporting. Who knows how they want to

Valerie Rock [41:24]: Yeah.

Miriam Murray [41:26]: It could, but AI is here to stay.

41:32 – Human-in-the-loop oversight and frontline responsibility

Valerie Rock [41:32]: so I think when we think about the human-in-the-loop aspect, it’s a little overwhelming to hear all these components that we need to address, and if you’re the compliance officer, you’re in the compliance department, you’re not going to be able to handle all of this on your own, right, but we’re in a phase of implementation where oversight of AI is really important, because AI is going to make mistakes, and we are going to make mistakes as we’re utilizing AI, so we need to make sure that we have all of that oversight in place, but we do anticipate over time. I think Erin mentioned this as well, that over time there’s going to be areas of AI that we can feel more comfortable with. If you think about the way that we utilize our smartphone, our computers, our, you know the way that we use technology today, we are more accepting of it, even though we do know how to look at it from a critical eye, like we’re looking at even now, we’re looking at our news, is this an AI-generated news, you know, video or anything, you know, like we need to be cautious about that, but we’re becoming more critical in our, like, each individual is becoming more aware, but we will utilize this differently over time. So, while we have something implemented now, we need to be able to be nimble and evolve our processes and our policies and the way we implement this over time, so human-in-the-loop, human interaction can be inserted at the input stage, so the data that’s input into a system, and then the output, so you may not be involved in the input of the system, which is where the vendor question comes into play, is where is the data coming from, you know, like clinical decision making, where is that data coming from? What is supporting the data that is going in and ingesting into the system to produce an output, but you can see generally the output, and is that output accurate? So, when you come into human-in-the-loop aspect, it really is training that frontline team to monitor that output. The physician that’s engaging with the AI scribe needs to review the output and confirm that it is accurate before he or she signs the note. So, while there’s a number of ways that AI can be embedded through the functionality of a tool, we’re going to have too many of these tools that are utilizing AI to monitor them all from one aspect. While the compliance team can review this from the highest risk area, you’re not going to be able to review everything that’s going on across the enterprise. The IT team cannot do that either, so it’s not like you say, “Well, this is a technology tool, so we’re going to have the IT team monitor this, or “This is a compliance question, so we’re going to have the compliance team monitor this. It needs to be everyone, so I think if we can train the teams on the front line well to identify where the problems are, that’s going to help mitigate our risk. Responsibility should be borne by the user, while we can’t have that ultimately be the case, and our contract should have some culpability on the tool itself. We need to make sure that our frontline teams understand that they are responsible in how they use information, and that they’re reviewing it, and if they don’t understand it, because they’re not the subject matter expert on that information, that they need to have someone else involved in reviewing that for quality as well, that could even go to the point of even just basic research within your own compliance department to say we’re researching this rule in copilot, for example, and through that it results in a different answer than what is accurate. And how are you vetting that? And how are you making sure that is true? And looking at primary source information would be one of the ways to do that. So, what are the common gaps in AI that we see? How do you identify those potential gaps in oversight? And so here are some examples of that.

45:50 – Shadow AI, automation risk, and ongoing monitoring

Valerie Rock [45:50]: If tools are adopted at the individual provider level, so say a physician is looking to adopt an AI scribe and not even telling the upline that he or she is doing that, and downloads that on their phone, and doesn’t follow the governance process that needs to be used to vet that, and has not implemented a BAA with that tool to make sure that they’re covering the PHI exposure concerns. Then you already have exposure, so your frontline has to know the rules, they need to know that a BAA needs to be in place, they need to know those things that you would have otherwise probably not worried about in the past, but we’ve already seen things, people, you know, physicians texting patients, and, all of those things have already occurred, we’ve already seen where, we’ve already seen where physicians are not reviewing their notes, so this is not anything new. We need to make sure that we’re doubling down to make sure that is occurring, so if no one knows the process to vet the AI, the employees will hear of other colleagues using AI, and they’ll say, “Oh, well, this must be appropriate, so I’m going to start using it too, and they’ll start using it, and they won’t know that it was not appropriate. So, how do you monitor that? And I think that is where data analytics is going to come into play, both from a, you know, your IT team monitoring the use, usage with in computers and devices and things like that. Is there ownership over what’s going on with a phone? Can they monitor what’s being applied, you know, applications that are being used and how it’s being used. What is the data within the billing? Are we spiking somewhere that’s different than what has occurred before? Because what is going to happen is anything that occurs wrong when you automate it is going to happen over and over again to the point that now you’ve extrapolated a problem that once was one problem, now is a thousand problems, so we want to make sure that we’re applying AI and then monitoring that, but also something that is something to think through is as we’re looking to apply AI, we want to make sure our processes are good from the front end, because we don’t want to take a bad process and apply AI to it, right? So we don’t want to increase the volume of a bad process. So making sure that we’ve got good compliance protocols in place, policies and procedures around those things from the front end, and then we’re multiplying that at that point, and then we’re monitoring on the back end will make us more efficient, but we don’t want to extrapolate problems quickly. Coding denial management, summarizing of medical dictation via AI scribe, or as I mentioned, just simple compliance research could result in those significant errors. So we want to be monitoring all the way throughout our system, but again, there’s no way for one department to review all of this, so how do we put the onus at the front line at each level of user, so that we’re well educating them, and that’s another aspect that I think you know, as we start delegating some of the actual data aggregation for a clinical decision making tool, so the clinical decision making tool is making some conclusions on what might be an option for a physician, that physician is then applying that, and so, as they become more, and more reliant on that, How do we make sure that they are both responsible, but also increasing their EQ? For example, we’re hearing recommendations for educating them on empathy and making sure that they’re not just, you know, reading the output.

49:59 – Patient engagement, bias, and frontline review

Valerie Rock [49:59]: But communicating that, and moving on, it is truly that we want to increase our engagement with the patient, making sure that we’re, you know, really, you know, as Miriam and Erin both mentioned, that we don’t want to have them looking at the back of the head of the physician while they’re, you know, typing out their notes and things like that into the system, so now that they’re engaging the patient, are they doing a good job at engaging that patient, or are they making the patient feel comfortable in communicating what’s going on? You know, how are we communicating what’s going on from an AI scribe perspective? Are we communicating some more confidential information outside of the room versus inside of the room. How do we make sure we capture all of that? So we need to be cognizant of how we’re managing all of that information and gathering all of that information. So we talked about bias, but you know, having that frontline monitoring for that bias is key, so not just that there could be bias, so we can look at, you know, the front end of how that data was pulled, you know, what population was used to create the outcomes, and but then having the providers looking and saying, well, this is not actually coming out with the right outcome. Why is it not? Is it because of the data that was generate, you know, used to generate the tool, or is it that there’s something else going on? So, making sure that they’re monitoring for that is also key. So, what is best practice? How can we make sure that we’re doing our best, and as we mentioned from the top, we want to make sure we’re doing our best. We know that there’s going to be problems that arise, but have a process, so all of the things we’re talking about from a monitoring perspective, we want to make sure we include a process to oversee that, whether it’s a survey that you’re saying, okay, how are you utilizing AI? Where are you utilizing AI, and or part of a like a systematized part of each of the departments at each meeting that they have, that is a standing agenda item talking about use cases for AI, and that’s reported up. You need something to be able to gather where people are using AI, so that you can record it, and then that you actually do record that and monitor for those use cases. Then define and train on how the tool can be used. We mentioned that based on written policies that can be communicated and then easily accessed. It’s one thing to put it in the policy, it’s another thing to have it easily accessible by the teams that are going to use them. We’re even contemplating how to utilize AI to easily identify where a policy is. So, how accessible are your policies? How can, how searchable are your policies? Can a kind of end user quickly get to those policies? Utilizing AI may be the way to go to help them get to those policies, and then to actually give them recommendations. You want to monitor that as well, but that is like the future state of our policy creation, expected review processes are implemented at the front line, and with it within the appropriate functions. We talked about, you know, making sure that those processes are implemented, but ensuring that each SME, you know, that subject matter expert are the ones that are engaging in that review. It’s not that it is the one that’s looking at all of this, it’s that it is looking at their component, security is looking at their component, risk is looking at their component, the legal team is looking at their component, the compliance team is looking at their component, and don’t forget that a lot of this is clinical decision making or has a clinical component, and that you need to have the physicians and nurses and other frontline people involved in the decision making, so it really is a multifunctional team that you want to have engaged in this process, so align the skill set and the expertise with the oversight, making sure that you’re not missing that. It is one thing for a compliance professional to look and see what all is going on.

54:22 – AI vendor due diligence and contract protections

Valerie Rock [54:22]: It’s another thing to engage the teams to help you in investigating that and have that as a part of your protocol and part of your governance process. So now let’s move into how we actually do diligence on your AI vendors, and really, what you can apply is all of the risk areas that we’ve talked about today. You want to vet those tools for those risk areas, but on the back end, you’re going to want to make sure that you’re contracting to ensure you’re mitigating risk going forward, output on a generative AI side is not just automation, it really is, you know, something that is a decision, like they’re coming to conclusions or summarization of the information, so you want to make sure that output is reviewed at a different level than just a plus b equals c, as I mentioned, that output can be medical decision making, so you want to make sure that you’re applying that clinical component, or those clinical people, into that review of that tool. Vetting and overseeing tools includes that multidisciplinary team to address the risks across all functionalities, so as AI is evolving, the ability to write code to support systems is making the changes to those systems faster, so the tool that you buy today will be different from three, six, and 12 months out, so you cannot just assume that you’re going to adopt a tool and then implement it and then let the teams use it and monitor it. It really is that you want to monitor it frequently enough that you know what’s going on, but How do you do that? How do you make sure that you are mitigating extra, you know, additional changes, so ensuring that your contracts include an expectation of the vendor to communicate any material changes to their system is really important to make sure you’re keeping up with that. Also, you want to have out clauses, because there’s all kinds of reasons for why you would not be able to be under this current contract anymore. The AI vendors of today will, some of them will fizzle out because they won’t be able to get the market share that they need, they will go bankrupt and things like that. So you need to be able to have an out clause to get out of the contract at that phase. You also may start implementing an AI tool that you find then is actually not working the way it said it would, so you need to be able to get out of the tool at that point, and then some of your AI tools will be bought or consolidated with other tools, and that contract will change, and they may actually increase your rates at that point, so how do you mitigate both the changes that you know from an investment perspective, from a change you know an acquisition or a change in fees, and so that you can make sure that you’re not then obligated to, for you know, at an increased rate, so here’s a the next two slides are a bit of a vendor due diligence checklist that we want to provide for you, but we don’t want to miss the intended use, you know, like when you’re thinking about an AI tool, you want to look at why you’re using it again, as I mentioned before, you don’t want to automate something that is not a good process in the first place, or automate a process that is actually not going to make it any faster, so we want to make sure that we’re looking for those aspects and make sure that we’re then addressing that on a routine basis, because the way that a tool could be used in the future may not be viable because of another tool that kind of takes over the need for that tool, so we want to make sure that we’re watching that intended use. The data use is really key, because if the data is going to be used for something other than your use, if you are not owning that data, ultimately, you need to know that. Do you maintain ownership of that data, whether it’s in raw form or in de-identified form? And once it’s de-identified, is someone going to sell your data and utilize that for, you know, their own benefit?

58:57 – Data ownership, risk tolerance, and governance frameworks

Valerie Rock [58:57]: Are you going to allow that? A lot of these questions are going to be based on your risk tolerance, based on your state requirements, based on as we apply more federal guidance, even though we don’t have as much federal guidance right now, as we mentioned, there are a lot of rules that are kind of giving us the guardrails, the HIPAA requirements, etc. that are giving us the guardrails right now, so making sure we’re applying those guardrails, but also considering our, you know, what risk we want to take with our data, leveraging NIST and ISO methods for actually applying these tools, and monitoring these tools, and diligencing these tools are our best practice, so if you follow those rules and follow those kind of guidance and processes, then that gives you a way to say, okay, we’re doing everything we can do, so lastly we want to make sure that we are monitoring it all. Also, for customization, where we can customize a tool, we can partner with a tool, but over time, as these tools scale, they’re less likely to be customizing with you, because they’re going to have to do this with 1000 people instead of the three that they started with, right? So you may have a lot of customization up front, and then over time, you’re going to reduce that. You want to make sure that, though, you still have the ability to give feedback to say, “Hey, this is a problem, we need to revise it, or, as I mentioned, contract to ensure that you can get out of the contract if they’re not applying correct rules, you know, per your risk stratification, so as we close, really at the end of the day, we want to make sure that we’re utilizing AI, that we’re functional with it, that we are really trying our best to leverage it while creating that infrastructure around it, allowing our teams to use it and be educated on what the right way to utilize it, and then the ability to quickly report when they’re having a problem, so that you are not the only one looking at this, and that your IT teams, your compliance teams, your legal teams are not the only ones that are educated on this, but all of your teams are educated on how to best utilize the tools and how to best report up when there’s a problem. Make them feel responsible and make them feel like part of the team that will be able to allow us to use, and that you want them to be able to be more efficient and go to bed and reduce their pajama time, and all of those things that the physicians and others are asking for. There are ways for us to implement this in an effective way, and we hope that you enjoyed this conversation today. And we welcome any questions from you after this. We’re happy to answer them. So, if you’ll put those out in the chat, we’ll be happy to answer any questions you have afterwards. Thank you for your time today, Jen.

1:02:10 – Closing remarks

PYA Moderator [1:02:10]: Thanks to our presenters, Valerie, Miriam, and Erin. Also, the slides and recordings for every episode of PYA’s Healthcare Regulatory Roundup series are available in the thought leadership section of PYA’s website, pyapc.com While at our website, you may register for other PYA webinars and learn more about the full range of services offered by PYA. On behalf of PYA, thank you for joining us. Have a great rest of your day. 1:02:40 Thank