Did a bot deny your claim? NC lawmakers want to regulate AI in health insurance

North Carolina Republican lawmakers are pushing a proposal that would limit how artificial intelligence can be used in healthcare, aiming to protect consumers against technology that could lead to bigger bills or denied claims. 

“We do not want robots or computers to deny care to sick people,” state Sen. Amy Galey, R-Alamance, said Wednesday during a committee meeting Wednesday.

House Bill 565 would prohibit insurers from using AI as the sole basis for denying healthcare claims or prior authorization requests, saying that humans must remain involved in those decisions.

The legislation also seeks to crack down on hospitals and other healthcare providers that might use AI to inflate their profits through a strategy called “upcoding.” The treatment patients receive, and which they or their insurers are billed for, are cataloged using a highly specific set of thousands of codes.  Higher billing codes can lead to higher reimbursements from insurers, Medicaid or the state health plan, increasing overall healthcare costs and potentially leading to people paying higher premiums as well.

When done intentionally, upcoding can constitute fraud. But when AI is making the decisions instead of humans, healthcare policy experts say, the AI could unintentionally push billing systems toward more expensive codes if there are not clear safeguards and human oversight.

“We know that AI can amplify what already exists,” said Alessandra Bazzano, a professor at UNC Gillings School of Public Health. “If there’s a reimbursement algorithm that rewards severity or intensity, those AI tools might accelerate pressure toward doing that kind of higher coding.”

Republican lawmakers have previously said North Carolina has among the highest healthcare costs in the nation; this is one of several policies they’ve proposed in an effort to change that. Some of those proposals, such as repealing the state’s certificate-of-need laws or requiring more transparency around medical billing, have failed to pass in the face of intense lobbying by healthcare companies and industry groups.

Under the new legislation proposed in House Bill 565, providers and vendors wouldn’t be allowed to use AI to recommend or generate billing codes that result in upcoding unless a treating provider reviews the documentation and confirms it is supported by the patient’s medical record.

The National Association of Insurance Commissioners has found that most health insurers already use AI in some form, including for claim authorization and fraud detection. So in situations where doctors and insurance companies clash over whether a patient receives care — and, if so, how much it will cost — it’s becoming increasingly possible that the decisions on both sides are being made by in part by artificial intelligence.

The proposal reflects a broader national push to ensure humans remain involved in insurance and billing decisions as AI becomes more common throughout the healthcare system.

Academic researchers generally recommend that insurers require human review of AI-assisted claim decisions, maintain transparency about how algorithms influence outcomes and preserve audit trails showing what recommendations AI systems made and who ultimately approved decisions.

Kandyce Brennan, co-chair of the AI task force at the UNC School of Nursing, said transparency is essential as AI expands into healthcare administration.

“Without that transparency,” Brennan said, “it’s hard to know whether AI is supporting better decision-making or simply automating denials, increasing costs, or reinforcing existing inequities.”

America’s Health Insurance Plans, a trade group representing insurers, has supported a national approach to AI oversight and says new legislation shouldn’t duplicate existing laws. In comments submitted to policymakers, the group has said AI tools should augment — not entirely replace — human work.

As the legislation moves forward in North Carolina’s Republican-controlled legislature, lawmakers say the legislation is an attempt to balance rapid advances in artificial intelligence with safeguards meant to ensure medical decisions — especially those involving billing and insurance coverage — are not left entirely to algorithms.

A 2019 study published in the journal Science found that an algorithm used by major insurers and hospitals found that a widely used healthcare algorithm treated Black patients as healthier than white patients even when they were equally sick.

The problem came from how the algorithm worked: instead of predicting who was actually the sickest, it predicted which patients would generate the highest future healthcare costs.

Because the healthcare system historically spends less money on Black patients — even when they have the same medical needs as White patients — the algorithm learned to associate lower spending with lower need for care.

As a result, Black patients were less likely to be flagged for extra medical support, despite often having more chronic illnesses than White patients with the same risk score.

Researchers found that when the algorithm was changed to measure actual health conditions instead of medical spending, the racial bias largely disappeared and far more Black patients qualified for additional care.

The legislation comes as AI use in healthcare rapidly expands nationwide. According to a DemandSage survey, about two-thirds of physicians use AI tools, up from about 38% in 2023.

On Wednesday, Galey pointed to a Rice University report showing hospital service prices have more than tripled since 2000.

“There is evidence that AI in hospital billing is driving the appearance of more serious diagnoses without corresponding changes in treatment,” Galey said.

The North Carolina Healthcare Association says that while artificial intelligence can help patients and clinicians, it worries about provisions of the legislation.

“We are concerned the bill could limit healthcare providers’ ability to use this technology in ways that support patient care and workforce needs,” the association, which represents hospitals in the state, said in a statement. 

Some lawmakers raised concerns about how the bill would work in practice.

Sen. Gale Adcock, a nurse practitioner, said upcoding is already prohibited under existing law and that the proposal still needs revisions.

“I think this bill needs a great deal of work on it to achieve the balance we’re looking for, and also the ability to enforce it,” said Adcock, D-Wake.

Sen. Natalie Murdock, D-Durham, said lawmakers should ensure hospitals, insurers and providers are involved in shaping the legislation. “I just want to make sure we get it right,” Murdock said.

The debate comes as North Carolina is simultaneously investing heavily in artificial intelligence research tied to medicine and healthcare.

NCInnovation, a public-private partnership focused on commercializing university research, has committed more than $19 million toward projects involving AI and healthcare innovation.

Projects include AI-assisted maternal ultrasounds at UNC-Chapel Hill, virtual reality nurse training at Winston-Salem State University and AI-powered livestock parasite detection at Appalachian State University.