Evaluating AI in Regulatory Technology
As AI spreads through pharma AI regulatory technology, evaluating a vendor gets harder. A traditional assessment looks at features, pricing and references. AI-powered platforms need additional scrutiny around compliance, data integrity and operational safety.
This guide gives twelve questions regulatory evaluation teams should ask. For each one we explain why it matters, what a good answer sounds like, and — in the interest of transparency — how DnXT answers it, including where our own answer falls short.
Audit Trail and Data Integrity
1. “Is your audit trail built in, or added on?”
Why it matters: when AI carries out hundreds of actions in a session, gaps become statistically likely if recording each one is something a developer has to remember. An audit trail created automatically as information is written closes that gap.
Good answer: “The part of our platform that writes information creates the audit record itself. Nobody can bypass it.”
Red flag: “Our developers follow best practice for audit logging.” Translation: it is manual, and sometimes forgotten.
DnXT’s answer: our platform creates a compliance-grade audit record automatically every time anything is created, changed or removed. Nobody can bypass it. Records follow the ALCOA+ framework and carry cryptographic protection against tampering. Honest gap: we do not yet record which version of which AI model influenced a given action.
2. “How do you tell AI actions from human ones in the audit trail?”
Why it matters: regulators will increasingly ask whether a specific action was taken by a person or by software. Your system needs to answer that directly, not through analysis afterwards.
Good answer: “Every record identifies where the action came from — a person on screen, a connected system, or AI.”
Red flag: “Everything is logged the same way.” Translation: you cannot tell them apart.
DnXT’s answer: every audit record identifies whether the action came from a person, a connected system, AI, or the platform’s own internal processes. An auditor can filter on that to isolate exactly what AI did. Honest gap: AI attribution is part of our design and is not yet live for customers.
Security and Access Control
3. “Can AI reach another customer’s information?”
Why it matters: in a shared platform, software has none of the awareness a person has. It will not notice that a result contains somebody else’s information. Separation has to be built into the foundations.
Good answer: “Which customer it belongs to is determined by us, from the credential. There is nowhere for the AI to specify a different one. And even if that failed, each customer’s information is held in its own database.”
Red flag: “We filter every request by customer.” Translation: it depends on somebody remembering, every time.
DnXT’s answer: we enforce separation at four independent levels: establishing who is asking, carrying that through the request, holding each customer’s information in its own database, and checking permissions. If any level cannot determine the right answer, the request is refused outright — there is no quiet fallback and no best guess.
4. “Can AI sign documents or bypass approvals?”
Why it matters: 21 CFR Part 11 requires an electronic signature to be attributable to a specific individual. AI is not an individual. The system must enforce that, not merely state it as policy.
Good answer: “There is no way for AI to apply a signature. The capability does not exist.”
Red flag: “Our AI respects signing workflows.” Translation: it is an instruction, not a limit.
DnXT’s answer: there is no capability that allows AI to apply an electronic signature. It simply does not exist. AI can request that a person signs, creating something for them to action, but signing always requires a person to authenticate and intend it. Our workflow explicitly marks signature steps as requiring a person.
AI Behaviour and Safety
5. “What happens when the AI is wrong?”
Why it matters: AI will make mistakes. The question is whether there is a preview step before a recommendation becomes an action.
Good answer: “Recommendations go through a preview where a person sees what will happen before confirming, and the system records both the recommendation and the decision.”
Red flag: “Our AI is highly accurate.” That does not answer the question.
DnXT’s answer: our workflow can evaluate what would happen without doing it. AI receives the warnings and blocks in advance, so it can explain the situation to a person rather than simply failing. Human review is required at classification, validation and sign-off.
6. “Am I locked to one AI provider?”
Why it matters: AI is evolving quickly. Being tied to one provider limits your ability to adopt improvements or negotiate.
Good answer: “We use an open standard that works with any provider. You choose the AI; we provide the compliance.”
Red flag: “Our AI features require our own AI service.”
DnXT’s answer: we use an open, provider-neutral standard. Our internal tools were built with one assistant, but the customer-facing design works with any — Claude, GPT, Gemini, or something built in-house. Honest gap: we have only tested extensively with one.
Compliance and Validation
7. “Which regulatory standards does your AI integration meet?”
Why it matters: vague compliance claims are common. Specific references can be checked.
Good answer: “Our audit trail meets 21 CFR Part 11, our electronic signatures address Sections 11.50, 11.70, 11.100 and 11.200 specifically, and we map to EU Annex 11.”
Red flag: “We comply with all major regulations.” Too vague to verify.
DnXT’s answer: our audit trail meets 21 CFR Part 11 and ALCOA+. Our electronic signatures address Sections 11.50, 11.70, 11.100 and 11.200 specifically. Our qualification approach produces IQ/OQ/PQ evidence with full traceability. Honest gap: no regulatory authority has reviewed or endorsed our AI-specific approach.
8. “Can I see a validation report for the AI features?”
Why it matters: marketing shows intent. Validation reports show evidence.
Good answer: “Here is our current qualification report, with test results, traceability to requirements, and risk assessments for the AI features.”
Red flag: “We can provide that after signing.” The documentation may not exist.
DnXT’s answer: every test we run is linked to the requirement and the regulation it exists to prove. Test records carry the same tamper-evident protection as everything else, and our compliance dashboard shows live qualification status at any moment. Honest gap: validation documentation specifically for AI features is still being formalised as those features mature.
Operations and Support
9. “How and where is it hosted?”
Why it matters: where data lives, how isolated it is, and what your own regulators require all vary by region and organisation.
Good answer: “Cloud hosting with network isolation, a choice of region for where data resides, and dedicated infrastructure available for sensitive work.”
DnXT’s answer: we host on Azure, in both standard and network-isolated environments, currently with data residency in the US East region. Honest gap: we do not yet offer on-premise hosting or EU data residency.
10. “How do you handle AI model updates?”
Why it matters: AI models change. An update could alter accuracy or behaviour. Change control has to extend to AI.
Good answer: “Model updates go through change control. We test new outputs against known baselines before anything reaches production.”
Red flag: “Models update automatically so you always get the latest improvements.” That is no change control at all.
DnXT’s answer: Honest gap: this is an area we are still developing. Model updates from providers are not currently gated through our change control, and we recognise that needs to be addressed.
11. “What is your process when AI gets something wrong?”
Why it matters: when a document is classified incorrectly and ends up in a submission, what happens next?
Good answer: “A documented process covering root cause, identifying everything affected, notifying the customer and correcting it — with every step recorded in the audit trail.”
DnXT’s answer: our continuous monitoring detects problems automatically, we track them with resolution timelines, and the audit trail captures the full sequence of events. Honest gap: our process is more mature for infrastructure problems than for AI-specific errors.
12. “Can I try it with a real submission first?”
Why it matters: a demonstration with sample data is useful. A pilot with your actual documents is evidence.
Good answer: “A 30–60 day pilot using your real documents in a separate environment, fully supported, with no commitment.”
DnXT’s answer: we support pilots in a separate environment. Your information stays entirely isolated in its own database. We will help configure the workflows for your submission types. No gap here — this is something we do well.
The best evaluation questions are the ones that make vendors uncomfortable. If a vendor answers every question perfectly without acknowledging a single limitation, be sceptical. Honest vendors — including us — have gaps. The question is whether they know what theirs are and have a plan.
Using This Guide
These twelve compliance questions work whichever vendor you are evaluating — including us. The goal is to help teams decide on verifiable answers rather than marketing claims.
If this guide helps you choose a competitor because they genuinely answered better, that is a good outcome. The industry benefits when buyers ask harder questions and vendors answer with substance.
This article was written by the DnXT Solutions team. We’ve included our own honest answers — including our gaps — because we believe transparency builds more trust than perfect marketing. Questions or feedback: se******@***********ns.com.
