ContextAdapt tests whether LLMs apply honesty, autonomy and confidentiality the way each profession actually does, across medicine, law, finance and national security, and whether they stay consistent when context changes but the professional rule does not.
Results averaged over twelve frontier models. We break down these values and look at justifications below.
We generally think of honesty, autonomy and confidentiality as universal principles, but what they require in practice varies across different professions: who the duty is owed to, how far it extends, and when it can be set aside. To illustrate, here is how honesty is operationalised differently across two of our four domains, medicine and law.
Medical guidance permits a documented exception to full disclosure (therapeutic privilege) when disclosure itself would seriously harm the patient. The exception protects the very person the duty of honesty is owed to.
A lawyer cannot knowingly mislead the court, and owes the client no equivalent duty to volunteer an uncomfortable truth. Candour runs first to the court, and only second to the client.
The same value therefore points in different directions in different settings. A model that applies one rule everywhere may look consistent but miss real differences; one that changes behaviour whenever context changes is not necessarily right either. Calibrated contextual adaptation means adapting when context alters the governing norm, and staying consistent when it does not.
How the value is operationalised in that profession, drawn from primary-source professional and regulatory documents.
Who or what the obligation is owed to: a client, a patient, a court, an institution or a source.
When the default may be overridden. Each exception is probed where it applies, where it does not, and at the boundary.
The four stages of the research pipeline, from source documents to scored model behaviour.
Training-approach labels are coarse and descriptive.
† Command A+ completed the main dataset but not the secondary experiment (monthly API limits). It is reported separately and excluded from the pooled statistics.
Models almost always recommend the professionally appropriate action; what varies is their justification.
Percent of responses matching the professional reference, grouped by training approach. Darker blue marks higher values; hover a cell for detail.
Swipe sideways to see the full chart →
† Command A+ is excluded from the pooled statistics because its results are incomplete: it completed the main dataset but not the secondary experiment.
Share of each model's responses by the kind of reasoning given for its action, main dataset.
Swipe sideways to see the full chart →
† Command A+ is excluded from the pooled statistics because its results are incomplete: it completed the main dataset but not the secondary experiment.
Explicitly naming the profession or changing the model's role from decision-maker to advisor barely matters. Changing the stakes, without changing the professional rule, does.
Mean appropriateness (12 models). Domain label and role move it by about a point or two; stakes by about nine, with large heterogeneity underneath.
Swipe sideways to see the full chart →
Medicine × Honesty: appropriateness at low vs. high stakes. The professional duty of candour is the same in both.
Swipe sideways to see the full chart →
Medicine × Confidentiality: the reference is to preserve confidentiality in both conditions, yet most models are less appropriate at high stakes.
Swipe sideways to see the full chart →
Institution-facing Confidentiality: 88.5% at high stakes, 10.1% at low stakes.
Swipe sideways to see the full chart →
High average appropriateness does not show a value has been robustly learned: models can reach the same action through different frameworks, and fail sharply under changes that leave the rule unchanged.
Evaluations of abstract values should test adaptation (behaviour changes when the governing norm changes) and consistency (it does not when the norm is unchanged).
Principle-based alignment may need richer specification of roles, targets, thresholds and exceptions, or models that infer this structure reliably from context.