An ECG AI model can perform well on data resembling its training population and less well on data from a different cohort. A large development dataset does not, by itself, settle whether the model will work equally well in another clinical setting.
A September study of an electrocardiogram foundation model helps explain why. Its findings concern a particular contrastive-learning approach, rather than a verdict on every ECG algorithm or a claim that diverse data are inherently harmful.
What the model learns before the prediction task
An electrocardiogram records the heart’s electrical activity. A foundation model first learns representations from a large collection of recordings, which can then be used in more specific prediction tasks. The pretraining dataset and the later task-specific evaluation are separate parts of that process.
The PLOS Digital Health study of CAPE, published on 2 September 2026, examined how the composition of pretraining cohorts influenced those later results. The researchers used more than five million ECG recordings across four pretraining cohorts, then examined prediction tasks including additional cohorts.
Those are recordings, not a count of five million different patients. The distinction matters whenever repeated measurements from the same person contribute to a large medical dataset.
A useful signal can carry information about its source
In this study, combining cohorts improved performance within the represented distributions but could reduce generalisation to an unseen cohort. The researchers suggest the learned representations captured features specific to the contributing cohorts, alongside clinically relevant information.
A recording can reflect more than the disease or outcome being investigated. Equipment, healthcare setting and the population being sampled can differ together. A model may use patterns associated with those differences without the evaluation revealing exactly which component produced the result.
This is one reason a random split within a familiar dataset and evaluation in a separate population answer different questions. The first can measure performance on held-out examples from similar circumstances; the second can expose a change in the relationship between inputs and outcomes.
The study does not isolate every cause
The CAPE paper’s limitations are important. Cohorts differed in several connected ways, including health status, demographics, healthcare systems and recording devices. The work did not independently manipulate each factor to establish its individual contribution.
The proposed batching approach improved generalisation in the reported experiments, but it is not a demonstrated solution for every learning method or clinical deployment. Some underlying datasets were restricted, and the trained model was not made available with the publication.
What an evaluation needs to make clear
The TRIPOD+AI reporting checklist promotes transparent reporting of prediction-model development, evaluation and updating. Applying that general reporting purpose to an ECG study means looking beyond a large sample count to the details needed to understand the result.
Useful questions include:
- Where did the development and evaluation recordings come from, and could the same patients contribute more than once?
- What exactly was predicted, using which reference outcome and follow-up period?
- Was performance assessed in genuinely separate settings or populations, rather than only a new split of familiar data?
- Which measures were reported, and what uncertainty or subgroup gaps remain?
These are evidence-reading questions, not a certification checklist or a claim that one metric can establish clinical usefulness. Transparent reporting makes scrutiny possible; it does not supply missing validation.
The issue also differs from whether people can use a health chatbot effectively. Here the question is how a model’s learned relationship transfers between data sources. A strong benchmark can be informative while leaving the evidence for a particular patient population unresolved.