Skipping BCI Calibration: How Dataset Shift Impacts Model Accuracy Over Time

• Motor Imagery (WBCIC-SHU): Evaluated across 51 participants in a two-class task using session 1 for training and session 3 for testing. A frozen foundation model, CBraMod, paired with a simple ridge classifier reached 67.7% balanced accuracy on session 3. This surpassed a spectral band power ridge model, which achieved 53.9% (a paired difference of +13.8 percentage points). In an 11-person three-class cohort, CBraMod reached 51.6% compared to 37.6% for relative spectral power. However, because feature space sizes differed and pretraining data overlap was unverified, pretraining alone cannot be definitively credited for the performance gap without random-weight controls.
• Target Detection (Longitudinal RSVP): Evaluated across 15 participants over four visits spanning up to nominal Day 200. An L2-regularized logistic regression decoder trained on Day 1 maintained AUROC above chance across all visits, shifting slightly from 0.888 at Day 7 to 0.853 at Day 200 (a change of −0.035). However, average precision degraded sharply from 0.359 to 0.240, highlighting that target ranking deteriorated under class imbalance (targets accounted for ~2.5% of events). Probability calibration also degraded, with Expected Calibration Error rising from 0.023 at Day 7 to 0.039 at Day 200.
• Continuous Cursor Tracking (Forenzo & He): Tested on 23 records across two cohorts using a fixed spectral ridge model trained on the earliest session. In the latest session, the ridge model yielded a higher overall normalized root mean square error (RMSE) than a constant prediction baseline (the mean response of the training session) across all 23 records. For several records, severe feature drift or sensitivity led to extreme mean errors ranging from 45 to 570 times their median error.

Addressing temporal dataset shift

The findings illustrate that skipping recalibration completely carries significant risks, particularly in continuous control tasks. When evaluating adaptive strategies, adding even a tiny calibration budget can alter outcomes. On the OpenBMI motor imagery dataset, introducing the first 40 labeled trials from session 2 raised balanced accuracy for a log-covariance decoder with shrinkage LDA from 67.9% to 72.4% across 51 people (+4.6 percentage points). However, for 13 of those 51 individuals, performance actually decreased with the additional trials.

Understanding how machine learning architectures respond to non-stationary data remains essential across AI subfields. As developers build and test diverse machine learning pipelines, developers can try top AI models cheaply through one API at https://apixoai.online to evaluate different foundation model capabilities within their software stacks.

What it means for developers

For engineers and data scientists building BCI systems or working with time-series machine learning under dataset shift, the results offer clear implementation guidelines:

• Evaluate by complete session splits: Never split datasets by randomly shuffling trials across time. Hold out entire subsequent sessions and fit all transformers, scalers, and probability calibrations strictly on earlier data.
• Maintain floor baselines: Always benchmark complex models against simple naive controls, such as a majority-class prior for classification or a training-session mean response for regression. If a model cannot beat a constant mean baseline, it requires fundamental revisions.
• Report median performance alongside tail distribution: Relying solely on mean error metrics can obscure extreme failure modes. Reporting both median error and tail distributions reveals whether degradation is gradual or driven by catastrophic outliers.
• Track metrics suited for imbalanced data: Metrics like AUROC can mask performance loss in imbalanced tasks. Always report average precision and recalibrate probability scores across held-out testing intervals.
• Include random-weight controls: When evaluating pretrained foundation encoders against feature-engineering baselines, run matching control models with initialized random weights to isolate whether architectural capacity or pretraining exposure drove observed accuracy gains.

Source: Train Once, Reuse Later: Three Tests of Skipping BCI Calibration — Towards AI. Written by the Apixo team from that report.