betaFeedback

WorksComputer Science

FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics

Abstract

Visible-light videos and multivariate diagnostic signals provide complementary observations of tokamak discharges. However, existing fusion pretraining methods primarily learn representations by reconstructing or predicting observations, leaving the correspondence between these modalities underexplored. Representation learning is further challenged by spatially localized plasma regions in video frames and substantial differences in diagnostic sampling rates. We propose FusionCLIP, a multimodal contrastive pretraining framework. A selective video encoder retains informative visual tokens, while a query-based time-series encoder integrates asynchronous diagnostic signals using their observation timestamps. Contrastive learning aligns representations of the two modalities. We construct the EAST Multimodal Diagnostics (EAST-MMD) dataset, comprising 1307 EAST discharges, and evaluate the model through independent fine-tuning on three tasks: disruption prediction, edge-localized mode (ELM) classification, and H98 regression. FusionCLIP achieves a Joint score of 89.07%, with absolute gains of 2.41% and 2.90% over the best competing baseline, and the in-domain contrastive pretraining baseline CLIP-Base, respectively. Ablations and attention visualizations further support the contributions of its key components.

Paper details

vlnc:W_uwx37e9m
Version
v2 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
arXiv nonexclusive license
Usage

views · 12

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…