betaFeedback

WorksComputer Science

Where Is Rank-1 DiM Refusal Intervenable in Multimodal LM Derivatives?

Abstract

Open-weight model hubs now ship not only chat checkpoints but a long tail of derivatives—fine-tunes, safety guards, vision–language models (VLMs), audio wrappers, and generative heads—yet residual-stream safety audits still often intervene at a single site, usually the text decoder TT. We ask a localization question under one shared mean / rank-1 difference-in-means (DiM) extract==intervene protocol, with matched-norm random and empty-rate controls: does causal chat attack lift concentrate at TT, or do perceptual encoders, projectors, and fusion paths (EE/PP/FF) supply a comparable bypass? On the mixed-input settings we stress-test—HADES adversarial images paired with harmful text prompts, and spoken AdvBench via text-to-speech (TTS) or silence—text DiM at TT raises attack success rate (ASR) while the same mean DiM at EE/PP/FF does not; attacks track harmful text rather than image or audio content. Margin-rank and leverage probes on a Vision leaf show that this recipe is extremal at TT and near chance at PP/EE, which pressures a weak-extractor reading of those flat cells for the DiM family we study. Catalog breadth and mapper-free base→\toderivative transfer when residual widths match provide supporting coverage under the same gates; generative U-Nets are a separate band under generation judges. The primary claim is this recipe-scoped TT vs.\ non-text intervenability asymmetry—intervention accessibility under mean / rank-1 DiM, not a claim that refusal is absent or fully represented outside TT (text vs.\ perceptual contrasts and pooling differ by site). On VLGuard Unsafe prompts (n=200n{=}200; Llama-Guard), the same TT vs.\ PP/EE directionality holds at modest absolute ASR (0.030→0.1500.030{\to}0.150 at TT). An intervention-oriented Class×\timessite ledger records where this protocol still hooks as supporting coverage, not as a universal multimodal bypass.

Paper details

vlnc:W_nan9wk99
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
arXiv nonexclusive license
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Large language models

View statement
Looking for related papers…