Research question
Can instructions be hidden in images and audio to hijack multimodal assistants?
Key findings
- Adversarial perturbations to images and audio steer LLaVA and PandaGPT to follow attacker instructions - Perturbations are imperceptible and persist across the dialogue
