Multimodal · Foundations and Architectures¶
This module covers continuous feature concatenation, cross-attention access, and joint generation with discrete tokens, distinguishing shared embedding spaces from generative distributions. Early/Late Fusion has different definitions across the literature; compare actual computation paths rather than mistaking input-level concatenation for shallow interaction.
Useful follow-up questions include: do visual representations occupy language sequence positions? How much information is compressed? Which parameters are frozen? Does understanding images imply the ability to generate them? These questions test architectural understanding better than memorizing model categories.
Chapters¶
Connections to other modules¶
This module covers only the architecture of integrating modalities. For training mixtures and alignment, see Training Data and Alignment. For modality-specific capabilities, see Vision and Document Intelligence, Speech and Audio, and Multimodal Generation. For an overview of multimodal training, inference, evaluation, and safety, see LLM · Multimodal Models.
Return to Multimodal Topics.