The demand for deep neural networks that conduct analysis tasks for medical data has expanded dramatically in recent years due to technological advances in deep learning and hardware computing capabilities. Data sources include multiple modalities, such as radiology and pathology images, health records, and genomic repositories. While these data were utilized in silos historically, however, today, can we leverage deep learning algorithms to integrate these data sources and produce multi-modal insights?