Heterogeneous data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while training data can be accurately collected to include a variety of sensory modalities, it is often the case that not all of them are available in real life (testing) scenarios, where a model has to be deployed. This raises the challenge of how to extract information from multimodal data in the training stage, in a form that can be exploited at test time, considering limitations such as noisy or missing modalities. This paper presents a new approach in this direction for RGB-D vision tasks, developed within the adversarial learning and privileged information frameworks. We consider the practical case of learning representations from depth and RGB videos, while relying only on RGB data at test time. We propose a new approach to train a hallucination network that learns to distill depth information via adversarial learning, resulting in a clean approach without several losses to balance or hyperparameters. We report state-of-the-art results for object classification on the NYUD dataset, and video action recognition on the largest multimodal dataset available for this task, the NTU RGB+D, as well as on the Northwestern-UCLA.

Download full-text PDF

Source
http://dx.doi.org/10.1109/TPAMI.2019.2929038DOI Listing

Publication Analysis

Top Keywords

learning privileged
8
test time
8
adversarial learning
8
learning
4
privileged adversarial
4
adversarial discriminative
4
discriminative modality
4
modality distillation
4
distillation heterogeneous
4
data
4

Similar Publications

Exploring the experiences of children's palliative care for forced migrant families in the United Kingdom: an interpretative phenomenological study.

Front Pediatr

January 2025

Department of Nursing and Midwifery, School of Health Sciences, College of Medicine and Health Sciences, University of Birmingham, Birmingham, United Kingdom.

Introduction: This study focused on understanding the experiences of forced migrant families and the health care professionals who care for them within palliative care. Palliative care for children requires an active, holistic approach to care, with a focus upon improving quality of life. Forced migrant families encounter a range of additional challenges including the loss of family, belongings, and all sources of familiarity and support.

View Article and Find Full Text PDF

Background: Medical professionals do not always manage their own health in an optimal way. Research conducted on medical students' response to illness is limited; however, some studies suggest they choose to self-manage and seek advice outside of the traditional pathways of care. This study aims to understand how medical students perceive and manage their health as they transition through medical school.

View Article and Find Full Text PDF

Between the racial reckoning of 2020 and wider spread policy development that is explicitly homophobic and transphobic, there have been consistent and resurgent calls for clinicians to address aspects of power and privilege in psychotherapy. This is especially important in a field that continues to be largely White, cisgender, and heterosexual (not to mention abled, socioeconomically privileged, and privileged in many other aspects of human diversity). However, too few models for how to accomplish this in actual practice are offered in the literature.

View Article and Find Full Text PDF

CovCysPredictor: Predicting Selective Covalently Modifiable Cysteines Using Protein Structure and Interpretable Machine Learning.

J Chem Inf Model

January 2025

Computer-Aided Drug Discovery, Global Discovery Chemistry, Novartis Biomedical Research, 181 Massachusetts Avenue, Cambridge, Massachusetts 02139, United States.

Targeted covalent inhibition is a powerful therapeutic modality in the drug discoverer's toolbox. Recent advances in covalent drug discovery, in particular, targeting cysteines, have led to significant breakthroughs for traditionally challenging targets such as mutant KRAS, which is implicated in diverse human cancers. However, identifying cysteines for targeted covalent inhibition is a difficult task, as experimental and in silico tools have shown limited accuracy.

View Article and Find Full Text PDF

What constitutes a medical miseducation? Ten mishaps, readily remedied.

Med Teach

January 2025

Peninsula School of Medicine, Faculty of Health, University of Plymouth, Penzance, Cornwall, United Kingdom.

In adopting reductive instrumentalism as a dominant discourse medical education can be seen to have cultivated a values monoculture resistant to innovation. This culture characteristically retreats to the safety of conservatism rather than diversifying and innovating to embrace values beyond the functional - such as the ethical, aesthetic, and political. Here - where teaching displaces facilitation of learning - training is privileged over education, competence over capability, linearity over complexity, and information over knowledge.

View Article and Find Full Text PDF

Want AI Summaries of new PubMed Abstracts delivered to your In-box?

Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!