Abstract
Visual as well as genetic biometrics are routinely employed to identify species and individuals in biological applications. However, no attempts have been made in this domain to computationally enhance visual classification of rare classes with little image data via genetics. In this paper, we thus propose aligned visual-genetic learning as a new application domain with the aim to implicitly encode cross-modality associations for improved performance. We demonstrate for the first time that such alignment can be achieved via deep embedding models and that the approach is directly applicable to boosting long-tailed recognition (LTR), particularly for rare species. We experimentally demonstrate the efficacy of the concept via application to microscopic imagery of 30k+ planktic foraminifer shells across 32 species when used together with independent genetic data samples. Most importantly for practitioners, we show that visual-genetic alignment can significantly benefit visual-only recognition of the rarest species. Technically, we pre-train a visual ResNet50 deep learning model using triplet loss formulations to create an initial embedding space. We re-structure this space based on genetic anchors embedded via a Sequence Graph Transform (SGT) and linked to visual data by cross-domain cosine alignment. We show that an LTR approach improves the state-of-the-art across all benchmarks and that adding our visual-genetic alignment improves per-class and particularly rare tail class benchmarks significantly further. Overall, visual-genetic LTR training raises rare per-class accuracy from 37.4% to benchmark-beating 59.7%. We conclude that visual-genetic alignment can be a highly effective tool for complementing visual biological data containing rare classes. The concept proposed may serve as an important future tool for integrating genetics and imageomics towards a more complete scientific representation of taxonomic spaces and life itself. Code, weights, and data splits are published for full reproducibility.
Original language | English |
---|---|
Title of host publication | Proceedings - 2024 IEEE Winter Conference on Applications of Computer Vision, WACV 2024 |
Publisher | Institute of Electrical and Electronics Engineers (IEEE) |
Pages | 7100-7110 |
Number of pages | 11 |
ISBN (Electronic) | 9798350318920 |
ISBN (Print) | 9798350318937 |
DOIs | |
Publication status | Published - 9 Apr 2024 |
Event | IEEE/CVF Winter Conference on Applications of Computer Vision - Waikoloa, United States Duration: 3 Jan 2024 → 8 Jan 2024 https://wacv2024.thecvf.com/ |
Publication series
Name | IEEE Workshop on Applications of Computer Vision (WACV) |
---|---|
Publisher | IEEE |
ISSN (Print) | 2472-6737 |
ISSN (Electronic) | 2642-9381 |
Conference
Conference | IEEE/CVF Winter Conference on Applications of Computer Vision |
---|---|
Abbreviated title | WACV |
Country/Territory | United States |
City | Waikoloa |
Period | 3/01/24 → 8/01/24 |
Internet address |
Bibliographical note
Publisher Copyright:© 2024 IEEE.