Earth embeddings reveal detailed landscape features with neuroscience methods

NeurEO: dissecting Earth observation embeddings with computational neuroscience

Computer Vision and Pattern Recognition

Summary

Earth embeddings are simplified representations of complex satellite data that help computers understand land features. These embeddings are usually hard to interpret, so the authors used methods from neuroscience, which study how brain cells respond to stimuli, to better understand what the embeddings represent. They found that the embeddings capture specific patterns related to land cover and elevation, such as differences between northern and southern forest edges. Removing strong common patterns revealed even finer details, helping us know what these data summaries actually mean. This understanding is important for creating trustworthy applications that use satellite data.

What this means in practice

Authors

Thijs L van der Plas, Jacob JW Bakermans, Ioannis N Athanasiadis

Abstract

Earth embeddings are compressed representations of rich Earth observation data that enable downstream models to tackle a wide range of complex geospatial tasks. Yet, this increase in performance comes at the cost of decreased interpretability, as the semantic meaning of Earth embeddings is unknown. To address this challenge, we borrow methods from the field of computational neuroscience, which aim to interpret activity in (biological) neurons. Here, we adapt and apply these methods to assign meaning to AlphaEarth embeddings. We analyse embedding features both as stand-alone data, equivalent to spontaneous neural activity, and in relation to land cover and elevation correlates, equivalent to stimulus-response neural activity. We demonstrate correlational and multi-scale structure in the embedding features. Embedding features can display highly specific and directional tuning to land cover and elevation, such as a `North vs South woodland edge' response. When we remove the main trends of combined land cover responses (like water versus land tuning), we discover fine-grained representational details with increased effective dimensionality. Together, our analyses start to unravel what these Earth embeddings represent, which is critical for developing trustworthy downstream applications.
View PDFOpen arXiv