WorksComputer Science
Learning Administrative Boundary-Aware Location Encoders via Reverse Geocoding
Abstract
Location encoders learn compact representations of latitude–longitude coordinates for downstream geospatial prediction tasks. Existing approaches are pretrained using continuous or naturally occurring supervision, such as imagery, climate variables, or species distributions, and therefore do not explicitly capture administrative boundaries or place names, despite their strong influence on many real-world geospatial phenomena. We introduce, to our knowledge, the first method for learning administrative boundary-aware location encoders, using reverse-geocoded text as a self-supervised training signal. Our approach can be applied to existing location encoder architectures and consistently improves their downstream performance. We further introduce RevGeoRQ, a custom architecture tailored to reverse-geocoding supervision, which yields additional gains and exceeds contemporary location encoders on several downstream tasks. Finally, the use of human-interpretable reverse-geocoded strings and residual-quantized semantic IDs provides a more interpretable representation of geographic location than existing approaches. Code and model weights to be released soon.
Introduction
Location encoders have emerged as a foundational building block for modern data-driven geographic information systems, enabling models to convert raw latitude–longitude coordinates into rich, high-dimensional representations. Learning effective such spatial representations is critical across a wide spectrum of downstream tasks, ranging from predicting species distributions and environmental variables to optimizing urban planning, route navigation, and localized economic forecasting. By replacing hand-engineered spatial features with expressive pre-trained neural embeddings, location encoders allow predictive models to better reason about distance, regional continuity, and geographic context directly from raw coordinate inputs, reducing the need for time-consuming implementation.
Despite their success, training general-purpose location encoders that transfer effectively across diverse downstream applications remains challenging. Existing representation learning strategies typically rely on continuous pretraining targets, such as multispectral satellite imagery, climate grids, or naturally occurring species presence logs. While these signals capture smooth environmental gradients and various visual patterns, they largely overlook the discrete, structured boundaries that define human geography. However, real-world geospatial phenomena are heavily shaped by administrative borders, regional nomenclature, and socio-political divisions - for example, boundaries between urban and natural terrain are often artificial, informed by administrative boundaries - exactly the type of information that purely physical or visual pretraining signals fail to encode explicitly. As a result, current encoders often struggle on downstream tasks where administrative context and localized place identity play a decisive role.
To bridge this gap, we use reverse-geocoded text as a rich, scalable, and relatively cheap signal for developing self-supervised location encoders. Reverse geocoding translates raw spatial coordinates into structured text strings containing hierarchical place names, administrative units, and other localized metadata. By training encoders to predict or align with these textual representations in a self-supervised manner, models can internalize both continuous spatial topology and discrete human-centric boundaries without requiring costly manual annotations. This supervision source applicable across diverse encoder architectures, providing a fine-grained, semantically rich representation of geographic space that aligns well with human mental models of geography. The novel self-supervised training is one of the main contributions of our work, alongside a novel architecture tailored to reverse-geocoding supervision. As an additional contribution, we show evidence that reverse geocoded output improves interpretability in downstream systems.
The rest of the paper is structured as follows. We discuss past work and how our contributions complement these past approaches in section 2. Section 3 contains the description of our novel model architecture and the location encoder usage. Our main experimental results on downstream tasks are discussed in section 4, with a detailed ablation study discussed in section 5. We perform further analysis of the interpretability of our embeddings in section 6. Finally, we briefly discuss limitations to our approach in section 7, and conclude with future work in section 8.
Related Work
Location encoders project continuous geographic coordinates (latitude and longitude) into high-dimensional representations for downstream prediction tasks. Early approaches relied on nonparametric sinusoidal encodings or explicit spherical coordinate parameterizations, such as Sphere2Vec Mai et al. [18], to preserve great-circle distances without planar distortion. Modern representation learning paradigms instead pretrain location encoders in a self-supervised manner by grounding coordinate representations in co-located modalities. Multimodal frameworks such as GeoCLIP Vivanco Cepeda et al. (2023) and SatCLIP Klemmer et al. [12] contrastively align coordinate embeddings with remote-sensing or street-level imagery, while ecological models like BioCLIP Stevens et al. [25] use hierarchical taxonomic occurrences. Further advancing image-coordinate alignment, RANGE Dhakal et al. [2] enhances visual feature density through retrieval-augmented neural fields over imagery. However, because these methods rely exclusively on visual, physical, or ecological co-occurrence signals, they treat geographic space purely as a continuous physical manifold, failing to explicitly capture discrete, human-defined administrative structures and localized place names.
Integrating natural language with spatial data has gained momentum as Large Language Models (LLMs) demonstrate strong capabilities in geographic reasoning and spatial representation. Linear probing and cognition studies, such as Gurnee & Tegmark [8], Godey et al. [6], and Xia et al. (2026), show that LLMs can internalize structured global spatial layouts purely from textual metadata and local descriptions of points of interest. To take advantage of this latent spatial knowledge for downstream predictions, frameworks like GeoLLM Manvi et al. [19] and LLMGeoVec He et al. [9] augment spatial coordinate prompts with auxiliary textual descriptions from OpenStreetMap or POI categories. While effective at unlocking spatial knowledge, these paradigms rely on heavy LLM backbones or expensive prompt-based inference at prediction time. In contrast, our approach uses reverse-geocoded text purely as a self-supervised pretraining signal, distilling rich administrative and textual spatial context directly into a compact, standalone location encoder without incurring LLM overhead at inference time.
Discrete quantization methods such as VQ-VAE Van Den Oord et al. [26] and VQGAN Esser et al. [5] represent continuous high-dimensional spaces using learned discrete codebooks. This idea was later adapted successfully to generative retrieval and recommendation for retail Rajput et al. [22], video Li et al. [15], audio Penha et al. [21], and points of interest Wang et al. (2025); ); Lv et al. [16]}.
Reverse geocoding maps raw spatial coordinates to structured human-readable text, such as street addresses, municipal divisions, and administrative hierarchies. While commercial APIs, such as the Google Maps Geocoding API Google Maps Platform [7], provide high-precision global reverse geocoding, their cost structure becomes prohibitively expensive when querying millions of coordinates required for large-scale machine learning pretraining. Conversely, free or open-source geocoding alternatives often suffer from spatial noise, incomplete boundary coverage, and positional inaccuracy, as systematically evaluated by Kounadi et al. [13]. To mitigate errors in raw geocoding outputs, probabilistic frameworks such as Yin et al. (2020) model candidate entity distributions over administrative zones to resolve geographic ambiguities. Rather than relying on online commercial APIs or static categorical lookups during downstream prediction, we use reverse-geocoded strings as an offline, self-supervised training signal. By learning predict structured text representations during pretraining, location encoders internalize fine-grained administrative hierarchies directly from raw coordinates, distilling high-quality geographic context into a fast, standalone encoder without incurring runtime API expenses or latency. While we work primarily with free, off-the-shelf reverse geocoding systems, we expect that improving the quality of the reverse geocoder will only result in improvements in downstream performance, as shown in past work, such as Kilic et al. [11].
Methodology
Dataset construction
Let denote a geographic coordinate and a reverse geocoder. Its reverse-geocoded string is
We format as city, admin1, admin2, country, continent. For example, maps to Manhattan, New York, New York County, United States, North America. To evaluate generalization across location encoder architectures, we apply our method to each model’s native pretraining dataset. For model , coordinates set are reverse geocoded to obtain , and the model is pretrained on the paired sets . We use the pycountry1, pycountry_convert2, and reverse-geocoder3 libraries.
Architecture
Our model architecture has three main components: a) a Location Encoder, b) a Text Encoder, and c) a Residual Quantized Variational Autoencoder (RQ-VAE), illustrated in Figure 1 (Left).

Figure 1. Left: Pretraining architecture. Right: Learned (level-0) codes from RQVAE Semantic IDs on S2-100K points
Location encoder
Given an encoder and coordinates , the location embedding is
The coordinates may be further transformed using spherical-harmonic, sinusoidal, or other frequency-based encodings (Klemmer et al., [12]; Rußwurm et al., [24]; Dollinger et al., [4]).
Text encoder
For a location with coordinates , let denote its reverse-geocoded string, obtained according to Eq. (1). The string is passed through a frozen pretrained text encoder to obtain the corresponding text embedding :
where denotes the space of reverse-geocoded strings. We use the pretrained frozen Qwen3-Embedding (Zhang et al. (2025)) models (0.6B and 4B variants) and use the unnormalized last-token hidden state from its final layer (, respectively).
Residual-Quantized Variational Autoencoder
Owing to the discrete nature of administrative boundaries introduced by reverse geocoding, and inspired by Rajput et al. [22], we use a Residual-Quantized Variational Autoencoder (RQ-VAE) [14] (Zeghidour et al. 2021) to turn the location encoder output (Eq. (2)) into a discrete, hierarchical code. Unlike a standard RQ-VAE, which reconstructs its own input, we train this to use as input to construct the frozen text embedding (Eq. (3)) of the location’s reverse-geocoded string (Eq. (1)), as shown below.
A trainable linear projection maps location encoder backbones of differing dimensions into the text space, and an encoder produces the latent . Residual quantization then runs over levels, each with its own codebook of candidate codewords, where indexes the entries of . Let denote the residual remaining after level , starting from . Level finds the codebook entry nearest to the current residual, records its index , and subtracts the corresponding codeword to form the next residual:
Thus is the discrete code chosen at level and the codeword it points to. Each level quantizes the residual left by the previous ones, so the tuple of chosen codes , which we call the location’s Semantic ID, is a coarse-to-fine descriptor indexing up to distinct locations. The decoder reconstructs the text embedding from the summed codewords, , and the tokenizer is trained end-to-end with the loss , where and . Here denotes the stopgradient operation [26] and weights the commitment loss; this objective jointly trains the location encoder , the projection , the encoder , the decoder , and all codebooks. Specific details about the RQ-VAE design are provided in Appendix A.
Experimental Setup
Pretraining
We experiment with five location encoder architectures: SatCLIP [12], GeoCLIP (Vivanco Cepeda et al., 2023), SINR [1], Climplicity [4], and a custom Reverse Geocoding Residual Quantized (RevGeoRQ) architecture.
Adapting existing backbones
Each model is pretrained on its native coordinate set, but we replace the original labels and training objective with our reverse-geocoded embeddings and objective. All parameters are randomly initialized to enable a direct comparison with native pretraining. During training, the location encoder and RQ-VAE outputs are aligned with the text encoder output using the loss described in § 3.2.3.
Since SatCLIP and GeoCLIP expose usable final-layer embeddings, the adapter is appended directly to them. In contrast, SINR and Climplicity have task-specific heads which are replaced by the adapter attached to the penultimate embedding layer. Our adapted SatCLIP model use uses unlike the native backbone which uses since we find that using performs substantially worse in our setting; we investigate this in § 5.
To demonstrate gains with minimal architectural changes, we use the 0.6B Qwen encoder for backbone adaptation. Its smaller embedding dimension enables strong performance with only a lightweight adapter and no backbone modification. Larger Qwen3 variants are used in selected settings. Full pretraining details are provided in Appendix B, Table 4.
Custom architecture
We introduce RevGeoRQ, a SatCLIP-based architecture optimized for our method, which uses Qwen3-Embedding-4B embeddings as reconstruction targets. A spherical-harmonic layer () maps coordinates to 100 dimensions, followed by a modified SirenNet Rußwurm et al. [24] with widths . Matching the 2560-dimensional Qwen output lets the representation feed directly into the RQ-VAE without an adapter. RevGeoRQ is pretrained on SatCLIP’s S2-100K dataset using the settings in § 4.1.1.
Downstream fine-tuning
We evaluate on four regression and three classification tasks. Classification includes country [12], ecoregion, and biome prediction (both from Dinerstein et al. (2017)); regression includes air temperature Hooker et al. [10], elevation and population density Rolf et al. [23], and housing price prediction Pace & Barry [20]. We evaluate five variants on each downstream task:
Baseline: reverse-geocoded text encoded by Qwen3-Embedding-0.6B and passed directly to the downstream MLP probe.
Native: the original pretrained location encoder using author-released weights.
Text Residual Quantized (TextRQ): our adaptation of each backbone, using the adapter output trained against Qwen3-Embedding-0.6B embeddings.
RevGeoRQ: our custom architecture described in § 4.1.2.
Combined: concatenates Native and TextRQ embeddings for adapted backbones; for RevGeoRQ, it concatenates RevGeoRQ with native ClimImplicit embeddings.
Each backbone is frozen and adapted using a two-layer MLP head (64 hidden units per layer), trained for 3000 full-batch epochs with Adam and a constant learning rate of . We use a 50/50 train-validation split across three random seeds and report the mean and standard deviation. For SatCLIP, Native uses the best-performing variant from Klemmer et al. [12], while SatCLIP-TextRQ uses , which performs better in our setting; we investigate this choice in § 5.
Results
Shown in Table 1, and discussed below.
| Model | air_temp | elevation | population | cali_housing | avg. | country | eco regions | biome | avg. acc. |
| Baseline | |||||||||
| Qwen0.6B | .820 ± .005 | .727 ± .006 | .692 ± .013 | .675 ± .003 | .729 | .926 ± .003 | .489 ± .007 | .705 ± .009 | .707 |
| SatCLIP | |||||||||
| Native | .890 ± .008 | .902 ± .006 | .812 ± .003 | .718 ± .001 | .831 | .907 ± .009 | .689 ± .010 | .841 ± .005 | .812 |
| TextRQ | __.906 ± .009__ | __.903 ± .005__ | __.825 ± .010__ | .687 ± .002 | .830 | __.946 ± .002__ | __.719 ± .013__ | __.849 ± .012__ | __.838__ |
| Combined | __.925 ± .008__ | __.909 ± .003__ | __.821 ± .004__ | __.696 ± .008__ | __.838__ | .944 ± .003 | __.722 ± .008__ | __.853 ± .009__ | __.840__ |
| GeoCLIP | |||||||||
| Native | .923 ± .003 | .773 ± .005 | .650 ± .013 | .631 ± .016 | .744 | .907 ± .003 | .575 ± .013 | .799 ± .005 | .760 |
| TextRQ | .874 ± .004 | .828 ± .006 | .782 ± .002 | .715 ± .010 | __.800__ | .906 ± .005 | .560 ± .014 | .754 ± .001 | .740 |
| Combined | __.950 ± .004__ | __.819 ± .001__ | __.719 ± .001__ | __.683 ± .017__ | __.793__ | __.929 ± .004__ | __.621 ± .010__ | __.810 ± .004__ | __.787__ |
| Climplicit | |||||||||
| Native | .950 ± .005 | .897 ± .001 | .685 ± .007 | .731 ± .013 | .816 | .895 ± .004 | .673 ± .011 | .846 ± .004 | .805 |
| TextRQ | .874 ± .014 | .894 ± .006 | .810 ± .007 | .700 ± .005 | __.820__ | __.945 ± .005__ | .680 ± .010 | .824 ± .007 | __.816__ |
| Combined | __.960 ± .006__ | __.909 ± .004__ | __.732 ± .009__ | __.722 ± .012__ | __.831__ | __.947 ± .003__ | __.711 ± .002__ | __.866 ± .003__ | __.841__ |
| SINR | |||||||||
| Native | .906 ± .011 | .884 ± .003 | .797 ± .002 | .714 ± .008 | .825 | .936 ± .002 | .710 ± .001 | .841 ± .005 | .829 |
| TextRQ | __.926 ± .007__ | .854 ± .006 | .806 ± .005 | .670 ± .015 | .814 | __.946 ± .002__ | .715 ± .009 | .850 ± .005 | __.837__ |
| Combined | __.942 ± .002__ | .882 ± .004 | __.811 ± .003__ | .698 ± .016 | __.833__ | .945 ± .002 | __.721 ± .013__ | __.852 ± .007__ | __.839__ |
| Custom | |||||||||
| RevGeoRQ | .929 ± .001 | .896 ± .004 | .831 ± .004 | .685 ± .007 | .835 | .949 ± .000 | .723 ± .011 | .852 ± .012 | .841 |
| Combined | .972 ± .001 | .928 ± .012 | .767 ± .003 | .703 ± .008 | .843 | .945 ± .001 | .725 ± .004 | .868 ± .003 | .846 |
Table 1. Results across downstream datasets. Bold marks the best mean per dataset; __underline__ indicates a TextRQ or Combined variant that outperforms its corresponding Native model. Regression tasks report ; classification tasks report accuracy.
TextRQ vs. Native: TextRQ outperforms the native frozen backbones in 16 of 28 comparisons (57%), with the largest gains on admin-boundary aligned tasks such as country and population prediction. Results are mixed on smoother targets such as air temperature and elevation, and California Housing is a notable exception, where TextRQ improves only over GeoCLIP. As explored in § 5, we attribute this to reverse-geocoded supervision favoring low spatial frequencies: raising SatCLIP-TextRQ to recovers California Housing performance but substantially degrades every other task, so we use .
Combining TextRQ and Native embeddings: Concatenating the native and TextRQ embeddings and using this as input to the MLP yields further gains, outperforming Native in 24 of 28 comparisons (86%). This suggests that reverse-geocoding self-supervision not only achieves competitive performance relative to continuous, naturally occurring supervision, but also provides complementary information that improves performance when the two are combined. Similar to TextRQ, the combined models underperform on California Housing. A drawback of the combined approach is that it doubles the backbone parameter count. To reduce this overhead, we explored single-backbone multi-task pretraining with both native and reverse-geocoded supervision; however, this consistently underperformed simple embedding concatenation.
RevGeoRQ (§ 4.1.2): This is the strongest single-embedding model, outperforming all Native backbones and single-embedding TextRQ models as measured by average accuracy and average . Concatenating RevGeoRQ with Climplicit embeddings results in the best overall model. Although this combination improves aggregate performance, it reduces performance on population prediction, a task on which Climplicit itself is relatively weak. These results further supports the value of combining complementary encoder representations and supervision signals, while also highlighting that task-specific specialization may be preferable when certain downstream tasks are prioritized. Together, the two custom variants establish a new state of the art for location encoders, with one of the two achieving the best performance on 6 of 7 tasks. California Housing remains the consistent exception, with both variants underperforming on this task.
Location Embeddings as a geo-prior
Prior work shows that location can provide a useful prior for geographically dependent predictions [12, 24]. We therefore evaluate our representations as geographic priors on iNaturalist 2018 (Van Horn, 2018). Following prior protocols [17, 24, 2], for each image , a pretrained image classifier produces , while a linear classifier trained on location embedding produces . At inference, we combine them multiplicatively as and report top- accuracy. We exclude SINR because its location encoder is supervised with iNaturalist species labels, preventing an independent transfer evaluation. Additional details are provided in Appendix § 12.
Table 2 show that TextRQ generally improves over Native, outperforming all backbones at top-1, matching or exceeding them at top-3, and improving top-5 for SatCLIP and GeoCLIP; only Climplicit retains a Native advantage at top-5. Combining Native and TextRQ helps only the weaker GeoCLIP backbone. RevGeoRQ outperforms the vision-only baseline and native GeoCLIP but trails the strongest adapted TextRQ models. SatCLIP-TextRQ is the best overall based on mean top-.
| Model | top-1 | top-3 | top-5 | mean |
| Baseline | ||||
| Vision-only (Inception-v3) | 0.6617 | 0.8331 | 0.8803 | 0.7917 |
| SatCLIP | ||||
| Native (L=40) | 0.7537 ± 0.0009 | 0.8945 ± 0.0003 | 0.9280 ± 0.0003 | 0.8587 |
| TextRQ (L=10) | 0.7730 ± 0.0008 | 0.9019 ± 0.0006 | 0.9306 ± 0.0004 | 0.8685 |
| Combined | 0.7677 ± 0.0007 | 0.8964 ± 0.0006 | 0.9256 ± 0.0003 | 0.8633 |
| GeoCLIP | ||||
| Native | 0.6901 ± 0.0001 | 0.8546 ± 0.0001 | 0.8975 ± 0.0000 | 0.8141 |
| TextRQ | 0.7584 ± 0.0003 | 0.8985 ± 0.0005 | 0.9291 ± 0.0008 | 0.8620 |
| Combined | 0.7647 ± 0.0003 | 0.9025 ± 0.0005 | 0.9320 ± 0.0008 | 0.8664 |
| Climplicit | ||||
| Native | 0.7653 ± 0.0005 | 0.8938 ± 0.0004 | 0.9237 ± 0.0001 | 0.8609 |
| TextRQ | 0.7669 ± 0.0010 | 0.8938 ± 0.0006 | 0.9222 ± 0.0002 | 0.8610 |
| Combined | 0.7604 ± 0.0005 | 0.8842 ± 0.0005 | 0.9138 ± 0.0005 | 0.8528 |
| Custom | ||||
| RevGeoRQ | 0.7545 ± 0.0002 | 0.8808 ± 0.0007 | 0.9104 ± 0.0004 | 0.8486 |
| Combined | 0.7489 ± 0.0011 | 0.8706 ± 0.0002 | 0.8998 ± 0.0007 | 0.8398 |
Table 2. iNaturalist 2018 geo-prior evaluation. Values report top- validation accuracy as mean ± std over three seeds; the mean column gives the per-row average over top-1, top-3, and top-5. Bold marks the best mean within each model family; underline indicates improvement over the corresponding Native baseline.
Ablation Study
Table 3 ablates key architectural choices. Variants with a location encoder use the adapted SatCLIP backbone, except RevGeoRQ variants, which follow § 4.2.
| Model | air temp | elevation | population | cali. housing | avg. | country | eco regions | biome | avg. acc. |
| Qwen-0.6B | .820 ± .005 | .727 ± .006 | .692 ± .013 | .675 ± .003 | .729 | .926 ± .003 | .489 ± .007 | .705 ± .009 | .707 |
| RQVAE-only (0.6B) | .913 ± .005 | .564 ± .165 | .683 ± .049 | .508 ± .021 | .667 | .907 ± .010 | .560 ± .022 | .771 ± .005 | .746 |
| SatCLIP-TextRQ (L=40, 0.6B) | −.028 ± .019 | .649 ± .019 | .458 ± .038 | .729 ± .013 | .452 | .382 ± .012 | .266 ± .003 | .415 ± .008 | .354 |
| SatCLIP-TextRQ (L=10, 0.6B) | .906 ± .009 | .903 ± .005 | .825 ± .010 | .687 ± .002 | .830 | .946 ± .002 | .719 ± .013 | .849 ± .012 | .838 |
| SatCLIP-TextRQ (L=10, 4B) | .922 ± .005 | .890 ± .011 | .831 ± .001 | .682 ± .019 | .831 | .949 ± .002 | .725 ± .005 | .853 ± .009 | .842 |
| RevGeoRQ | .929 ± .001 | .896 ± .004 | .831 ± .004 | .685 ± .007 | .835 | .949 ± .000 | .723 ± .011 | .852 ± .012 | .841 |
| RevGeoRQ + SatCLIP-L40 | .940 ± .003 | .907 ± .004 | .834 ± .005 | .673 ± .010 | .839 | .947 ± .004 | .727 ± .009 | .856 ± .006 | .843 |
| RevGeoRQ + Climplicit | .972 ± .001 | .928 ± .012 | .767 ± .003 | .703 ± .008 | .843 | .945 ± .001 | .725 ± .004 | .868 ± .003 | .846 |
Table 3. Model architecture and parameter ablations. Regression reports , classification reports accuracy, and avg. denotes the row mean within each group. Bold marks the best value per column.
Text encoder and RQ-VAE reconstruction loss: Qwen-0.6B alone performs best on tasks aligned with administrative names, especially country classification, but is weaker on finer-grained spatial tasks and poorest on eco regions, whose 649 classes often cross administrative boundaries. Adding RQ-VAE layers with DIRECT latitude–longitude encoding (Rüßwurm et al., 2022) improves air temperature, ecoregion, and biome performance, but hurts elevation and California housing. Adding the SatCLIP backbone yields the full TextRQ model (Sec. 4.2), combining reverse-geocoded semantics with continuous geographic features and substantially improving overall performance. Replacing Qwen-0.6B with Qwen-4B provides further but modest gains.
Effect of spatial frequency: Increasing the spherical-harmonic degree from to degrades all tasks except California Housing, where performs best. Consistent with Klemmer et al. [12], higher supports finer spatial variation, benefiting tasks such as housing prices that change over small areas, while smoother targets such as temperature and elevation gain little. We hypothesize that reverse-geocoded supervision further favors lower frequencies: it assigns shared representations to large, sometimes heterogeneous regions, making a better match to both the training signal and predominantly low-frequency downstream tasks.
Effect of concatenation: Concatenating TextRQ embeddings with native location encoder embeddings generally improves performance, suggesting that our administrative-boundary-based text supervision provides a complementary signal to the naturally occurring supervision used by the native backbones. Moreover, the relative strength of the native backbones is largely preserved after concatenation: when one native embedding outperforms another on a given task, its corresponding concatenated representation typically remains stronger as well.
Objective function: Using a CLIP-style contrastive loss to align the geospatial embedding with the Qwen text embedding shows equivalent or better performance to the RQVAE loss for SatCLIP and SINR, but significantly worse performance for GeoCLIP and Climplicit. Due to its better cross-encoder generalizability, we choose the RQVAE loss/architecture. Details in Appendix C Table 5.
| Model (all TextRQ) | Regression | Classification | |||||||
| air temp | elevation | population | cali housing | avg. | country | regions | biome | avg. acc. | |
| SatCLIP () | |||||||||
| CLIP | .930 .009 | .888 .007 | .831 .002 | .671 .006 | .830 | .948 .004 | .723 .009 | .859 .007 | .843 |
| RQVAE | .906 .009 | .903 .005 | .825 .010 | .687 .002 | .830 | .946 .002 | .719 .013 | .849 .012 | .838 |
| GeoCLIP | |||||||||
| CLIP | .754 .010 | .436 .044 | .253 .027 | .694 .007 | .534 | .767 .003 | .414 .001 | .651 .004 | .611 |
| RQVAE | .796 .022 | .693 .007 | .652 .003 | .702 .003 | .711 | .793 .006 | .417 .008 | .666 .006 | .625 |
| ReSIREN | |||||||||
| CLIP | .573 .044 | .896 .004 | .767 .003 | .705 .008 | .735 | .913 .006 | .652 .005 | .808 .008 | .791 |
| RQVAE | .874 .014 | .894 .006 | .810 .007 | .700 .005 | .820 | .945 .005 | .680 .010 | .824 .007 | .816 |
| SINR | |||||||||
| CLIP | .905 .014 | .867 .005 | .812 .007 | .758 .001 | .836 | .942 .003 | .703 .008 | .842 .007 | .829 |
| RQVAE | .926 .007 | .854 .006 | .806 .005 | .670 .015 | .814 | .946 .002 | .715 .009 | .850 .005 | .837 |
Table 5. Downstream performance after ablating the pretraining objective on each encoder’s own data. Bold shows best per backbone-task pair between CLIP and RQVAE objectives
We also tuned codebook size, number of levels, and latent dimension (final settings in Appendix A).
Embeddings analysis
Semantic IDs
Emergent geographic hierarchy: Similar to prior work [22, 15], we observe that the hierarchical nature of RQ-VAEs induces a coarse-to-fine structure in the learned Semantic IDs; in our case, this coarse-to-fine structure manifests as a geographic multiresolution zoom. To illustrate this, we train a SatCLIP TextRQ model as in § 4.1.1, but with codebook sizes 8, 32, 128, and 256 for 1000 epochs, and use this model to compute the Semantic IDs for the S2-100K points. As shown in Figure 1 (Right), the coarsest codebook with selected codes partitions the latitude–longitude space into eight regions at continental scale. Cross-tabulating each code against its reverse-geocoded continent labels shows that every code roughly aligns with a single continent or a coherent macro-region (Figure 2 Left), showing that the coarsest level estimates continent-scale semantics. The next codebook refines each coarse region into finer, still-contiguous sub-regions. Figure 2 (Center) shows this within the South and Central Asian region (), where the eight largest codes carve out coherent sub-continental clusters—South Asia, the Arabian Peninsula, East Asia, and Southeast Asia—each dominated by a single country or country group. The codes thus form a nested spatial hierarchy that zooms from continents () to sub-continental regions () learned implicitly through the Qwen-encoded reverse geocoded strings.

Figure 2. Left: the reverse-geocoded continent share within each code. Center: the eight largest codes within . Right: attribute accuracy vs. cumulative prefix depth.
Admin Boundary Aware Semantic IDs: We measure how well a Semantic ID predicts a point’s administrative attributes at an unseen location. From the training split, we record the majority attribute label for every cumulative code prefix — , then , up to . Then, for each point in the validation split, we form its Semantic ID and predict the majority label recorded in training for its prefix at that depth to compare with the ground truth attribute. If the prefix never occurs in training, we back off to the next-shorter prefix — recursively, down to and finally the global majority. We report accuracy at each prefix depth, which quantifies how much locational information the codes carry. Accuracy rises monotonically with prefix depth for every attribute (Figure 2 Right), showing that the codes carry encode coarse-to-fine boundary awareness: continent is largely fixed by alone (83%) and country by (70%), while admin1 and city keep improving up to the full Semantic ID (61% and 33%). Admin1 and city lag; this is not a sign that the Semantic ID lacks city/admin1 information, but rather the fact that even the finest cell (368 km - explored in Appendix E) is larger than most cities. Therefore, one cell covers many cities and a majority vote can’t separate them; this may be mitigated by using larger codebooks at deeper levels.
We additionally study the spatial resolution induced by the codebook prefixes, whose cell spread shrinks from 4244 to 368 km over successive levels; details are provided in Appendix E.
Location Encoder Outputs
Since the location-encoder outputs feed the downstream tasks, we qualitatively examine what this layer has learned. As illustration, we mark a query point in the Himalayan range (Nepal, 27.7°N 85.3°E) and one in the Atacama desert (23.9°S 69.2°W), and color every land location on a dense global 0.5° grid by its cosine similarity to that query point. These two cases are shown at the top and bottom of Figure 3, for our TextRQ encoder alongside the native SatCLIP L=10 and L=40 encoders.

Figure 3. Cosine similarity of each land embedding to a query point (★) — a Himalayan query (top) and an Atacama query (bottom) — for our TextRQ encoder (left) vs. native SatCLIP L=10 and L=40 (center/right).
All three encoders light up the immediate neighborhood of every query, confirming that location encoders learn the basic prior that spatially nearby points are similar (spatial autocorrelation), regardless of training signal. Beyond that, the fields diverge. Our TextRQ model’s similar regions align cleanly with administrative structure — bright at the country level, tapering to medium similarity across the surrounding subcontinent and then the continent — evidencing a well-formed country → subcontinent → continent hierarchy; the Himalaya query lights Nepal and South Asia coherently, and the Atacama query lights Chile and the Andean/southern-cone belt. SatCLIP does not show this: points within the same country can read as dissimilar, and this fragmentation intensifies with spherical-harmonic degree (worse in the high-frequency L=40 than in L=10). In exchange, SatCLIP, which is trained on imagery, recovers long-range, cross-continental similarities from shared physical character: the Himalaya query also lights the Andes and Rockies, and the Atacama query lights other arid west-coast deserts worldwide; these are largely missed by our TextRQ embedding. In summary, TextRQ captures administrative hierarchies while image-trained encoders like SatCLIP capture natural, physical characteristics; both learn spatial autocorrelation well. These are complementary signals, and combining them improves over either alone, as shown in Table 1.
Limitations
A limitation of reverse-geocoded supervision is that distinct locations within the same administrative boundaries may map to identical strings, even when they are far apart or differ substantially in local characteristics; we analyze this duplication effect in Appendix § 14. The hierarchical discretization of our model can also obscure non-local similarities, particularly between geographically distant regions on different continents. Finally, performance depends on reverse-geocoding quality. More precise geocoders may recover finer distinctions, but coverage remains inherently limited in large, sparsely populated regions where neighborhood- and locality-level boundaries are coarse or absent.
Conclusion
We introduce reverse-geocoded text as a cheap, scalable self-supervision signal that distills administrative structure into compact location encoders without inference-time LLM or geocoding cost. Our method improves existing backbones and also achieves gains when used in custom optimized RevGeoRQ architecture, showing that administrative supervision complements naturally-occurring and continuous signals. RevGeoRQ is the strongest single-embedding model in our experiments; combining it with Climplicit achieves the best overall results. The learned Semantic IDs form an interpretable hierarchy, continent-scale down to local regions. Future work includes using finer-grained geocoders, experimenting with larger codebooks, and extending Semantic IDs beyond pure location encoding to search, retrieval, and generative applications.
Residual-quantized variational autoencoder design
Encoder/decoder dimensions. The encoder and decoder are symmetric MLPs, : and : .
Codebook initialization. To prevent codebook collapse, where most inputs are mapped to only a few codewords, each level’s codebook is initialized with k-means centroids computed on the residuals of the first training batch (Zeghidour et al., 2021).
Hyperparameters. We set , , levels, codewords per level, and ; the resulting four-level, 32-entry codebooks index up to distinct Semantic IDs.
Pretraining Details
| SatCLIP | GeoCLIP | Climplicit | SINR | |
| Backbone variant | SatCLIP ViT16-L10 | GeoCLIP | Climplicit | SINR (Coords., 1000/class) |
| Training data (# points) | S2-100K (91,858) | MP-16 Flickr (4.72M) | CHELSA land grid (224M) | iNaturalist (35.5M) |
| Budget | max_epochs=100 | max_steps=50k | max_steps=50k | max_epochs=50 |
| Optimizer | AdamW (betas 0.9, 0.999; eps ) | |||
| LR | peak , weight decay | |||
| Scheduler | linear warmup (5% of steps) cosine decay to 10% floor | |||
| Batch size | 4096 |
Table 4. Per-encoder pretraining setup
Contrastive vs. RQ-VAE Objectives Across Backbones
Shown in Table 5.
iNaturalist Geo-prior Finetuning
For the iNaturalist 2018 geo-prior evaluation, we train the location probe for 30 epochs using Adam with an initial learning rate of , ExponentialLR with , batch size 1024, pos_weight=2048, and a background bank of 100k locations. Following prior work [24, 2], we use the full “assume negative” loss of Cole et al. [1], which is designed for the presence-only annotations in iNaturalist. Results are reported as mean standard deviation over seeds ; the seed controls probe initialization, background-bank sampling, and minibatch ordering, while cached frozen embeddings are shared across seeds.
Spatial Resolution of Semantic IDs
We summarize the spatial resolution induced by each Semantic ID prefix using the geographic dispersion of points assigned to the same prefix. For each cumulative prefix depth (, , ..., ), we group points sharing that prefix into a cell, map each point to a unit vector on the sphere, and take the mean vector as the cell’s spherical centroid. We then measure the great-circle (haversine) distance from each point to its cell centroid () and report the point-weighted root-mean-square of these distances as the cell dispersion or spread at that depth. As baselines, we apply the same calculation to marginal codes independently and to the entire dataset treated as a single cell. The cumulative cell dispersion shrinks monotonically from 4244 to 1384 to 654 to 368 km ( 2–3× per level), i.e. the Semantic ID acts as a coarse-to-fine geographic zoom from continent to province scale (Figure 4). Individual marginal codes instead stay pinned near the whole-corpus spread (7692 km), confirming that a single level carries no standalone location and that only the nested prefix denotes a place. The 368 km floor at full depth coincides with the city-probe ceiling, since a city is far smaller than the finest cell.

Figure 4. Cell spread per level (cumulative vs. marginal).
Loss of Spatial Granularity from Reverse Geocoding
Self-supervised labels can be no more precise than the observation that produces them. The spatial resolution of our text targets is therefore bounded by the reverse geocoder, not by the input coordinates, and our approach depends heavily on the geocoder’s quality. We use the free, offline reverse-geocoder Python package, which snaps each coordinate to its nearest GeoNames populated place and offers only moderate coverage and precision. Commercial geocoders with finer resolution and broader coverage are costly at the scale of hundreds of millions of points. As Table 6 shows, this maps between 92K and 224M distinct input locations onto at most 101K text targets, a reduction of 4.3× (S2-100K) to 2,375× (CHELSA), so all points near the same place receive an identical supervisory signal. The cost appears in Table 1 on tasks that depend on fine-grained local structure or smooth spatial variation. On California Housing, TextRQ falls below its Native counterpart for SatCLIP (), ReSIREN (), and SINR (), and none of these Combined variants recovers the Native score. For air temperature, TextRQ drops sharply for ReSIREN (), whose training set has the largest reduction rate, and for GeoCLIP (), while elevation declines for SINR (). By contrast, tasks aligned with administrative units benefit: population improves for every backbone and country for three of four. TextRQ thus inherits its geocoder’s granularity: it performs well where the signal changes between places, especially at the step-function transitions typically induced by administrative boundaries, but loses information where the signal varies within a place.
| Model (dataset) | Total points | Distinct points | Distinct reverse geocoded strings | Reduction rate (distinct points per string) |
| SatCLIP (S2-100K) | 91,858 | 91,858 | 21,188 | 4.3× |
| GeoCLIP (MP-16) | 4,723,695 | 2,875,487 | 81,997 | 35.1× |
| Climptilit (CHELSA) | 224,368,589 | 224,368,589 | 94,473 | 2,375.0× |
| SINR (iNat) | 35,500,262 | 27,862,308 | 101,411 | 274.7× |
Table 6. Input-space reduction from reverse geocoding. The reduction rate is the number of distinct points divided by the number of distinct reverse geocoded strings, i.e. how many distinct input locations collapse onto each text target.
References
- [1]Elijah Cole, Grant Van Horn, Christian Lange, Alexander Shepard, Patrick Leary, Pietro Perona, Scott Loarie, and Oisin Mac Aodha. Spatial implicit neural representations for global-scale species mapping. In International conference on machine learning, pp. 6320–6342. PMLR, 2023.arxiv.org/abs/2306.02564
- [2]Aayush Dhakal, Srikumar Sastry, Subash Khanal, Adeel Ahmad, Eric Xing, and Nathan Jacobs. Range: Retrieval augmented neural fields for multi-resolution geo-embeddings. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24680–24689. IEEE, 2025.arxiv.org/abs/2502.19781
- [4]Johannes Dollinger, Damien Robert, Elena Plekhanova, Lukas Drees, and Jan Dirk Wegner. Climplicit: Climatic implicit embeddings for global ecological tasks. arXiv preprint arXiv:2504.05089, 2025.arxiv.org/abs/2504.05089
- [5]Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873–12883, 2021.arxiv.org/abs/2012.09841
- [6]Nathan Godey, Éric de la Clergerie, and Benoît Sagot. On the scaling laws of geographical representation in language models. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 12416–12422, Torino, Italia, May 2024. ELRA and ICCL. URL https://aclanthology.org/2024.lrec-main.1087/.arxiv.org/abs/2402.19406
- [7]Google Maps Platform. Geocoding api documentation. https://developers.google.com/maps/documentation/geocoding/overview, 2026. Accessed: 2026-09-22.
- [8]Wes Gurnee and Max Tegmark. Language models represent space and time. In International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2310.02207.
- [9]Junlin He, Tong Nie, and Wei Ma. Geolocation representation from large language models are generic enhancers for spatio-temporal learning. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’25/IAAI’25/EAAI’25. AAAI Press, 2025. ISBN 978-1-57735-897-8. doi: 10.1609/aaai.v39i16.33879. URL https://doi.org/10.1609/aaai.v39i16.33879.
- [10]Josh Hooker, Gregory Duvviller, and Alessandro Cescatti. A global dataset of air temperature derived from satellite remote sensing and weather stations. Scientific data, 5(1):180246, 2018.
- [11]Batuhan Kilic, Müslüm Hacar, and Fatih Gülgen. Effects of reverse geocoding on openstreetmap tag quality assessment. Transactions in GIS, 27(5):1599–1613, 2023. doi: https://doi.org/10.1111/tgis.13089. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/tgis.13089.
- [12]Konstantin Klemmer, Esther Rolf, Caleb Robinson, Lester Mackey, and Marc Rußwurm. Satclip: Global, general-purpose location embeddings with satellite imagery. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 4347–4355, 2025.arxiv.org/abs/2311.17179
- [13]Ourania Kounadi, Thomas J. Lampoltshammer, Michael Leitner, and Thomas Heistracher. Accuracy and privacy aspects in free online reverse geocoding services. Cartography and Geographic Information Science, 40(2):140–153, 2013. doi: 10.1080/15230406.2013.777138. URL https://doi.org/10.1080/15230406.2013.777138.
- [14]Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In 2022 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp. 11513–11522. IEEE, 2022.
- [15]Baolei Li, Yiping Yuan, Yilin Zheng, Likang Yin, Ling Liu, Fabio Soldo, Romer Rosales, Xinyang Yi, and Lichan Hong. Tokens are all you need: Dual-purpose semantic ids for achieving llm-level i/o efficiency in recommendation systems. arXiv preprint arXiv:2607.24865, 2026.
- [16]Dongyi Lv, Qiuyu Ding, Heng-Da Xu, Zhaoxu Sun, Zhi Wang, Feng Xiong, and Mu Xu. Reasoning over space: Enabling geographic reasoning for LLM-based generative next POI recommendation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7322–7336, San Diego, California, United States, 2026. Association for Computational Linguistics. doi: 10.18653/v1/2026.acl-long.332. URL https://aclanthology.org/2026.acl-long.332/.DOI
- [17]Oisin Mac Aodha, Elijah Cole, and Pietro Perona. Presence-only geographical priors for fine-grained image classification. In 2019 IEEE/CVF international conference on computer vision (ICCV), pp. 9595–9605. IEEE, 2019.
- [18]Gengchen Mai, Yao Xuan, Weining Zuo, Yutong He, Jiaming Song, Stefano Ermon, Krzysztof Janowicz, and Ni Lao. Sphere2vec: A general-purpose location representation learning over a spherical surface for large-scale geospatial predictions. ISPRS Journal of Photogrammetry and Remote Sensing, 202:439–462, 2023. doi: 10.1016/j.isprsjprs.2023.06.016. URL https://arxiv.org/abs/2306.17624.
- [19]Rohin Manvi, Samar Khanna, Gengchen Girdhar, Stefano Ermon, and David Lobell. GeoIll: Extracting geospatial knowledge from large language models. International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2310.06213.
- [20]R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33 (3):291–297, 1997.DOI
- [21]Gustavo Penha, Edoardo D’Amico, Marco De Nadai, Enrico Palumbo, Alexandre Tamborrino, Ali Vardasbi, Max Lefarov, Shawn Lin, Timothy Heath, Francesco Fabbri, et al. Semantic ids for joint generative search and recommendation. In Proceedings of the Nineteenth ACM Conference on Recommender Systems, pp. 1296–1301, 2025.
- [22]Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Tran, Jonah Samost, et al. Recommender systems with generative retrieval. Advances in Neural Information Processing Systems, 36:10299–10315, 2023.arxiv.org/abs/2305.05065
- [23]Esther Rolf, Jonathan Proctor, Tamma Carleton, Ian Bolliger, Vaishaal Shankar, Miyabi Ishihara, Benjamin Recht, and Solomon Hsiang. A generalizable and accessible approach to machine learning with global satellite imagery. Nature communications, 12(1):4392, 2021.
- [24]Marc Rußwurm, Konstantin Klemmer, Esther Rolf, Robin Zbinden, and Devis Tuia. Geographic location encoding with spherical harmonics and sinusoidal representation networks. In International Conference on Learning Representations, volume 2024, pp. 1746–1759, 2024.
- [25]Samuel Stevens et al. Bioclip: A vision foundation model for the tree of life. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19412–19424, 2024. URL https://arxiv.org/abs/2311.18803.
- [26]Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.arxiv.org/abs/1711.00937