deepspot-m

SUMMARY

What it does

DeepSpot-M is a multimodal foundation model that maps a 224x224 H&E histology tile to spatial gene expression in log1p-CPM. The output is virtual spatial transcriptomics: one value per queried gene per tile, laid out on the grid the tiles came from. A LoRA-adapted pathology foundation backbone (Midnight) tokenises the tile. A cross-attention gene decoder lets each gene query attend to the patch tokens, and a gene router hypernetwork builds gene-specific projections from frozen biological embeddings (Evo 2, Orthrus, ProtT5, scGPT, Apertus). Genes enter the model as queryable embeddings rather than fixed output slots, so the released model covers a ~19k protein-coding gene panel including genes unseen in training. The panel ships with the weights as `tokens.csv` and is exposed as `model.gene_names`; genes outside it cannot be queried in this release. Applied to TCGA, the model produced a virtual spatial transcriptomics atlas of 28,664 slides across 32 cancer types. The code is PolyForm Noncommercial 1.0.0 and the weights are CC-BY-NC-SA-4.0. Use it for noncommercial research and check both licences before redistributing outputs. Installation: `uv pip install deepspotm==1.0.0`. Version 1.0.0 targets Python 3.10 to 3.13 and pulls in PyTorch. The weights are gated: request access at https://huggingface.co/ratschlab/DeepSpotM and authenticate with `huggingface-cli login`. Quick start: `from deepspotm import DeepSpotM; model, image_processor = DeepSpotM.from_pretrained("ratschlab/DeepSpotM", source="scgpt"); vals = model.predict_genes(image_processor(pil_tile).unsqueeze(0), ["EPCAM", "CD3D"])`. Tiles must be 224x224 RGB at roughly 20x magnification (about 0.5 microns per pixel). The `source` parameter selects which frozen gene embedding the router builds projections from: `evo2`, `orthrus`, `prott5`, `scgpt`, or `apertus`. Whole slide workflow: extract 224x224 tiles on a grid with the `histolab` skill, keeping each tile's coordinates; process and stack tiles into batches with `torch.stack`; call `predict_genes` once per batch with the same gene list; concatenate the batches into a tiles-by-genes matrix and attach the coordinates. That matrix is the virtual spatial transcriptomics map for the slide, and it drops straight into `AnnData` for downstream spatial analysis. Common use cases: spatial expression maps for marker genes across a tumour section; transcriptome-wide prediction over a slide cohort with no matching assay run; querying any of the ~19k panel genes by symbol, including genes unseen in training; adding an expression channel to a morphology-only histology pipeline; building a slide-level cohort atlas, as done for TCGA. Detailed references: `references/api.md` and `references/whole_slide.md`. Primary sources: paper at https://doi.org/10.64898/2026.06.19.26356060 (medRxiv, posted 22 June 2026); code at https://github.com/ratschlab/DeepSpotM; weights at https://huggingface.co/ratschlab/DeepSpotM; PyPI at https://pypi.org/project/deepspotm/.