xiangjx/MuPaD-256

Usage

Please run demo.py for full demonstrations. This repo generates 256x256 images.

Load Model

from diffusers import DiffusionPipeline
import torch

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

pipeline = DiffusionPipeline.from_pretrained(
    "xiangjx/MuPaD-256",
    custom_pipeline="xiangjx/MuPaD-256",
    trust_remote_code=True,
)
pipeline.to(device)

Text-to-Image Generation

Generate histopathology images from a text prompt.

# Text-to-Image generation
prompt = "Invasive breast carcinoma with poorly formed glands, hyperchromatic nuclei, and dense fibrous stroma"

output_t2i = pipeline(
    prompt=prompt,
    modality="text",
    num_images_per_prompt=4,
    num_inference_steps=250,
    guidance_scale=2.5,
    guidance_high=0.75,
    guidance_low=0.0,
    mode="sde",
    path_type="linear",
    seed=42
)

for i, img in enumerate(output_t2i["images"]):
    img.save(f"text2image_{i}.png")

Image-to-Image Generation

Generate images conditioned on a reference image.

from PIL import Image

# Load reference image
# Ensure you have a reference image path
raw_image = Image.open("test_image.png").convert("RGB")

output_i2i = pipeline(
    image=raw_image,
    modality="image",
    num_images_per_prompt=4,
    num_inference_steps=250,
    guidance_scale=2.5,
    guidance_high=0.75,
    guidance_low=0.0,
    mode="sde",
    path_type="linear",
    seed=42
)

for i, img in enumerate(output_i2i["images"]):
    img.save(f"image2image_{i}.png")

RNA-to-Image Generation

The RNA ("st") modality is conditioned on two parts, which must be supplied together:

Part	Shape	What it is
values	(331,)	Pathway scores, in the order defined by the SurvPath combine_signatures.csv signatures. Projected internally by st_value_proj.
meta	(100, 1024)	Metadata description encoded with MUSK. Use pipeline.encode_st_meta(...).

Both are required — the model rejects RNA conditioning without exactly 331 value tokens and 100 metadata tokens.

import torch

# 331 pathway scores derived from your RNA-seq sample (log1p-mean expression per
# pathway, ordered to match the SurvPath signature list used in training).
pathway_scores = torch.load("my_pathway_scores.pt")  # shape (331,)

# 100 metadata tokens describing the sample.
meta = pipeline.encode_st_meta(
    "Glioblastoma multiforme — the most aggressive adult brain tumour, "
    "studied for signaling pathways and therapeutic resistance."
)[0]  # shape (100, 1024)

output_rna = pipeline(
    embeddings={"values": pathway_scores, "meta": meta},
    modality="st",
    num_images_per_prompt=4,
    num_inference_steps=250,
    guidance_scale=2.5,
    guidance_high=0.75,
    guidance_low=0.0,
    mode="sde",
    path_type="linear",
    seed=42
)

for i, img in enumerate(output_rna["images"]):
    img.save(f"rna2image_{i}.png")

To generate several samples at once, pass a list of dicts as embeddings.

Software Dependencies

• torch>=2.0.0
• diffusers>=0.35.1
• timm>=0.9.0
• pillow
• huggingface-hub
• dictdot
• einops
• fairscale
• transformers==4.57.3
• sentencepiece