Plastid & Mitochondrial Graph-Pangenome Toolkit

MineGraph is a Docker-orchestrated pipeline for constructing, analysing, annotating, and visualising graph-pangenomes of plastid and mitochondrial genomes — from raw FASTA files to publication-ready phylogenetic trees and gene-arrow plots.

What it does:

• Auto-tunes PGGB parameters (percent identity, segment length) from your data
• Constructs a variation graph with PGGB / ODGI
• Classifies nodes into Cloud / Shell / Core with tunable thresholds
• Generates interactive and publication-quality plots + graph statistics (XLSX)
• Builds phylogenetic trees — graph-based UPGMA with Felsenstein or adaptive UFBoot-PAV bootstrap, or RAxML-NG MSA-based
• Injects gene annotations and produces gggenes gene-arrow visualisations
• Extracts subgraphs and inspects path composition
• Converts GFA graphs to FASTA or VG format
• Launches interactive visualisation via SequenceTubeMap
• Resumes interrupted runs from any step — including partial Step 5 sub-steps
• Benchmarks every pipeline step: wall time, peak CPU%, peak RAM

_{End-to-end MineGraph workflow. Click the figure for full resolution, or open the vector SVG.}

Every coloured track is a pipeline variant; stations are the tools run at each step; file icons mark the artifacts produced:

Stage 2 exposes the PGGB internals (wfmash → seqwish → smoothxg → odgi) and Stage 5 drops pink spurs from each stand-alone hub onto its sub-functions and outputs. A simpler overview diagram and the full source are in docs/pipeline_workflow.md.

Host machine
MineGraph.py              ← single entry point (pure host-side Python)
    ├── construct ──────────── Step 1: prepare_and_mash_input.py  (rakanhaib/opggb)
    │                          Step 2: RepeatMasker               (pegi3s/repeat_masker)
    │                          Step 3: run_repeatmask.py          (rakanhaib/opggb)
    │                          Step 4: run_pggb.py                (rakanhaib/opggb)        ← default
    │                               OR pggb directly              (--pggb-image, any image) ← optional
    │                          Step 5: run_stats.py               (rakanhaib/opggb)
    ├── extract ────────────── extract.py                         (rakanhaib/opggb)
    ├── inject  ────────────── inject.py + gggenes                (rakanhaib/opggb)
    ├── convert ────────────── gfa2fasta.py / vg convert          (rakanhaib/opggb)
    └── tubemap ────────────── SequenceTubeMap                    (rakanhaib/sequencetubemap)

No docker-in-docker. Each step is an independent docker run.

The metro map above shows how these containers, inputs, and outputs connect end-to-end.

MineGraph is fully dockerised. Pull the latest images before the first run so no step has to wait on a download:

docker pull rakanhaib/opggb:latest              # construct / extract / inject / convert / stats
docker pull rakanhaib/sequencetubemap:latest    # tubemap interactive viewer
docker pull pegi3s/repeat_masker:latest         # RepeatMasker step

Always re-pull rakanhaib/opggb:latest after updating MineGraph — the host scripts and the container toolchain are version-locked.

# ── Full pipeline (recommended) ─────────────────────────────────────────────
python MineGraph.py construct -- \
  --data_dir  ./plastid_fasta \
  --output_dir ./out \
  --metadata  ./samples.csv \
  --threads   32

# ── Resume an interrupted run ────────────────────────────────────────────────
python MineGraph.py construct -- \
  --data_dir ./plastid_fasta --output_dir ./out \
  --metadata ./samples.csv --threads 32 --resume

# ── Build graph only, skip statistics ────────────────────────────────────────
python MineGraph.py construct -- \
  --data_dir ./plastid_fasta --output_dir ./out \
  --metadata ./samples.csv --mode construct-graph

# ── Use clade-specific RepeatMasker database ─────────────────────────────────
python MineGraph.py construct -- \
  --data_dir ./plastid_fasta --output_dir ./out \
  --metadata ./samples.csv --species monocotyledons

# ── Use official PGGB image for graph construction (Step 4 only) ─────────────
python MineGraph.py --pggb-image ghcr.io/pangenome/pggb:latest construct -- \
  --data_dir ./plastid_fasta --output_dir ./out \
  --metadata ./samples.csv --threads 32


# ── Benchmark every pipeline step ────────────────────────────────────────────
python MineGraph.py --benchmark construct -- \
  --data_dir ./plastid_fasta --output_dir ./out --metadata ./samples.csv

# ── Extract a subgraph ───────────────────────────────────────────────────────
python MineGraph.py extract -- subgraph \
  -i out/pggb_output/graph.smooth.final.gfa \
  -w wanted_paths.txt -o subgraph.gfa

# ── Inject gene annotations and plot gene arrows ─────────────────────────────
python MineGraph.py inject \
  --graph graph.og --bed genes.bed \
  --genomes reps.txt --output ./inject_out --threads 8

# ── Convert to FASTA ─────────────────────────────────────────────────────────
python MineGraph.py convert fasta -i graph.gfa -o graph.fasta

# ── Visualise interactively ───────────────────────────────────────────────────
python MineGraph.py tubemap -i graph.gfa
# then open http://localhost:3210

python MineGraph.py construct -- [OPTIONS]

All options after -- are forwarded to the pipeline steps inside the container.

sample_A.fasta
sample_B.fasta
sample_C.fa

One FASTA filename per row, first column. Additional columns are ignored.

python MineGraph.py extract -- <subcommand> [OPTIONS]

python MineGraph.py extract -- subgraph \
  -i graph.gfa  -w wanted.txt  -o subgraph.gfa

wanted.txt — one path name per line. The first listed path is the extraction anchor.

python MineGraph.py extract -- stats -i graph.gfa -o stats.txt

python MineGraph.py extract -- paths -i graph.gfa -o paths.txt [--prefix Elymus_]

python MineGraph.py inject [OPTIONS]

Embeds gene annotations from a BED file into a pangenome graph as traversal paths, resolves their positions per representative genome using odgi untangle, and produces gene-arrow diagrams and bin-coverage heatmaps.

<output>/
├── inject_gene_arrows.png / .pdf   — gene-arrow diagram (gggenes)
├── inject_bin_coverage.png / .pdf  — per-path bin-coverage heatmap
├── inject_gene_presence.tsv        — gene copy counts per genome
├── untangle.tsv                    — raw odgi untangle positions
├── bin_coverage.tsv                — raw odgi bin coverage
└── inject_stats.txt                — graph statistics summary

python MineGraph.py convert fasta -i graph.gfa -o graph.fasta

All segment sequences are concatenated into a single FASTA record labelled <stem>#1.

python MineGraph.py convert vg -i graph.gfa -o graph.vg

Uses vg convert internally.

python MineGraph.py tubemap -i graph.gfa

Then open http://localhost:3210 in your browser. Press Ctrl+C to stop — the container is cleaned up automatically.

MineGraph can resume an interrupted run from any step — no recomputation of completed work.

python MineGraph.py construct -- \
  --data_dir ./data --output_dir ./out --metadata ./samples.csv \
  --threads 32 --resume

The host inspects --output_dir and determines the earliest incomplete step:

Intra-step 5 resume — when stats_partial is detected, the host checks which of the four Step 5 sub-steps have sentinel output files and skips only those that are already done:

MineGraph classifies each graph node by the fraction of haplotypes that contain it, controlled by --sc_th (default 5%):

Both methods are parallelised via joblib (loky backend) with mmap matrix sharing — all threads are used and no memory is duplicated.

Standard Felsenstein column-resampling bootstrap. Runs exactly --tree_bs replicates (default: 1000). Each replicate resamples columns with replacement and rebuilds the UPGMA tree. Support values are the percentage of replicates recovering each bipartition.

# 1000 Felsenstein bootstrap replicates with 32 threads
python MineGraph.py construct -- ... --tree_type graph \
  --bootstrap-method felsenstein --tree_bs 1000 --threads 32

Adapts UFBoot2 (Hoang et al. 2018, MBE 35:518) to PAV-Jaccard distances:

• RELL-weighted Jaccard via BLAS DGEMM ((X * m) @ X.T) — mathematically equivalent to explicit resampling but uses numpy's multi-threaded BLAS
• Convergence-based stopping: runs in batches; stops when Pearson r between consecutive support vectors ≥ --ufboot-convergence for 2 consecutive batches AND ≥ --ufboot-min-rep replicates are done

Typically converges in 1000–3000 replicates; capped at --ufboot-max-rep (default 10000).

python MineGraph.py construct -- ... --tree_type graph \
  --bootstrap-method ufboot \
  --ufboot-min-rep 1000 --ufboot-max-rep 10000 \
  --ufboot-convergence 0.99 --ufboot-batch 100 --threads 32

<output_dir>/
├── benchmark_report.json              # Step profiling (wall, CPU, RAM) — with --benchmark
├── params.yaml                        # Auto-tuned PGGB parameters
├── panSN_output.fasta.gz              # Merged, renamed, compressed FASTA
├── downsampled_panSN_output.fasta     # Subsampled FASTA used for mash / RepeatMasker
├── pggb_output/
│   ├── *.smooth.final.gfa             # Final variation graph
│   ├── *.smooth.final.vcf             # Variant calls
│   ├── *.smooth.maf                   # Multiple alignment format
│   ├── *.alignments.wfmash.paf        # Pairwise alignments
│   └── multiqc_report.html            # MultiQC summary
└── MineGraph_output/
    ├── statistics/
    │   ├── graph_stats.xlsx                 # Node/edge/diversity summary (Cloud/Shell/Core)
    │   └── graph_Node_Plot_frequency.*      # Histogram CSV + PNG
    ├── plots/
    │   ├── graph_top_N_interactive.html     # Interactive PyVis graph (top N nodes)
    │   ├── node_histogram_by_paths.png      # Cloud/Shell/Core bar chart
    │   ├── similarity.*                     # Window-based similarity heatmap
    │   └── paths.*                          # Path statistics heatmap
    ├── phylogenetics_msa/
    │   ├── Consensus_sequence.fasta
    │   ├── odgi_matrix.tsv                  # PAV presence/absence matrix
    │   ├── graph_phylo_tree.nwk             # Newick with bootstrap support (graph mode)
    │   ├── graph_phylo_tree_bootstrap_support.tsv
    │   ├── graph_phylo_tree.png             # Rendered tree figure
    │   └── MSA_result.fasta                 # (msa mode only)
    └── gfa_convert/                         # (with --convert-gfa)
        ├── gfa_to_vg.vg
        └── gfa_to_fasta.fasta

A minimal end-to-end example is shipped with the repository so every command in this README can be exercised without first building a multi-gigabyte graph.

Decompress the subgraph before use:

gunzip -k data/subgraph.gfa.gz

See data/README.md for provenance and ready-to-run demo commands.

Each sub-folder is a drop-in reference of what a successful run looks like on the 192-genome graph.

UPGMA tree built from the Jaccard distance matrix of the ODGI PAV matrix, with Felsenstein column-resampling bootstrap support. Raw Newick (graph_phylo_tree.nwk) and per-branch support (graph_phylo_tree_bootstrap_support.tsv) are both shipped.

Produced by projecting data/genes.bed onto representative chloroplast paths with odgi untangle and rendering the result with gggenes. gene_arrows_gggenes.tsv is the truncated intermediate table that feeds the plot.

Once data/subgraph.gfa.gz is decompressed, launch the interactive viewer:

python MineGraph.py tubemap -i data/subgraph.gfa
# then open http://localhost:3210

The reference rendering below is Figure 4 of the accompanying manuscript — the same Triticum × Elymus subgraph shipped in data/, captured from SequenceTubeMap and annotated for publication:

PGGB outputs are not bundled as examples — PGGB is already dockerised and self-contained. Pull rakanhaib/opggb:latest (see Pull the container images above) and MineGraph will invoke it for you.

All scripts that turn MineGraph outputs into the figures and tables of the manuscript live under analysis/, grouped by analysis (not by reviewer comment):

analysis/
├── phylogenetic_comparison/     # graph tree vs published phylogenies
├── displacement_analysis/       # per-taxon displacement + pairwise matrices
├── site_concordance/            # SCF on the MAF alignment
├── pav_linkage_disequilibrium/  # LD on PAV nodes + t-SNE sensitivity
├── compartment_provenance/      # LSC / IR / SSC BLAST partition
├── gene_pav/                    # gene-level PAV heatmaps
├── topology_analysis/           # path convergence, bubbles, symmetry
├── go_enrichment/               # topGO consensus vs non-consensus
├── family_node_enrichment/      # family-unique node enrichment
├── ward_clustering/             # Ward dendrogram with BS × IR overlay
├── pav_tsne/                    # t-SNE of PAV matrix
└── figure_composition/          # multi-panel figure assembly

Each folder is independently runnable once MineGraph has produced odgi_matrix.tsv, graph_phylo_tree.tree, and graph_stats.xlsx. See analysis/README.md for inputs, outputs, and dependency lists.

If you use MineGraph, please cite the tools it depends on:

Garrison E. et al. Building pangenome graphs. Nature Methods (2024). https://doi.org/10.1038/s41592-024-02430-3

Beyer W. et al. SequenceTubeMap: visualization for graph-based genomes. Bioinformatics (2019). https://doi.org/10.1093/bioinformatics/btz597

Vorbrugg S. et al. Gretl – Variation GRaph Evaluation TooLkit. bioRxiv (2024). https://doi.org/10.1101/2024.03.04.580974

Hoang D.T. et al. UFBoot2: Improving the Ultrafast Bootstrap Approximation. Molecular Biology and Evolution (2018). https://doi.org/10.1093/molbev/msx281

MineGraph — fast, reproducible, topology-native plastid and mitochondrial pangenome analysis.