Benchmarks¶

GIAE is benchmarked against Bakta on three reference phage genomes. Same FASTA, same scoring, same overlap criterion. The numbers below are reproducible from any clone of this repo.

Headline result¶

Genome	GIAE F1	Bakta F1	Δ	Verdict
phiX174	60.0 %	60.0 %	—	tied
λ phage	79.2 %	72.6 %	+6.6 %	GIAE wins
T7	88.1 %	85.2 %	+2.9 %	GIAE wins

GIAE matches or beats Bakta on every genome without using any of Bakta's databases — no UniProt, no Pfam, no AMRFinderPlus, no ncRNA-region search. Just pyrodigal + GIAE's rescue and nested-finder layers.

Per-genome breakdown¶

phiX174 — 5.4 kb, 11–13 reference genes¶

Tool	Pred	TP	FP	FN	Precision	Recall	F1
GIAE	7	6	1	7	85.7 %	46.2 %	60.0 %
Bakta	7	6	1	7	85.7 %	46.2 %	60.0 %

Both tools tie at 60 %. The remaining seven false negatives are nested / overlapping genes (A*, B, E, K) that PhiX174 encodes via translational coupling, not canonical Shine-Dalgarno signals — a documented biological feature of compact phages that defeats sequence- only ORF detection. Beating phiX174 will require either translational- coupling detection or structural homology (Foldseek / AlphaFold).

λ phage — 48.5 kb, 96 reference CDSs¶

Tool	Pred	TP	FP	FN	Precision	Recall	F1
GIAE	63	63	0	33	100.0 %	65.6 %	79.2 %
Bakta	61	57	4	39	93.4 %	59.4 %	72.6 %

GIAE's +6.6 % F1 advantage comes from zero false positives — every prediction GIAE makes on λ matches a real annotated gene. Bakta's expert-protein-database search produces four spurious hits.

T7 — 39.9 kb, 65 reference CDSs¶

Tool	Pred	TP	FP	FN	Precision	Recall	F1
GIAE	53	52	1	13	98.1 %	80.0 %	88.1 %
Bakta	50	49	1	16	98.0 %	75.4 %	85.2 %

GIAE recovers three more true positives than Bakta on T7 thanks to the Phase 7 nested-ORF detection — at zero cost in precision.

What does GIAE actually do?¶

The benchmarked configuration:

Interpreter(
    use_uniprot=False,        # no online UniProt
    use_interpro=False,       # no online InterPro
    use_local_blast=False,    # no BLAST DB
    use_diamond=False,        # no Diamond DB
    use_hmmer=False,          # no Pfam HMMER
    use_esm=False,            # no protein language model
    use_aragorn=False,        # no tRNA detection
    use_barrnap=False,        # no rRNA detection
    use_cache=False,
    use_rescue=True,          # Phase 5 short-ORF rescue
    phage_mode=True,          # Phase 7 nested ORF detection
)

So: pyrodigal + RBS / codon-usage rescue + phage-aware nested ORF detection — and that's it. No database lookups, no API calls.

What does Bakta actually do?¶

The benchmarked Bakta configuration uses the light database (~400 MB) and all available CDS subsystems active:

bakta \
  --db ~/.bakta_db/db-light \
  --skip-trna     \  # tRNAscan-SE not installed
  --skip-tmrna    \  # same
  --skip-rrna     \  # barrnap optional
  --skip-ncrna    \  # cmscan / Infernal not installed
  --skip-ncrna-region \
  --skip-crispr   \  # PILER-CR not installed
  --skip-pseudo   \
  --skip-sorf     \  # small ORF expert search
  --skip-gap      \
  --skip-ori      \  # blastn not installed
  --skip-plot     \
  --threads 4 \
  --complete \
  --min-contig-length 1

The CDS pipeline runs (pyrodigal + Diamond against the bundled expert-protein database) — that's the workhorse. We only skip non-CDS subsystems whose external tools weren't installed.

The AMRFinderPlus expert-CDS step was patched out at runtime (the binary isn't available outside conda/bioconda).

Methodology¶

Ground truth¶

Each genome's GenBank file as published by NCBI.
The CDS features (genes only — tRNA / rRNA / source excluded) define the truth set.
Truth coordinates are 0-based, half-open (Python convention).

Prediction¶

Both tools receive the same FASTA sequence (extracted from the GenBank file with annotations stripped).
Each tool runs its full CDS pipeline.
Predictions are extracted as (start, end, strand) tuples from the tool's output (Bakta's GFF3, GIAE's Genome.genes list).

Matching¶

A prediction matches a truth gene if:

Same strand, AND
Reciprocal coordinate overlap ≥ 50 % — the intersection length is at least half of the longer of the two spans.

Reciprocal-overlap is more robust than one-sided overlap for genes of varying length.

Scoring¶

TP — predictions that match a unique truth gene
FP — predictions that don't match any truth gene
FN — truth genes that aren't matched by any prediction
Precision = TP / (TP + FP)
Recall = TP / (TP + FN)
F1 = 2 · P · R / (P + R)

Reproducing these numbers¶

# 1. Set up GIAE
git clone https://github.com/Ayo-Cyber/GIAE.git
cd GIAE
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,annotation]"

# 2. Install Bakta + light db (~400 MB download)
pip install bakta
brew install diamond                              # or apt install diamond-aligner
bakta_db download --type light --output ~/.bakta_db

# 3. Run the comparison
.venv/bin/python post_assets/bakta_comparison.py

Expected output (~2 minutes total on a modern laptop):

Genome             Tool      Truth   Pred    TP    FP    FN  Precision  Recall      F1
-----------------------------------------------------------------------------------
phiX174            GIAE         13      7     6     1     7     85.7%   46.2%   60.0%
phiX174            Bakta        13      7     6     1     7     85.7%   46.2%   60.0%
λ phage            GIAE         96     63    63     0    33    100.0%   65.6%   79.2%
λ phage            Bakta        96     61    57     4    39     93.4%   59.4%   72.6%
T7                 GIAE         65     53    52     1    13     98.1%   80.0%   88.1%
T7                 Bakta        65     50    49     1    16     98.0%   75.4%   85.2%
-----------------------------------------------------------------------------------

Δ F1 (GIAE − Bakta):
  phiX174            = +0.0%
  λ phage            ▲ +6.6%
  T7                 ▲ +2.9%

The script (post_assets/bakta_comparison.py) is short and readable — fork it, change the genomes, rerun.

GIAE-only validation suite¶

A second script tests GIAE in isolation across three modes (pyrodigal only, +rescue, +rescue +phage_mode):

.venv/bin/python post_assets/phase4_validation.py

── pyrodigal only ───────────────────────────────────────────────────────
phiX174   13   7   6  1  7   85.7%  46.2%  60.0%
λ phage   96  62  62  0 34  100.0%  64.6%  78.5%
T7        65  50  49  1 16   98.0%  75.4%  85.2%

── + rescue pass ─────────────────────────────────────────────────────────
phiX174   13   7   6  1  7   85.7%  46.2%  60.0%
λ phage   96  63  63  0 33  100.0%  65.6%  79.2%
T7        65  51  50  1 15   98.0%  76.9%  86.2%

── + rescue + phage_mode (nested ORFs) ──────────────────────────────────
phiX174   13   7   6  1  7   85.7%  46.2%  60.0%
λ phage   96  63  63  0 33  100.0%  65.6%  79.2%
T7        65  53  52  1 13   98.1%  80.0%  88.1%

The progression shows each layer's contribution. Rescue gives lambda +0.7 % F1 with zero new false positives. phage_mode gives T7 +1.9 % F1, also with zero new false positives.

Limits & caveats¶

What this benchmark doesn't measure¶

Functional annotation accuracy. F1 measures whether GIAE found the gene at the right coordinates, not whether it correctly named the protein. Function calling needs UniProt / Pfam / Diamond — the benchmark above runs GIAE without those to isolate the ORF-finding capability.
Bacterial genomes. Currently validated on three phage genomes. Bacterial-scale validation (4–6 Mb) is on the roadmap.
All-pairs comparison. We only compare against Bakta. Future rounds should include Prokka, RAST, and direct prodigal output.
PhiX174's hard floor. Both tools tie at 60 % on PhiX174 because its overlapping genes use translational coupling. This is a biological limit, not a tool limit.

What this benchmark does show¶

The combination of pyrodigal + ShortOrfRescue + NestedOrfFinder beats Bakta's pyrodigal + expert-database approach on the phages it can be tested on.
GIAE's rescue and nested-finder layers add real true positives at zero cost in precision when properly tuned.
Annotation quality and ORF-finding quality are separable. GIAE's ORF-finding is competitive even with no functional databases at all.