Master's thesis · M.Tech Biotechnology · 2025–26
Benchmarking AI Tools for Mutation‑Aware PIK3CA Recognition
Tested co-folding, docking and affinity-prediction tools on 300 pairs of PIK3CA variants and inhibitors. They rank known inhibitors well. Every tool that builds a pose misses the allosteric pocket that the newest mutant-selective drugs bind.
Abstract
Generative and deep-learning tools for protein–ligand modelling are increasingly used, but it is unclear whether they can reliably detect structural changes caused by clinical mutations and how an inhibitor engages its target. This study benchmarks co-folding, docking and affinity-rescoring tools on PIK3CA, the catalytic p110α subunit of PI3Kα and one of the most frequently mutated oncogenes in human cancer. A 300-pair cross-docking matrix was built from fourteen clinical p110α variants and the wild type, with twenty ligands spanning FDA-approved inhibitors, clinical candidates, allosteric and isoform-selective compounds, and inactive controls. Performance was assessed on four independent axes: experimental affinity ranking, cross-docked specificity, mutation-driven affinity change (Δp), and pose correctness against crystallographic references.
GatorAffinity rescoring on Boltz-2 poses gave the best affinity ranking (Spearman ρ = 0.85) and the best separation of binders from controls (AUROC = 0.88). CheapNet (Δp ρ = 0.75) and GatorAffinity (Δp ρ = 0.67) on Boltz-2 poses both signed all five significant mutational shifts correctly, so current models partly capture mutation-dependent changes (preliminary). Performance depended heavily on the pose source, and some generative models produced chemically invalid geometries.
Most critically, every pose tool placed the allosteric inhibitors in or beside the orthosteric ATP pocket (LRMSD > 19 Å), often with high confidence, mirroring the near-absence of allosteric PIK3CA structures in training data. Mutation-aware affinity is partly captured; allosteric pocket recognition remains an open problem.
The question
PIK3CA encodes p110α, the catalytic subunit of PI3Kα, and is one of the most frequently mutated oncogenes in human cancer. Most clinical mutations sit in two hotspots: the helical domain (E542K, E545K) and the kinase domain (H1047R). The first PI3Kα drugs, such as alpelisib, bind the ATP pocket and hit wild-type and mutant enzyme alike. Newer inhibitors try to tell them apart. Inavolisib binds the ATP pocket and preferentially degrades the mutant protein. RLY-2608 and STX-478 bind a separate allosteric pocket and favour the mutant enzyme.
Structure-based AI tools are now used to triage compounds like these. For them to help here, they have to do four things:
- Rank inhibitors by measured potency.
- Separate real binders from compounds that do not bind p110α.
- Shift the prediction in the right direction when a mutation changes potency.
- Place the ligand in the right pocket.
The benchmark
The matrix crosses 15 p110α sequences (wild type and 14 clinical variants) with 20 ligands: approved and clinical PI3Kα inhibitors, the two allosteric compounds, pan-PI3K and isoform-selective inhibitors, and drugs for unrelated targets as negative controls. Every one of the 300 pairs went through every tool.
Labels came from literature mining. 52 pairs have binding evidence, 19 of them with a measured IC₅₀ or Kᵢ. These 19 are used for ranking. 150 pairs are negative controls. The remaining pairs had no usable evidence and were left out of scoring.
Structures came from co-folding (Boltz-2, Protenix) and docking (DiffDock-L, FlowDock, GNINA, Vina, TankBind). Affinity rescorers (GatorAffinity, CheapNet, GIGN, DualBind) were run on both Boltz-2 and DiffDock-L poses. PLAPT predicts affinity from sequence and SMILES alone, with no structure.
The ligands are chemically diverse. Their mean pairwise Tanimoto similarity is 0.13, and only one pair, duvelisib and idelalisib, is above 0.4 (0.53). A tool cannot score well here by recognising near-copies of the same scaffold.
Pairwise Tanimoto similarity
Ligands in similarity space
Ranking affinities
Co-folding followed by rescoring ranked the 19 measured affinities well. GatorAffinity on Boltz-2 poses reached Spearman ρ = 0.85 (95% CI 0.63–0.93), and Boltz-2’s own affinity head reached 0.80. Nesso-1, which Recursion and Valence Labs describe as “a fast, structure-based protein–ligand binding-affinity model”, reached 0.83 and gave the best binder-versus-control separation (AUROC 0.93). It was run after submission.
The pose source mattered as much as the scorer. The same GatorAffinity model dropped to ρ = 0.62 on DiffDock-L poses. Docking scores (GNINA, Vina) and PLAPT were weaker still.
There is a catch. If every pair’s Boltz-2 score is replaced by that ligand’s average over all 15 variants, ρ rises to 0.83, above the per-pair score. For GatorAffinity the same baseline gives 0.79. Most of the ranking signal says which ligand it is, not which variant it is binding.
A right pose is not a right answer
Idelalisib is a PI3Kδ drug and weakly active on PI3Kα, so the benchmark uses it as a control. Boltz-2 placed it in the PI3Kα ATP pocket in almost exactly the pose idelalisib takes in its PI3Kδ crystal structure (4XE0): 0.5 Å after fitting the two pockets. It gave that pose a confidence of 0.92 and predicted strong binding (p-scale 6.59). DiffDock-L also used the pocket, 3.4 Å off the PI3Kδ pose, but with a low confidence of −1.43. A plausible pose in a conserved pocket says little about whether the drug binds this isoform.
Mutation response
The harder test is whether a tool’s prediction moves when a mutation changes potency. Only 9 mutant–wild-type pairs have measured shifts (Δp), so this axis is underpowered. CheapNet on Boltz-2 poses gave ρ = 0.75 (permutation p = 0.03) and GatorAffinity 0.67 (p = 0.06).
All five shifts larger than 0.3 log units are potency gains. A constant “mutant binds tighter” guess also gets all five signs right, so a correct sign alone shows nothing. I treat this result as preliminary.
The wrong pocket
Six crystal structures gave a direct check of pose. The three orthosteric ligands are alpelisib (7PG6), inavolisib (8AM0) and taselisib (8EXL). The allosteric ones are RLY-2608 (8TSD) and STX-478 on wild type (8TDU) and H1047R (8TGD). Boltz-2, Protenix and DiffDock-L placed the orthosteric ligands within 2 Å. FlowDock’s top poses landed 8–10 Å away: near the right pocket, but not in the crystal pose.
No tool recovered the allosteric pocket. All twelve allosteric poses were 20–27 Å from the crystal pose, and I checked where they went instead. Eleven sit in the ATP pocket: their centre is within 5 Å of where alpelisib binds, and they come within about 4 Å of the hinge residue Val851. FlowDock’s RLY-2608 pose sits at the edge of that pocket, 6.6 Å out. Boltz-2 reported its RLY-2608 pose with a confidence of 0.91. Allosteric PIK3CA structures are almost absent from the training data, and the models fall back on the pocket they have seen thousands of times.
Nesso-1 and PLAPT predict affinity without building an atomic pose, so they have nothing to check here.
Watching FlowDock place a ligand
FlowDock builds a pose by flowing the ligand and pocket from a random start to a final structure. These trajectories come from a separate sampling run with FlowDock’s default predicted apo structure as the starting protein. In that run inavolisib was FlowDock’s one good pose: it settles within about 1 Å of the crystal ligand (1.6 Å LRMSD), the only FlowDock pose under 2 Å. Alpelisib’s centre stays about 3 Å out. RLY-2608 goes straight to the ATP pocket and stays there.
Guidance fixes geometry, not placement after thesis
Protenix often produced poses that fail PoseBusters checks: atoms overlapping the protein, or ligands too close to it. Protenix v2 has an optional training-free guidance (TFG) step that steers sampling toward physically valid geometry during diffusion. After submission I reran the six crystal pairs with and without TFG, each with five seeds.
TFG fixed the geometry. The top pose passed every PoseBusters check in 30 of 30 runs, against 15 of 30 without it. It did not move any ligand. Orthosteric poses stayed within about 2 Å either way, and the allosteric poses stayed 20–27 Å off. Across all 15 variants, 42 guided allosteric poses passed PoseBusters and none came closer than 18.7 Å. Each job took 23% longer: 256 s instead of 209 s on one NVIDIA L40.
What it means
- Ranking known PI3Kα ligands works, using co-folding plus rescoring, but most of the signal is ligand identity. A good ρ here is not evidence that the model reads the mutation.
- Mutation effects are unresolved. Nine measured shifts, all in one direction, cannot separate a model that understands the mutation from one that guesses “tighter”.
- Pocket blindness with overconfidence. No tool finds the allosteric pocket; every pose lands in or at the ATP pocket, sometimes at high confidence. For an allosteric series, check where the ligand landed before trusting any affinity number built on that pose.
- Physical validity is cheap to fix and is not the bottleneck. Guidance cleans up geometry for about a quarter more compute. It does not fix placement.
Thesis, corrections and reruns
In November 2026 every reported result was recomputed from the raw tool outputs. The tool runs and figure values reproduced. A few numbers in the text did not, and they are corrected in the PDF above. The main ones:
- Mutation response. The text gave ρ = 0.90 for GatorAffinity. The data give 0.67, and the best tool is CheapNet on Boltz-2 poses at 0.75.
- Protenix PoseBusters rate. The text gave a 94.7% failure rate. The correct figure is 35.5% of top poses failing at least one check, against 3.5% for Boltz-2.
- FlowDock was missing from the pose figure. It is included in the charts above.
- Idelalisib is weakly active on PI3Kα, not inactive, and the taselisib potency comes from Ndubaku et al. (2013).
- Serabelisib. The Boltz-2 affinity runs for serabelisib had used the wrong molecule. After correction Boltz-2 gives ρ = 0.80 (was 0.79).
Two sets of runs postdate the thesis and are tagged after thesis on this page: Nesso-1 on all 300 pairs, and Protenix v2 with and without guidance across five seeds.