How Different Model Architectures "See" the Chemical World

In its most fundamental sense, a model is a mathematical representation of a real-world phenomenon. While this principle applies across countless fields, from economics to meteorology, we focused our research on the critical domain of drug toxicity—specifically predicting mutagenicity using the AMES dataset.

In drug discovery and safety assessment, predicting whether a new compound might cause genetic mutations is a vital binary classification task. This testing is a cornerstone of preclinical studies and is a mandatory regulatory requirement to ensure safety before a drug can transition to clinical studies. Given that classical experimental laboratory testing is incredibly resource-intensive and costly, computational models provide an essential, efficient alternative for early-stage screening. However, no single model is exhaustive; rather, each model "sees" the chemical world from a different angle, capturing unique patterns based on its underlying architecture.

The Contenders: Diverse Architectures

To explore these different perspectives, we compared several architectures with distinct ways of "reading" molecular structures:

• Message Passing Neural Network (MPNN): A standard approach that encodes SMILES into graphs to capture atom-to-atom relationships. Internally, it uses GraphSAGE layers as the primary learning component to project atom features into embeddings, capturing the surrounding chemical context.
• Directed Message Passing Neural Network (DMPNN): This model uses a directed line graph to specifically encode the bonds between atoms, offering a different perspective on molecular connectivity.
• Junction Tree Neural Networks (JTNN): These represent compounds as a hierarchy of chemical substructures, focusing on how functional groups are assembled.
• Morgan-Multi-Layer Perceptron (MLP): Our baseline model. It uses standard Morgan features (fingerprints) as inputs to a dense network—a classic and easy-to-understand approach.
• AMESFormer: A sophisticated, pre-trained graph-transformer. Its encoder maps atoms, bonds, and topological paths into dense multi-dimensional embeddings, incorporating spatial, edge, and centrality encodings.

Performance: The Power of Fine-Tuning

Our analysis of Validation AUC scores reveals a clear hierarchy in performance. GraphSAGE (MPNN) achieved the highest score at 0.860, followed closely by DMPNN at 0.858.

A significant finding was the performance of AMESFormer. In its zero-shot state (using pre-trained weights without dataset-specific tuning), it had the lowest performance at 0.727. However, after fine-tuning on our specific AMES dataset, its performance jumped to 0.815. This suggests the original model was trained on a different data distribution and reinforces that tuning a model on your specific data is essential for maximizing accuracy. On the other hand, zero-shot models remain an important factor when labeled data is unavailable or limited.

Identifying Distinct Decision Logics

To understand why these models perform differently, we used Principal Component Analysis (PCA) on the predicted probabilities for each model.

The resulting PCA plot shows that different categories of models cluster together based on their underlying logic. For instance, MPNN, DMPNN, and JTNN occupy similar regions, while Morgan-MLP and the AMESFormer variants are distinct outliers. This confirms that different encoding and learning architectures lead to different "points of view." Because of these diverse decision logics, we often find uniquely correctly predicted cases per model, where one architecture succeeds in identifying a toxic compound that others miss.

A Deeper Look: Scaffolds and Structural Alerts

To truly understand where models succeed or fail, we analyzed performance across specific chemical subgroups:

1. Bemis-Murcko Scaffolds: The molecular backbone that remains after stripping away all atom labels, types, and non-ring side chains.
2. Structural Alerts: Specific functional groups, such as aromatic amines, alkyl halides, or epoxides, known in toxicology literature to trigger chemical reactivity and cell mutation.

Despite similarities for some scaffolds and alerts, we see model-specific changes in accuracy. For example, AMESFormer Zero-shot outperforms the rest in Nitrosamines, whereas the performance spread across all models for Alkyl Halide groups is significantly larger. Similarly, accuracy varies widely across different Bemis-Murcko scaffolds, with some models showing significant performance drops on specific complex backbones while others maintain stability.

Summary and Closing

Our exploration into AMES toxicity prediction highlights several key benefits:

• Domain-Specific Training is Vital: While zero-shot models offer immediate results without labeled data, training models on domain-specific data is necessary to bridge the gap in data distribution and improve accuracy.
• Diverse Perspectives: Different architectures "see" different patterns, often leading to uniquely correctly predicted cases for specific molecules.
• Architectural Synergy: Using a variety of architectures to approach a problem allows us to capture a broader range of chemical nuances, from basic backbones (scaffolds) to reactive functional groups (alerts).

By understanding these "different angles," we can build more robust, cost-effective computational pipelines to ensure chemical safety and support the critical transition from preclinical research to clinical trials.