AI Article Synopsis
- * Recent advancements involve large Transformers trained on unlabeled protein sequence data, demonstrating effective performance in contact prediction while utilizing attention mechanisms to capture complex interactions.
- * The introduction of an energy-based attention layer, called factored attention, reveals how Transformers can model hierarchical signals in protein data, suggesting potential for more sophisticated models beyond traditional approaches.
Article Abstract
The established approach to unsupervised protein contact prediction estimates coevolving positions using undirected graphical models. This approach trains a Potts model on a Multiple Sequence Alignment. Increasingly large Transformers are being pretrained on unlabeled, unaligned protein sequence databases and showing competitive performance on protein contact prediction. We argue that attention is a principled model of protein interactions, grounded in real properties of protein family data. We introduce an energy-based attention layer, factored attention, which, in a certain limit, recovers a Potts model, and use it to contrast Potts and Transformers. We show that the Transformer leverages hierarchical signal in protein family databases not captured by single-layer models. This raises the exciting possibility for the development of powerful structured models of protein family databases.
Download full-text PDF |
Source |
|---|---|
| http://www.ncbi.nlm.nih.gov/pmc/articles/PMC8752338 | PMC |
Publication Analysis
Top Keywords
Similar Publications
Want AI Summaries of new PubMed Abstracts delivered to your In-box?
Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!