… protein Transformer models through the lens of attention. We show that attention: (1) captures
the folding structure of proteins… , a key functional component of proteins, and (3) focuses on …

… However, being largely black-box models and thus challenging to interpret, current protein
… the pre-trained embeddings, interpreting attention pattern heatmaps and saliency maps), the …

… in protein language modeling and their applications to downstream protein property … are
needed to encode strong biological priors into protein language models and to increase their …

… We expect future research on attention based models to offer more comprehensive analysis
of protein to protein interactions, thorough model interpretation of the semantic similarity at a …

… advantage of attention weights and hidden states of the model that are interpreted to extract
… The fine-tuned models demonstrated high accuracy in predicting hidden residues within the …

… , BERTology, a research program dedicated to interpreting the BERT language model, has
… the pre-trained embeddings, interpreting attention pattern heatmaps and saliency maps) can …

… However, existing models are often difficult to interpret and require additional information …
BERTology, intend to elucidate how BERT learns contextual information by analyzing attention, …

… the close relationship between Transformer attention and biological features. Following
this, [44] and [29] further studied the interpretability of the attention map as contact map. …

… We demonstrate that Transformer protein language models learn contacts in the self-attention
maps with state-of-the-art performance. We compare ESM-1b (Rives et al., 2020), a large-…

… attention to combine MSAs and protein language models. Although MSA-Transformer
showed average performance in our benchmarks, it was found to be successful on secondary …