So in this article, I want to try to break this module apart and look at how it works. When I read this part of the paper, it raised some questions in my head, which unfortunately the author had not provided sufficient information to answer them.

In this article, I don’t plan to explain its architecture in depth as there are currently several great tutorials on this topic ( here,Īnd here), but alternatively, I want to discuss one specific part of the transformer’s architecture - the positional encoding.

Even though making it more accessible is a great thing, but on the downside it may cause the details of the model to be ignored. Thanks to the several implementations in common deep learning frameworks, it became an easy option to experiment with for many students (including myself). Its ability for parallelizable training and its general performance improvement made it a popular option among NLP (and recently CV) researchers. Transformer architecture was introduced as a novel pure attention-only sequence-to-sequence architecture by Vaswani et al.

Encoding in a sentence