In pre-trained transformer models (GPT), fine-tuning occurs in the Decoder. The decoder is responsible for generating the output text based on the representation created by the encoder. Like the encoder, the decoder is typically made up of multiple layers of multi-head self-attention and feed-forward neural networks.