This repository provides a script and recipe to run the highly optimized transformer-based encoder and decoder component, and it is tested and maintained by NVIDIA.
In NLP, encoder and decoder are two important components, with the transformer layer becoming a popular architecture for both components. FasterTransformer implements a highly optimized transformer layer for both the encoder and decoder for inference. On Volta and Turing GPUs, the computing power of Tensor Cores are used automatically when the precision of the data and weights are FP16.
In FasterTransformer 1.0, we implemented a highly optimized BERT transformer layer, which is used in the encoder.
In FasterTransformer 2.0, we have added a highly optimized decoder and decoding models based on OpenNMT-TF, an open-source library. Here, the decoder is the model that contains some transformer layers. On the other hand, decoding refers to the whole translating process, including the lookup embedding table, position encoding, a decoder and beam search.
The following graph demonstrates the model architecture.
FasterTransformer is built on top of CUDA and cuBLAS, providing the C++ API and TensorFlow OP. Users can integrate them into TensorFlow or other inference service codes that are built in native C++. We also provide some simple sample code to demonstrate how to use the encoder, decoder and to carry out decoding in C++ and TensorFlow.
### Configuration support matrix
The following configurations are supported in the FasterTransformer encoder.
- Batch size (B<sub>1</sub>): smaller or equal to 512
Note: For Encoder-Decoding structure, the sequence length of Encoder and Decoding must be the same.
### Model architecture
#### Encoder
The encoder requires the following inputs:
1. An input tensor. The shape is \[ B<sub>1</sub>, S, H x N\].
2. An attention mask.
3. The weights of all parameters.
The encoder will return the following outputs:
1. The encoder output feature. The shape is \[ B<sub>1</sub>, S, H x N \].
#### Decoder
The decoder requires the following inputs:
1. The features vector obtained by looking up the embedding table, or the previous result of the decoder. The shape is \[ B<sub>1</sub> x B<sub>2</sub>, 1, H x N \].
2. The output of the encoder.
3. The sequence length of the source sentence. Note that the lengths should be expanded by beam width times.
4. A memory cache space to store the K, V of masked multi-head attention. The size will grow for each step.
5. A memory cache space to store the K, V of cross attention. Since K, V is computed by the encoder result, we only compute them in the first step, storing them into the cache, and then reuse in the other steps.
6. The weights of all parameters.
7. In order to prevent the parallel computing of TensorFlow decoder and FasterTransformer Decoder, we put the TensorFlow result as a pseudo input in the TensorFlow OP. Otherwise, the results of FasterTransformer Decoder will incorrect. This input is useless for computing. Users can remove it when applying Decoder into a real application.
The decoder will return the following outputs:
1. Memory cache of masked multi-head attention.
2. Memory cache of cross attention.
3. The decoder output feature. The shape is \[ B<sub>1</sub> x B<sub>2</sub>, 1, H x N \].
#### Decoding
Decoding refers to the whole translating process, including position encoding, embedding lookup, and a simple beam search kernel.
Decoding requires the following inputs:
1. The output of the encoder. The shape is \[ B<sub>1</sub>, memory sequence length, H x N \].
2. The sequence length of the source sentence. Note that the lengths should be expanded by beam width times.
3. The table for embedding lookup. The shape is \[ V, H x N \].
4. The start id and end id for the vocabulary.
5. The weights of all parameters.
Decoding returns the following outputs:
1. The output ids. The shape is \[ B<sub>1</sub> x B<sub>2</sub> \].
2. The parent ids, which are the chosen beam ids.
3. The sequence lengths of each sentence.
Note that these results are required to be finalized by TensorFlow's `tf.contrib.seq2seq.gather_tree` or other progress.
#### Decoder and Decoding
Although the decoding process of most methods is similar, we find that there are lots of different kinds to compute the probability and implement the beam search. Therefore, if your chosen beam search algorithm is different from our implementation and it is hard for you to modify the beam search kernel, TensorFlow decoding with FasterTransformer Decoder is the recommended choice. However, the performance of the TensorFlow decoding with the FasterTransformer Decoder is worse than the performance of the FasterTransformer Decoding, especially for small batch sizes.
## Setup
The following section lists the requirements in order to use FasterTransformer.
- [TensorFlow 19.07-py2+](https://ngc.nvidia.com/catalog/containers/nvidia:tensorflow) NGC container
- [NVIDIA Pascal](https://www.nvidia.com/en-us/data-center/pascal-gpu-architecture/) or [Volta](https://www.nvidia.com/en-us/data-center/volta-gpu-architecture/) or [Turing](https://www.nvidia.com/en-us/geforce/turing/) based GPU
For more information about how to get started with NGC containers, see the following sections from the NVIDIA GPU Cloud Documentation and the Deep Learning Documentation:
- [Getting Started Using NVIDIA GPU Cloud](https://docs.nvidia.com/ngc/ngc-getting-started-guide/index.html)
- [Accessing And Pulling From The NGC Container Registry](https://docs.nvidia.com/deeplearning/frameworks/user-guide/index.html#accessing_registry)
For those unable to use the TensorFlow NGC container, to set up the required environment or create your own container, see the versioned [NVIDIA Container Support Matrix](https://docs.nvidia.com/deeplearning/frameworks/support-matrix/index.html).
## Quick Start Guide
The following section shows how to use FasterTransformer on the NGC container.
### Build the FasterTransformer
1. Run the container.
```bash
nvidia-docker run -ti nvcr.io/nvidia/tensorflow:19.07-py2 bash
The following script demonstrates how to integrate the FasterTransformer into a BERT model. This requires the repo of [BERT](https://github.com/google-research/bert).
c. Finetune the pretrained model on MRPC datasets. This takes some minutes. The accuracy would be better or worse because the MRPC dataset is very small.
I0403 08:55:07.423686 140260684429120 run_classifier.py:925] loss = 0.4561844
```
This shows that we use the FasterTransformer to run the inference successfully. In this case, using FP16 to do inference will reduce the accuracy with about 0.3%.
e. Compare the speed of BERT of TensorFlow and FasterTransformer under both FP32 and FP16.
b. Run the decoder in TensorFlow by running the following script:
```bash
python decoder_sample.py \
--batch_size 32 \
--beam_width 4 \
--max_seq_len 32 \
--head_number 8 \
--size_per_head 64 \
--memory_hidden_dim 768 \
--num_layer 6 \
--data_type fp32 \
--decoder_type 2
```
c. Run the decoding in TensorFlow by running the following script:
```bash
python decoding_sample.py \
--batch_size 32 \
--beam_width 4 \
--max_seq_len 32 \
--head_number 8 \
--size_per_head 64 \
--memory_hidden_dim 768 \
--num_layer 6 \
--data_type fp32
```
3. Run the encoder and decoding at the same time.
```bash
python encoder_decoding_sample.py \
--batch_size 32 \
--beam_width 4 \
--max_seq_len 32 \
--encoder_head_number 12 \
--encoder_size_per_head 64 \
--decoder_head_number 8 \
--decoder_size_per_head 64 \
--encoder_num_layer 6 \
--decoder_num_layer 6 \
--data_type fp32
```
## Advanced
The following sections provide greater details.
### Scripts and sample codes
The following code lists the directory structure of FasterTransformer:
```bash
/fastertransformer: source code of transformer
|--/cuda: some CUDA kernels and multi-head attention implementation, both are compiled with cuda/cuBLAS.
|--/tf_op: custom Tensorflow OP implementation
|--/trt_plugin: TensorRT plugin implementation
/sample: c++ and tensorflow transformer interface samples
|--/cpp: c++ interface samples
|--/tensorflow_bert: samples that show of how to integrate our Tensorflow OP into the open source BERT model for sentence (and sentence-pair) classification tasks (GLUE), the samples support both FP16 and FP32, see readme file within this folder more details
|--/tensorflow:TensorFlow OP samples
|--/tensorRT: both FP16 and FP32 tensorRT plugin samples
/tools/gemm_test: loop over all GEMM algorithms to pick the best one
```
In the root directory of FasterTransformer, the most important directories are:
*`fastertransformer/`
*`sample/`
*`tools/`
The `fastertransformer/` folder encapsulates all the source codes of FasterTransformer:
*`tf_op/` - Contains the TensorFlow Op source files of encoder, decoder and decoding
*`cuda/` - Contains all CUDA kernels of FasterTransformer
*`bert_encoder_transformer.h` - Contains the encoder transformer layer
*`open_decoder.h` - Contains the decoder transformer layer
*`beam_search_opennmt.h` - Contains the beam search progress for decoding
*`decoding_opennmt.h` - Contains the decoding progress
The `tools/` folder contains the tools to generate the GEMM configuration of FasterTransformer for different settings:
Assume the settings of the encoder are as follows:
-`batch_size`=1
-`sequence_length`=32
-`head_number`=12
-`size_per_head`=64
-`data_type`=FP32
Then the following scripts can generate the best GEMM configuration under such settings, and record the configuration into the `gemm_config.in.in` file.
```bash
./bin/encoder_gemm 1 32 12 64 0
```
2. Run the encoder.
Assume the settings are the same as above, and the encoder contains 12 transformer layers.
a. Run the encoder in C++ by running the following scripts:
`./bin/encoder_sample` runs the encoder in the `cpp`. The arguments of `encoder_sample` is:
Then the following scripts can run the encoder under the above settings.
```bash
./bin/encoder_sample 1 12 32 12 64 0
```
The outputs should be similar to the following:
```bash
Device Tesla T4
before allocate free 14.65 GB total 14.76 GB
After allocate free 14.61 GB used 0.14 GB total 14.76 GB
[batch_size 1 seq_len 32 12 transformer layers] costs 3.08 ms
```
b. Run the encoder in TensorFlow by running the following scripts:
The following script demonstrates the cross check between the encoder of TensorFlow and the encoder of FasterTransformer, and the execution time of them.
```bash
python encoder_sample.py \
--batch_size 1 \
--seq_len 32 \
--head_number 12 \
--size_per_head 64 \
--num_layer 12 \
--data_type fp32 \
--test_time 1
```
The outputs should be similar to the following:
```bash
[INFO] Encoder Cross check True
[INFO] Max diff 3.57627868652e-06
[INFO] min diff 0.0
[INFO] TF decoder time costs: 6.63149 ms
[INFO] OP decoder time costs: 4.64135 ms
```
c. Run the encoder in FP16:
Note that the configuration of FP32 and FP16 are different, so it is necessary to generate the configuration again.
For C, users only need to set the `<is_use_fp16>` flag as 1.
For TensorFlow, users can use the arguments `--data_type fp16` to change the computing mode.
```bash
./bin/encoder_gemm 1 32 12 64 1
./bin/encoder_sample 1 12 32 12 64 1
python encoder_sample.py \
--batch_size 1 \
--seq_len 32 \
--head_number 12 \
--size_per_head 64 \
--num_layer 12 \
--data_type fp16 \
--test_time 1
```
#### Decoder and decoding process
1. Generate the `decoding_gemm_config.in` file.
`./bin/decoding_gemm` can generate the best GEMM configuration. The arguments of `decoding_gemm` are:
Then the following scripts can generate the best GEMM configuration under such settings, and record the configuration into the `decoding_gemm_config.in` file.
```bash
./bin/decoding_gemm 32 4 8 64 30000 32 768 0
```
2. Run the decoder and decoding.
Assume the settings are the same as above, and the decoder contains 6 transformer layers.
a. Run the decoding in C++ by running the following script:
`./bin/decoding_sample` runs the decoding in the `cpp`. The arguments of `encoder_sample` is:
The results show that the differences between the decoder of TensorFlow and decoder are smaller than threshold. Note that the differences are absolute differences, so the differences may be large when the op val is large. In this case, the differences are larger than the threshold and the checking will return "False", but it may be not affect the final results.
The option `decoder_type` decides to use the decoder of TensorFlow or decoder of FasterTransformer. `decoder_type 2` uses both decoders and compares their results.
The following script demonstrates the execution time of the FasterTransformer decoder.
```bash
python decoder_sample.py \
--batch_size 32 \
--beam_width 4 \
--max_seq_len 32 \
--head_number 8 \
--size_per_head 64 \
--memory_hidden_dim 768 \
--num_layer 6 \
--data_type fp32 \
--decoder_type 1 \
--test_time 1
```
The outputs should be similar to the following:
```bash
[INFO] time costs of OP decoder: 248.046 ms.
```
The following script demonstrates the execution time of the TensorFlow decoder.
```bash
python decoder_sample.py \
--batch_size 32 \
--beam_width 4 \
--max_seq_len 32 \
--head_number 8 \
--size_per_head 64 \
--memory_hidden_dim 768 \
--num_layer 6 \
--data_type fp32 \
--decoder_type 0 \
--test_time 1
```
c. Run the decoding in TensorFlow by running the following script:
This subsection demonstrates how to use FasterTansformer decoding to translate a sentence. We use the pretrained model and testing data in [OpenNMT-tf](https://opennmt.net/Models-tf/), which translate from English to German.
Because the FasterTransformer Encoder is based on BERT, we cannot restore the model of encoder of OpenNMT to FasterTransformer Encoder. Therefore, we use OpenNMT-tf to build the encoder and preprocess the source sentence.
Another problem is that the implementation of FasterTransformer Decoder and decoder of OpenNMT-tf is a little different. For example, the decoder of OpenNMT-tf uses one convolution to compute query, key and value in masked-multihead-attention; but FasterTransformer Decoder splits them into three gemms. The tool `utils/dump_model.py` will convert the pretrained model to fit the model structure of FasterTransformer Decoder.
- batch_size should be smaller or equal to 1024 in Decoder.
- batch_size x beam_width should be smaller or equal to 1024 in Decoding.
- Results of TensorFlow and OP would be different in decoding. This problem is caused by the accumulated log probability, and we do not avoid this problem.
