- Pre-training and Finetuning on BioMedical tasks and corpus - Disabling Grappler Optimizations for improved performance
6 KiB
# Licensed under the Apache License, Version 2.0 (the "License")
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and limitations under the License.
Table Of Contents
- BERT Question Answering Fine-Tuning and Inference with Mixed Precision
- BioBERT Named-Entity Recognition Inference with Mixed Precision
BERT Question Answering Inference/Fine-Tuning with Mixed Precision
1. Overview
Bidirectional Embedding Representations from Transformers (BERT), is a method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks.
The original paper can be found here: https://arxiv.org/abs/1810.04805.
NVIDIA's BERT 19.10 is an optimized version of Google's official implementation, leveraging mixed precision arithmetic and tensor cores on V100 GPUS for faster training times while maintaining target accuracy.
1.a Learning objectives
This repository contains multiple notebooks which demonstrate:
- Inference on QA task with BERT Large model
- The use/download of pretrained NVIDIA BERT models
- Fine-Tuning on SQuaD 2.0 Dataset
- Use of Mixed Precision for Inference and Fine-Tuning
Here is a short description of each relevant file:
- bert_squad_tf_inference.ipynb : BERT Q&A Inference with TF Checkpoint model
- bert_squad_tf_finetuning.ipynb : BERT Fine-Tuning on SQuaD dataset
2. Quick Start Guide
2.a Build the BERT TensorFlow NGC container:
To run the notebook you first need to build the Bert TensorFlow container using the following command from the main directory of this repository:
docker build . --rm -t bert
2.b Dataset
We need to download the vocabulary and the bert_config files:
python3 /workspace/bert/data/bertPrep.py --action download --dataset google_pretrained_weights # Includes vocab
This is only needed during fine-tuning in order to download the Squad dataset:
python3 /workspace/bert/data/bertPrep.py --action download --dataset squad
2.c Start of the NGC container to run inference:
Once the image is built, you need to run the container with the --publish 0.0.0.0:8888:8888
option to publish Jupyter's port 8888
to the host machine
at port 8888
over all network interfaces (0.0.0.0
):
nvidia-docker run \
-v $PWD:/workspace/bert \
-v $PWD/results:/results \
--shm-size=1g \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
--publish 0.0.0.0:8888:8888 \
-it bert:latest bash
Then you can use the following command within the BERT Tensorflow container under
/workspace/bert
:
jupyter notebook --ip=0.0.0.0 --allow-root
And navigate a web browser to the IP address or hostname of the host machine
at port 8888
:
http://[host machine]:8888
Use the token listed in the output from running the jupyter
command to log
in, for example:
http://[host machine]:8888/?token=aae96ae9387cd28151868fee318c3b3581a2d794f3b25c6b
BioBERT Named-Entity Recognition Inference with Mixed Precision
1. Overview
Bidirectional Embedding Representations from Transformers (BERT), is a method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks.
BioBERT is a domain specific version of BERT that has been trained on PubMed abstracts.
The original BioBERT paper can be found here: https://arxiv.org/abs/1901.08746
NVIDIA's BioBERT is an optimized version of the implementation presented in the paper, leveraging mixed precision arithmetic and tensor cores on V100 GPUS for faster training times while maintaining target accuracy.
1.a Learning objectives
This repository contains an example notebook that demonstrates:
- Inference on NER task with BioBERT model
- The use/download of fine-tuned NVIDIA BioBERT models
- Use of Mixed Precision for Inference
Here is a short description of the relevant file:
- biobert_ner_tf_inference.ipynb : BioBERT Inference with TF Checkpoint model
2. Quick Start Guide
2.a Build the BERT TensorFlow NGC container:
To run the notebook you first need to build the Bert TensorFlow container using the following command from the main directory of this repository:
docker build . --rm -t bert
2.b Start of the NGC container to run inference:
Once the image is built, you need to run the container with the --publish 0.0.0.0:8888:8888
option to publish Jupyter's port 8888
to the host machine
at port 8888
over all network interfaces (0.0.0.0
):
nvidia-docker run \
-v $PWD:/workspace/bert \
-v $PWD/results:/results \
--shm-size=1g \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
--publish 0.0.0.0:8888:8888 \
-it bert:latest bash
Then you can use the following commands within the BERT Tensorflow container under
/workspace/bert
:
Install spaCy. You'll use this to pre-process text and to visualize the results using displaCy.
pip install spacy
python -m spacy download en_core_web_sm
Launch Jupyter.
jupyter notebook --ip=0.0.0.0 --allow-root
And navigate a web browser to the IP address or hostname of the host machine
at port 8888
:
http://[host machine]:8888
Use the token listed in the output from running the jupyter
command to log
in, for example:
http://[host machine]:8888/?token=aae96ae9387cd28151868fee318c3b3581a2d794f3b25c6b