Merge branch 'main' into main

* [Scaling Language Model Training to a Trillion Parameters Using Megatron](https://arxiv.org/pdf/2104.04473.pdf)

* [Training Question Answering Models From Synthetic Data](https://www.aclweb.org/anthology/2020.emnlp-main.468.pdf)

Our codebase is capable of efficiently training very large (hundreds of billions of parameters) language models with both model and data parallelism. To demonstrate how the code scales with multiple GPUs and model sizes, we consider GPT models from 1 billion all the way to 1 trillion parameters. All models use a vocabulary size of 51,200 and a sequence length of 2048. We vary hidden size, number of attention heads, and number of layers to arrive at a specific model size. As the model size increases, we also modestly increase the batch size. We leverage [NVIDIA's Selene supercomputer](https://www.top500.org/system/179842/) to perform scaling studies and use up to 3072 [A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for the largest model. The table below shows the model configurations along with the achieved FLOPs (both per GPU and aggregate over all GPUs). Note that the FLOPs are measured for end-to-end training, i.e., includes all operations including data loading, optimization, and even logging.

Megatron is also used in [NeMo Megatron](https://developer.nvidia.com/nvidia-nemo#nemo-megatron), a framework to help enterprises overcome the challenges of building and training sophisticated natural language processing models with billions and trillions of parameters.

Our codebase is capable of efficiently training very large (hundreds of billions of parameters) language models with both model and data parallelism. To demonstrate how the code scales with multiple GPUs and model sizes, we consider GPT models from 1 billion all the way to 1 trillion parameters. All models use a vocabulary size of 51,200 and a sequence length of 2048. We vary hidden size, number of attention heads, and number of layers to arrive at a specifc model size. As the model size increases, we also modestly increase the batch size. We leverage [NVIDIA's Selene supercomputer](https://www.top500.org/system/179842/) to perform scaling studies and use up to 3072 [A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for the largest model. The table below shows the model configurations along with the achieved FLOPs (both per GPU and aggregate over all GPUs). Note that these results are from benchmark runs and these models were not trained to convergence; however, the FLOPs are measured for end-to-end training, i.e., includes all operations including data loading, optimization, and even logging.

![Cases](images/cases_april2021.png)

Several downstream tasks are described for both GPT and BERT models below. They can be run in distributed and model parallel modes with the same changes used in the training scripts.

## GPT Text Generation

`bash examples/generate_text.sh`

We generate text samples using largely the GPT pretraining script. Few changes need to make, such as we need to provide the path to the pretrained checkpoint, the length of the output samples, whether to generate texts unconditionally (`--num-samples` to denote how many samples to generate) or conditional (need to pass `--sample-input-file <filename>` where each line of the file will be used as the conditional texts). There are few optional parameters to play, e.g. `top-k`, `top-p`, or `greedy` (set top-k and top-p to 0) sampling..

We have included a simple REST server to use for text generation in `tools/run_text_generation_server.py`. You run it much like you would start a pretraining job, specifying an appropriate pretrained checkpoint. There are also few optional parameters: `temperature`, `top-k`and `top-p`. See `--help` or the source file for more information. See [examples/run_text_generation_server_345M.sh](examples/run_text_generation_server_345M.sh) for an example of how to run the server.

Once the server is running you can use `tools/text_generation_cli.py` to query it, it takes one argument which is the host the server is running on.

<pre>

CHECKPOINT_PATH=checkpoints/gpt2_345m

VOCAB_FILE=gpt2-vocab.json

MERGE_FILE=gpt2-merges.txt

GPT_ARGS=&#60;same as those in <a href="#gpt-pretraining">GPT pretraining</a> above&#62;

tools/text_generation_cli.py localhost

</pre>

MAX_OUTPUT_SEQUENCE_LENGTH=1024

TEMPERATURE=1.0

TOP_P=0.9

NUMBER_OF_SAMPLES=2

OUTPUT_FILE=samples.json

You can also use CURL or any other tools to query the server directly:

python tools/generate_samples_gpt.py \

       $GPT_ARGS \

       --load $CHECKPOINT_PATH \

       --out-seq-length $MAX_OUTPUT_SEQUENCE_LENGTH \

       --temperature $TEMPERATURE \

       --genfile $OUTPUT_FILE \

       --num-samples $NUMBER_OF_SAMPLES \

       --top_p $TOP_P \

       --recompute

<pre>

curl 'http://localhost:5000/api' -X 'PUT' -H 'Content-Type: application/json; charset=UTF-8'  -d '{"prompts":["Hello world"], "tokens_to_generate":1}'

</pre>

See [megatron/text_generation_server.py](megatron/text_generation_server.py) for more API options.

## GPT Evaluation

We include example scripts for GPT evaluation on WikiText perplexity evaluation and LAMBADA Cloze accuracy.

#!/bin/bash

CHECKPOINT_PATH=checkpoints/gpt2_345m

VOCAB_FILE=gpt2-vocab.json

MERGE_FILE=gpt2-merges.txt

python tools/generate_samples_gpt2.py \

       --tensor-model-parallel-size 1 \

       --num-layers 24 \

       --hidden-size 1024 \

       --load $CHECKPOINT_PATH \

       --num-attention-heads 16 \

       --max-position-embeddings 1024 \

       --tokenizer-type GPT2BPETokenizer \

       --fp16 \

       --batch-size 2 \

       --seq-length 1024 \

       --out-seq-length 1024 \

       --temperature 1.0 \

       --vocab-file $VOCAB_FILE \

       --merge-file $MERGE_FILE \

       --genfile unconditional_samples.json \

       --num-samples 2 \

       --top_p 0.9 \

       --recompute

# Multi-Stage Prompting for Knowledgeable Dialogue Generation

This directory contains all the scripts of multi-stage prompting for knowledgeable dialogue generation that includes data preparation, and knowledge and response generations. More details are available on [`knowledgeable task directory`](../../tasks/msdp).

#!/bin/bash

# Data preparation for our framework: preprocessing the WoW and WoI datasets

# The datasets can be downloaded through the following links:

# WoW: https://parl.ai/projects/wizard_of_wikipedia/

# WoI: https://parl.ai/projects/sea/

DIR=`pwd`

# Before running the preprocessing, please download 

# the wizard of wikipedia and wizard datasets

WOW_DATA_FOLDER=<PATH_OF_WIZARD_OF_WIKIPEDIA_DATA_FOLDER>

WOI_DATA_FOLDER=<PATH_OF_WIZARD_OF_INTERNET_DATA_FOLDER>

# We provide examples for processing the raw data from Wizard of Wikipedia

# Processing the train dataset (train.json)

python ${DIR}/tasks/msdp/preprocessing.py \

        --func process_wow_dataset \

        --raw_file ${WOW_DATA_FOLDER}/train.json \

        --processed_file ${WOW_DATA_FOLDER}/train_processed.txt

# Processing test seen dataset (test_random_split.json)

python ${DIR}/tasks/msdp/preprocessing.py \

        --func process_wow_dataset \

        --raw_file ${WOW_DATA_FOLDER}/test_random_split.json \

        --processed_file ${WOW_DATA_FOLDER}/testseen_processed.txt \

        --knwl_ref_file ${WOW_DATA_FOLDER}/output_testseen_knowledge_reference.txt \

        --resp_ref_file ${WOW_DATA_FOLDER}/output_testseen_response_reference.txt

# processing test unseen dataset (test_topic_split.json)

python ${DIR}/tasks/msdp/preprocessing.py \

        --func process_wow_dataset \

        --raw_file ${WOW_DATA_FOLDER}/test_topic_split.json \

        --processed_file ${WOW_DATA_FOLDER}/testunseen_processed.txt \

        --knwl_ref_file ${WOW_DATA_FOLDER}/output_testunseen_knowledge_reference.txt \

        --resp_ref_file ${WOW_DATA_FOLDER}/output_testunseen_response_reference.txt

# We provide the following script to process the raw data from Wizard of Internet

# Processing the test dataset (test.jsonl)

python ${DIR}/tasks/msdp/preprocessing.py \

        --func process_woi_dataset \

        --raw_file ${WOI_DATA_FOLDER}/test.jsonl \

        --processed_file ${WOI_DATA_FOLDER}/test_processed.txt \

        --knwl_ref_file ${WOI_DATA_FOLDER}/output_test_knowledge_reference.txt \

        --resp_ref_file ${WOI_DATA_FOLDER}/output_test_response_reference.txt

# Get the knowledge generation prompts for the each test dataset in WoW and WoI

MODEL_FILE=<PATH_OF_THE_FINETUNED_DPR_MODEL> 

# WoW test seen

python ${DIR}/tasks/msdp/preprocessing.py \

        --func get_knwl_gen_prompts \

        --test_file ${WOW_DATA_FOLDER}/testseen_processed.txt \

        --train_file ${WOW_DATA_FOLDER}/train_processed.txt \

        --model_file ${MODEL_FILE} \

        --processed_file ${WOW_DATA_FOLDER}/output_testseen_knowledge_prompts.json \

        --data_type wow_seen

# WoW test unseen

python ${DIR}/tasks/msdp/preprocessing.py \

        --func get_knwl_gen_prompts \

        --test_file ${WOW_DATA_FOLDER}/testunseen_processed.txt \

        --train_file ${WOW_DATA_FOLDER}/train_processed.txt \

        --model_file ${MODEL_FILE} \

        --processed_file ${WOW_DATA_FOLDER}/output_testunseen_knowledge_prompts.json \

        --data_type wow_unseen

# WoI

python ${DIR}/tasks/msdp/preprocessing.py \

        --func get_knwl_gen_prompts \

        --test_file ${WOI_DATA_FOLDER}/test_processed.txt \

        --train_file ${WOW_DATA_FOLDER}/train_processed.txt \

        --model_file ${MODEL_FILE} \

        --processed_file ${WOI_DATA_FOLDER}/output_test_knowledge_prompts.json \

        --data_type woi

# Get the response generation prompts (can be applied for all the test datasets)

python ${DIR}/tasks/msdp/preprocessing.py \

        --func get_resp_gen_prompts \

        --train_file ${WOW_DATA_FOLDER}/train_processed.txt \

        --processed_file ${WOW_DATA_FOLDER}/output_response_prompts.txt

#!/bin/bash

#########################

# Evaluate the F1 scores.

#########################

WORLD_SIZE=1

DISTRIBUTED_ARGS="--nproc_per_node $WORLD_SIZE \

                  --nnodes 1 \

                  --node_rank 0 \

                  --master_addr localhost \

                  --master_port 6000"

MODEL_GEN_PATH=<PATH_OF_THE_KNOWLEDGE_GENERATION> \ 

        (e.g., /testseen_knowledge_generations.txt)

GROUND_TRUTH_PATH=<PATH_OF_THE_GROUND_TRUTH_KNOWLEDGE> \ 

        (e.g., /testseen_knowledge_reference.txt)

python -m torch.distributed.launch $DISTRIBUTED_ARGS ./tasks/msdp/main.py \

        --num-layers 24 \

        --hidden-size 1024 \

        --num-attention-heads 16 \

        --seq-length 2048 \

        --max-position-embeddings 2048 \

        --micro-batch-size 4 \

        --task MSDP-EVAL-F1 \

        --guess-file ${MODEL_GEN_PATH} \

        --answer-file ${GROUND_TRUTH_PATH}

############################################

# Evaluate BLEU, METEOR, and ROUGE-L scores.

############################################

# We follow the nlg-eval (https://github.com/Maluuba/nlg-eval) to 

# evaluate the BLEU, METEOR, and ROUGE-L scores. 

# To evaluate on these metrics, please setup the environments based on 

# the nlg-eval github, and run the corresponding evaluation commands.

nlg-eval \

    --hypothesis=<PATH_OF_THE_KNOWLEDGE_GENERATION> \

    --references=<PATH_OF_THE_GROUND_TRUTH_KNOWLEDGE>