push current code to gitlab

Megatron is also used in [NeMo Megatron](https://developer.nvidia.com/nvidia-nemo#nemo-megatron), a framework to help enterprises overcome the challenges of building and training sophisticated natural language processing models with billions and trillions of parameters.

Our codebase is capable of efficiently training very large (hundreds of billions of parameters) language models with both model and data parallelism. To demonstrate how the code scales with multiple GPUs and model sizes, we consider GPT models from 1 billion all the way to 1 trillion parameters. All models use a vocabulary size of 51,200 and a sequence length of 2048. We vary hidden size, number of attention heads, and number of layers to arrive at a specifc model size. As the model size increases, we also modestly increase the batch size. We leverage [NVIDIA's Selene supercomputer](https://www.top500.org/system/179842/) to perform scaling studies and use up to 3072 [A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for the largest model. Each cluster node has 8 NVIDIA 80GB A100 GPUs. The table below shows the model configurations along with the achieved FLOPs (both per GPU and aggregate over all GPUs). Note that these results are from benchmark runs and these models were not trained to convergence; however, the FLOPs are measured for end-to-end training, i.e., includes all operations including data loading, optimization, and even logging.

Additionally, the model parallel size column reports a combined tensor and pipeline parallelism degrees. For numbers larger than 8, typically tensor parallel of size 8 was used. So, for example, the 145B model reports the total model parallel size of 64, which means that this setup used TP=8 and PP=8.

Our codebase is capable of efficiently training very large (hundreds of billions of parameters) language models with both model and data parallelism. To demonstrate how the code scales with multiple GPUs and model sizes, we consider GPT models from 1 billion all the way to 1 trillion parameters. All models use a vocabulary size of 51,200 and a sequence length of 2048. We vary hidden size, number of attention heads, and number of layers to arrive at a specifc model size. As the model size increases, we also modestly increase the batch size. We leverage [NVIDIA's Selene supercomputer](https://www.top500.org/system/179842/) to perform scaling studies and use up to 3072 [A100](https://www.nvidia.com/en-us/data-center/a100/) GPUs for the largest model. The table below shows the model configurations along with the achieved FLOPs (both per GPU and aggregate over all GPUs). Note that these results are from benchmark runs and these models were not trained to convergence; however, the FLOPs are measured for end-to-end training, i.e., includes all operations including data loading, optimization, and even logging.

![Cases](images/cases_april2021.png)

      * [Data Preprocessing](#data-preprocessing)

      * [BERT Pretraining](#bert-pretraining)

      * [GPT Pretraining](#gpt-pretraining)

      * [GPT Pretraining](#gpt-pretraining)

      * [T5 Pretraining](#t5-pretraining)

      * [Distributed Pretraining](#distributed-pretraining)

      * [GPT-3 Example](#gpt-3-example)

## T5 Pretraining

Very similar to BERT and GPT, the `examples/pretrain_t5.sh` script runs single GPU "base" (~220M parameter) T5 pretraining. The primary difference from BERT and GPT is the addition of the following arguments to accommodate the T5 architecture:

Very similar to BERT and GPT, the `examples/pretrain_t5.sh` script runs single GPU "base" (~220M parameter) T5 pretraining. The primary difference from BERT and GPT is the addition of the following arguments to accomodate the T5 architecture:

* `--kv-channels` sets the inner dimension of the "key" and "value" matrices of all attention mechanisms in the model. For BERT and GPT this defaults to the hidden size divided by the number of attention heads, but can be configured for T5.

<!-- The number of microbatches in a per-pipeline minibatch is controlled by the `--num-microbatches-in-minibatch` argument. With `WORLD_SIZE` GPUs, `TENSOR_MP_SIZE` tensor-model-parallel size, `PIPELINE_MP_SIZE` pipeline-model-parallel-size, `WORLD_SIZE`/(`TENSOR_MP_SIZE` * `PIPELINE_MP_SIZE`) GPUs will be used for data parallelism. The default values for `--tensor-model-parallel-size` and `--pipeline-model-parallel-size` is 1, which will not implement either form of model parallelism. -->

We have examples of how to use these two different forms of model parallelism the example scripts ending in `distributed_with_mp.sh`:

We have examples of how to use these two different forms of model parallelism the example scripts ending in `distributed_with_mp.sh`, note that pipeline parallelism is not currently supported in the T5 model:

Other than these minor changes, the distributed training is identical to the training on a single GPU.

We provide several command line arguments, detailed in the scripts listed below, to handle various zero-shot and fine-tuned downstream tasks. However, you can also finetune your model from a pretrained checkpoint on other corpora as desired. To do so, simply add the `--finetune` flag and adjust the input files and training parameters within the original training script. The iteration count will be reset to zero, and the optimizer and internal state will be reinitialized. If the fine-tuning is interrupted for any reason, be sure to remove the `--finetune` flag before continuing, otherwise the training will start again from the beginning.

Because evaluation requires substantially less memory than training, it may be advantageous to merge a model trained in parallel for use on a single GPU in downstream tasks. The following script accomplishes this. Currently only tensor model parallelism is supported on input and pipeline model parallelism on the output. This example reads in a model with 2-way tensor model parallelism and writes out a model with 2-way pipeline model parallelism.

Because evaluation requires substantially less memory than training, it may be advantageous to merge a model trained in parallel for use on a single GPU in downstream tasks. The following script accomplishes this. Currently only tensor model parallelism is supported on input and pipeline model parallelsim on the output. This example reads in a model with 2-way tensor model parallelism and writes out a model with 2-way pipeline model parallelism.

<pre>

TENSOR_MODEL_PARALLEL_SIZE=2

### LAMBADA Cloze Accuracy

To compute LAMBADA cloze accuracy (the accuracy of predicting the last token given the preceding tokens) we utilize a detokenized, processed version of the [LAMBADA dataset](https://github.com/cybertronai/bflm/blob/master/lambada_test.jsonl).

To compute LAMBADA cloze accuracy (the accuracy of predicting the last token given the preceeding tokens) we utilize a detokenized, processed version of the [LAMBADA dataset](https://github.com/cybertronai/bflm/blob/master/lambada_test.jsonl).

We use the following command to run LAMBADA evaluation on a 345M parameter model. Note that the `--strict-lambada` flag should be used to require whole word matching. Make that `lambada` is part of the file path.

#!/bin/bash

#BSUB -nnodes 2

#BSUB -W 0:45

#BSUB -P med107

#BSUB -W 2:00

#BSUB -P med106

#BSUB -alloc_flags "smt4 nvme"

#BSUB -J hoc_FULL

#BSUB -o hoc_FULL.%J

#module load open-ce/1.4.0-py38-0

module load open-ce

conda deactivate

conda activate /gpfs/alpine/med106/world-shared/irl1/rhel8/myt_py1.11

conda activate /gpfs/alpine/med106/world-shared/irl1/rhel8/mytorch

module list

export OMP_NUM_THREADS=1

ulimit -n 65536

rm -f `find -name *lock`

#export PYTHONPATH=$PYTHONPATH:/gpfs/alpine/med106/world-shared/irl1/rhel8/fork-megatron/megatron/fused_kernels

#export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/gpfs/alpine/med106/world-shared/irl1/rhel8/mytorch/lib/python3.8/site-packages/torch/lib

#export PATH=$PATH:/gpfs/alpine/med106/world-shared/irl1/rhel8/mytorch/lib/python3.8/site-packages/torch/include

#export VALID_DATA=picodata/dev.tsv

export VOCAB_FILE=/gpfs/alpine/world-shared/med106/g8o/pubmed_bert-vocab.txt

export CHECKPOINT_PATH=/gpfs/alpine/med106/world-shared/irl1/rhel8/fork-megatron/finetune-pubmed_bert_1x1_b8

export PRETRAINED_CHECKPOINT=/gpfs/alpine/med106/world-shared/irl1/rhel8/fork-megatron/pubmed_bert_1x1_b8_chckpttt

export CHECKPOINT_PATH=/gpfs/alpine/med106/world-shared/irl1/rhel8/fork-megatron/finetune-HOC_BIG

export PRETRAINED_CHECKPOINT=/gpfs/alpine/med106/world-shared/irl1/rhel8/fork-megatron/chkptt

jsrun --smpiargs="-disable_gpu_hooks" -n $nnodes -r 1 -g 6 -a 6 -c 42 python tasks/main.py \

       --task HOC \

       --tensor-model-parallel-size 1 \

       --pipeline-model-parallel-size 1 \

       --num-layers 12 \

       --hidden-size 768 \

       --num-attention-heads 12 \

       --tensor-model-parallel-size 2 \

       --pipeline-model-parallel-size 2 \

       --num-layers 24 \

       --hidden-size 1024 \

       --num-attention-heads 16 \

       --seq-length 512 \

       --max-position-embeddings 512 \

       --fp16 \

       --micro-batch-size 4 \

       --lr 0.0001 \

       --lr-warmup-fraction 0.06 \

       --num-workers 0 \

       --distributed-backend nccl

       #--DDP-impl torch \

from megatron import get_args, get_timers, mpu, print_rank_0

from megatron.data.vit_dataset import build_train_valid_datasets

from megatron.model import ModelType

from megatron.model.vision.classification import VitClassificationModel

from megatron.model.vit_model import VitModel

from megatron.training import pretrain

from megatron.utils import average_losses_across_data_parallel_group

    print_rank_0("building VIT model ...")

    args = get_args()

    model = VitClassificationModel(num_classes=args.num_classes,

    model = VitModel(num_classes=args.num_classes,

                     pre_process=pre_process,

                     post_process=post_process)

    return model

    print_rank_0(

        "> building train, validation, and test datasets " "for VIT ..."

    )

    train_ds, valid_ds = build_train_valid_datasets(

        data_path=args.data_path,

        image_size=(args.img_h, args.img_w)

    )

    train_ds, valid_ds = build_train_valid_datasets(data_path=args.data_path)

    print_rank_0("> finished creating VIT datasets ...")

    return train_ds, valid_ds, None

class HOCDataset(Dataset):

    #def __init__(self, dataset_name, datapaths, tokenizer, max_seq_length,ignore_index=-100, tasks=['I-PAR', 'I-INT', 'I-OUT']):

    def __init__(self, dataset_name, datapaths, tokenizer, max_seq_length):

    def __init__(self, dataset_name, datapaths, tokenizer, max_seq_length,ignore_index=-100, tasks=['I-PAR', 'I-INT', 'I-OUT']):

        args = get_args()

        self.dataset_name = dataset_name

            for i in range(num_classes):

                percent = float(correct[i]) * 100.0 / float(total)

                print(' >> |epoch: {}| overall: correct / total = {} / {} = '

                    '{:.4f} %'.format(epoch+1, correct[i], total, percent))

                    '{:.4f} %'.format(epoch, correct[i], total, percent))

        if output_predictions and is_last_rank():

            assert args.load is not None

            elapsed_time = time.time() - start_time

            print_rank_last(' > |epoch: {}| metrics for {}: correct / total '

                            '= {} / {} = {:.4f} %, elapsed time (sec): {:.3f}'.format(

                                epoch+1, name, correct_ans[i], total_count,

                                epoch, name, correct_ans[i], total_count,

                                percent, elapsed_time))

        if output_predictions: