Merge branch 'github_small_prs' into 'main'

## BERT Pretraining

The `examples/pretrain_bert.sh` script runs single GPU 345M parameter BERT pretraining. Debugging is the primary use for single GPU training, as the code base and command line arguments are optimized for highly distributed training. Most of the arguments are fairly self-explanatory. By default, the learning rate decays linearly over the training iterations starting at `--lr` to a minimum set by `--min-lr` over `--lr-decay-iters` iterations. The fraction of training iterations used for warmup is set by `--lr-warmup-fraction`. While this is single GPU training, the batch size specified by `--micro-batch-size` is a single forward-backward path batch-size and the code will perform gradient accumulation steps until it reaches `global-batch-size` whcih is the batch size per iteration. The data is partitioned into a 949:50:1 ratio for training/validation/test sets (default is 969:30:1). This partitioning happens on the fly, but is consistent across runs with the same random seed (1234 by default, or specified manually with `--seed`). We use `train-iters` as the training iterations requested. Alternatively, one can provide `--train-samples` which is total number of samples to train on. If this option is present, then instead of providing `--lr-decay-iters`, one will need to provide `--lr-decay-samples`.

The `examples/pretrain_bert.sh` script runs single GPU 345M parameter BERT pretraining. Debugging is the primary use for single GPU training, as the code base and command line arguments are optimized for highly distributed training. Most of the arguments are fairly self-explanatory. By default, the learning rate decays linearly over the training iterations starting at `--lr` to a minimum set by `--min-lr` over `--lr-decay-iters` iterations. The fraction of training iterations used for warmup is set by `--lr-warmup-fraction`. While this is single GPU training, the batch size specified by `--micro-batch-size` is a single forward-backward path batch-size and the code will perform gradient accumulation steps until it reaches `global-batch-size` which is the batch size per iteration. The data is partitioned into a 949:50:1 ratio for training/validation/test sets (default is 969:30:1). This partitioning happens on the fly, but is consistent across runs with the same random seed (1234 by default, or specified manually with `--seed`). We use `train-iters` as the training iterations requested. Alternatively, one can provide `--train-samples` which is total number of samples to train on. If this option is present, then instead of providing `--lr-decay-iters`, one will need to provide `--lr-decay-samples`.

The logging, checkpoint-saving, and evaluation intervals are specified. Checkpointing the activations facilitates the training of larger models and/or batches. Note that the `--data-path` now includes the additional `_text_sentence` suffix added in preprocessing, but does not include the file extensions.

       --num-attention-heads 16 \

       --micro-batch-size 2 \

       --global-batch-size 16 \

       --seq-length 512 \

       --max-position-embeddings 512 \

       --train-iters 1000000 \

       --save $CHECKPOINT_PATH \

    MLP will take the input with h hidden state, project it to 4*h

    hidden dimension, perform nonlinear transformation, and project the

    state back into h hidden dimension. At the end, dropout is also

    applied.

    state back into h hidden dimension.

    """

    def __init__(self, init_method, output_layer_init_method):

            init_method=output_layer_init_method,

            skip_bias_add=True)

    def forward(self, hidden_states):

        # [s, b, 4hp]

def destroy_model_parallel():

    """Set the groups to none."""

    global _MODEL_PARALLEL_GROUP

    _MODEL_PARALLEL_GROUP = None

    global _TENSOR_MODEL_PARALLEL_GROUP

    _TENSOR_MODEL_PARALLEL_GROUP = None

    global _PIPELINE_MODEL_PARALLEL_GROUP

    _PIPELINE_MODEL_PARALLEL_GROUP = None

    global _DATA_PARALLEL_GROUP

    _DATA_PARALLEL_GROUP = None

    global _EMBEDDING_GROUP

    _EMBEDDING_GROUP = None

        keep_master_weight_for_test: This was added for testing and should be

                                     set to False. It returns the master weights

                                     used for initialization.

        skip_bias_add: This was added to enable performance optimations where bias

                       can be fused with other elementwise operations. we skip 

        skip_bias_add: This was added to enable performance optimization where bias

                       can be fused with other elementwise operations. We skip

                       adding bias but instead return it.

    """