Commit 2ef470e8 authored by mshoeybi's avatar mshoeybi
Browse files

Merge branch 'update-sample-bert-pretrain-args' of... 

Merge branch 'update-sample-bert-pretrain-args' of https://github.com/roclark/Megatron-LM into github_small_prs
parents 2c26bf78 3f57f46a

examples/pretrain_bert_distributed_with_mp.sh

+1 −0
Original line number Diff line number Diff line
@@ -23,6 +23,7 @@ python -m torch.distributed.launch $DISTRIBUTED_ARGS \ 
       --num-attention-heads 16 \

       --micro-batch-size 2 \

       --global-batch-size 16 \

       --seq-length 512 \

       --max-position-embeddings 512 \

       --train-iters 1000000 \

       --save $CHECKPOINT_PATH \