Commit c99fa80c authored Apr 14, 2020 by Mohammad Shoeybi

Merge branch 'move_openwebtext' into 'staging'

Move openwebtext scripts to tools directory.

See merge request ADLR/megatron-lm!50

parents 69d8ff1c 0dc10d09

openwebtext/README.md→tools/openwebtext/README.md

+0 −0

File moved.

openwebtext/blacklist_urls.py→tools/openwebtext/blacklist_urls.py

+0 −0

File moved.

openwebtext/cleanup_dataset.py→tools/openwebtext/cleanup_dataset.py

+0 −0

File moved.

openwebtext/find_duplicates.py→tools/openwebtext/find_duplicates.py

+0 −0

File moved.

openwebtext/group_duplicates_url.py→tools/openwebtext/group_duplicates_url.py

+0 −0

File moved.