added readme

shuf <cleaned deduped data file> -o train_data.json

```

# Deduplicating ngrams

To deduplicate the downstream tasks from the training dataset, we run the following command.

```

python filter_ngrams.py <down stream task dataset> <training dataset to deduplicate> <output training dataset>

```

We use 13-grams for the deduplication. When we find a 13-gram match in a training document, we split the document into two pieces and remove the 13-gram along with 200 characters from the both side of the 13-gram. We also remove any splitted document with less than 200 characters or if a document got splitted more than 10 times.

# See the License for the specific language governing permissions and

# limitations under the License.

"""

Deduplicate downstream tasks from training dataset. 13-grams have been used.

All split documents with less than 200 characters got filtered. Any document

with more than 10 splits got filtered as well.

"""

from functools import partial

import json

import multiprocessing

import time

def get_words(text):

    # get all the lowercase words from text

    words, positions = [], []

    for match in re.finditer(r'\w+', text.lower()):

        words.append(match.group(0))

def free_ngram(line, ngrams, ngram_size, filter_text_len, 

    splits_count, split_window_each_size):

    # remove all the ngrams

    try:

        myjson = json.loads(line)

        text_buf = [myjson['text']]