Commit c2a32e12 authored by Neel Kant's avatar Neel Kant
Browse files

Add more documentation

parent 11f76cd3
Loading
Loading
Loading
Loading
+3 −1
Original line number Diff line number Diff line
@@ -324,7 +324,7 @@ After having trained an ICT model, you can now embed an entire dataset of blocks
and wrap it with a `FaissMIPSIndex` to do fast similarity search which is key in the learned information retrieval pipeline. The initial index can be built with the following script, meant to be run in an interactive session. It can leverage multiple GPUs on multiple nodes to index large datasets much more quickly. 

<pre>
python indexer.py \
python tools/create_doc_index.py \
    --num-layers 12 \
    --hidden-size 768 \
    --ict-head-size 128 \
@@ -337,6 +337,8 @@ python indexer.py \
    --data-path /path/to/indexed_dataset \
    --titles-data-path /path/to/titles_indexed_dataset \
    --block-data-path embedded_blocks.pkl \
    --indexer-log-interval 1000 \
    --indexer-batch-size 128 \
    --vocab-file /path/to/vocab.txt \
    --num-workers 2 \
    --fp16
+2 −0
Original line number Diff line number Diff line
@@ -13,6 +13,8 @@ def main():
        --block-data-path: path to write to
        --ict-load or --realm-load: path to checkpoint with which to embed
        --data-path and --titles-data-path: paths for dataset
        --indexer-log-interval: reporting interval
        --indexer-batch-size: size specific for indexer jobs

    Check README.md for example script
    """