Text Processing

Vector space based Information Retrieval System for Text Processing – Information retrieval

Sequence of operations

  1. Install Requirements
  2. Add given wikipedia files to the corpus directory.
  3. Download glove.6B.100d.txt dataset (Ignore if already present) and place it in the project root directory.
  4. Run construct_index.py
  5. Run construct_index.py –zoned_index True
  6. Run trim_embeddings.py
  7. Run test_queries.py
  8. Run test_queries.py –score_title True
  9. Run test_queries.py –expand_query True

Installing Requirements:

   pip install -r requirements.txt

corpus

Contains the files to be indexed. Add files directly to this directory. Do not create subdirectories.
For this assignment, we have used the following files present in the AA folder of wikipedia files.
wiki_00
wiki_01
wiki_05
wiki_06
wiki_10
wiki_11
wiki_15
wiki_16
wiki_20
wiki_21
wiki_25
wiki_26
wiki_30
wiki_31

index_files

Contains the

 

 

 

To finish reading, please visit source site