Setting cut-off on idf values while calculating cosine similarity between two documents

text_mining
tf-idf
python

#1

I have two nodes both of which contains a list of 5 documents each. In each document there are a number of words. I want to find the distance of a keyword (which also has a list of 5 documents) with both the nodes and assign it to the node having minimum cosine similarity.

For example: A and B are two nodes containing 5 document each and I want to assign C (keyword containing 5 document) to A or B by calculating cosine similarity with each of the nodes.

At both the nodes, I have stored the idf values for each word present in the documents. Based on both the nodes idf values, two different vectors will be calculated using all the words in documents stored at keyword and then cosine similarity is calculated. The nodes having minimum cosine similarity will lie closer to the keyword and the keyword will be assigned to it.

The problem with the above approach is that I don’t want to consider those words of keyword documents which are present in both the nodes most of the time as considering those words will pull the two vectors closer and thus it will be difficult to judge where exactly the keyword will lie.

How can i remove those words from the vector calculation or is there any way in which I can cap to not consider those words which are present in the documents of both the nodes a large number of times.