Volltext-Downloads (blau) und Frontdoor-Views (grau)

On the effect of word frequency on distributional similarity

  • The dependency of word similarity in vector space models on the frequency of words has been noted in a few studies, but has received very little attention. We study the influence of word frequency in a set of 10 000 randomly selected word pairs for a number of different combinations of feature weighting schemes and similarity measures. We find that the similarity of word pairs for all methods, except for the one using singular value decomposition to reduce the dimensionality of the feature space, is determined to a large extent by the frequency of the words. In a binary classification task of pairs of synonyms and unrelated words we find that for all similarity measures the results can be improved when we correct for the frequency bias.

Download full text files

Export metadata

Additional Services

Search Google Scholar


Author:Christian WartenaORCiDGND
Parent Title (English):Proceedings of the 12th edition of the KONVENS conference Vol. 1
Place of publication:Hildesheim
Contributor(s):Gertrud Faaß
Document Type:Conference Proceeding
Year of Completion:2014
Publishing Institution:Hochschule Hannover
Release Date:2016/07/08
Tag:Statistische Methoden
Machine Learning; Statistical Methods
Page Number:10
Link to catalogue:798893532
Institutes:Fakultät III - Medien, Information und Design
DDC classes:400 Sprache, Linguistik
Licence (German):License LogoCreative Commons - Namensnennung 3.0