Volltext-Downloads (blau) und Frontdoor-Views (grau)
(Leider keine statistischen Daten vom 26.05. – 18.06.2018)
  • search hit 13 of 14
Back to Result List

On the effect of word frequency on distributional similarity

  • The dependency of word similarity in vector space models on the frequency of words has been noted in a few studies, but has received very little attention. We study the influence of word frequency in a set of 10 000 randomly selected word pairs for a number of different combinations of feature weighting schemes and similarity measures. We find that the similarity of word pairs for all methods, except for the one using singular value decomposition to reduce the dimensionality of the feature space, is determined to a large extent by the frequency of the words. In a binary classification task of pairs of synonyms and unrelated words we find that for all similarity measures the results can be improved when we correct for the frequency bias.

Download full text files

Export metadata

Additional Services

Share in Twitter Search Google Scholar

Statistics

frontdoor_oas
Metadaten
Author:Christian WartenaORCiDGND
URN:urn:nbn:de:gbv:hil2-opus-2634
ISBN:978-3-934105-46-1
Parent Title (English):Proceedings of the 12th edition of the KONVENS conference Vol. 1
Place of publication:Hildesheim
Contributor(s):Gertrud Faaß
Document Type:Conference Proceeding
Language:English
Year of Completion:2014
Release Date:2016/07/08
Tag:Statistische Methoden
Machine Learning; Statistical Methods
Pagenumber:10
Link to catalogue:798893532
Institutes:Fakultät III - Medien, Information und Design
DDC classes:400 Sprache, Linguistik
Licence (German):License LogoCreative Commons - Namensnennung 3.0