On the effect of word frequency on distributional similarity
- The dependency of word similarity in vector space models on the frequency of words has been noted in a few studies, but has received very little attention. We study the influence of word frequency in a set of 10 000 randomly selected word pairs for a number of different combinations of feature weighting schemes and similarity measures. We find that the similarity of word pairs for all methods, except for the one using singular value decomposition to reduce the dimensionality of the feature space, is determined to a large extent by the frequency of the words. In a binary classification task of pairs of synonyms and unrelated words we find that for all similarity measures the results can be improved when we correct for the frequency bias.
| Author: | Christian WartenaORCiDGND |
|---|---|
| URN: | urn:nbn:de:gbv:hil2-opus-2634 |
| DOI: | https://doi.org/10.25968/opus-870 |
| ISBN: | 978-3-934105-46-1 |
| Parent Title (English): | Proceedings of the 12th edition of the KONVENS conference Vol. 1 |
| Place of publication: | Hildesheim |
| Contributor(s): | Gertrud Faaß |
| Document Type: | Conference Proceeding |
| Language: | English |
| Year of Completion: | 2014 |
| Publishing Institution: | Hochschule Hannover |
| Release Date: | 2016/07/08 |
| Tag: | Statistische Methoden Machine Learning; Statistical Methods |
| Page Number: | 10 |
| Link to catalogue: | 798893532 |
| Institutes: | Fakultät III - Medien, Information und Design |
| DDC classes: | 400 Sprache, Linguistik |
| Licence (German): |  Creative Commons - Namensnennung 3.0 |
