Madatov Khabibulla, Bekchanov Shukurla, Vičič Jernej
Urgench state university, 14, Kh. Alimdjan str, Urgench city, 220100, Uzbekistan.
Research Centre of the Slovenian Academy of Sciences and Arts, The Fran Ramovš Institute, Novi trg 2, 1000 Ljubljana, Slovenija.
Data Brief. 2022 Jun 3;43:108351. doi: 10.1016/j.dib.2022.108351. eCollection 2022 Aug.
Filtering stop words is an important task when processing text queries to search for information in large data sets. It enables a reduction of the search space without losing the semantic meaning. The stop words, which have only grammatical roles and not contributing to information content still add up to the complexity of the query. Existing mathematical models that are used to tackle this problem are not suitable for all families of natural languages [1]. For example, they do not cover families of languages to which Uzbek can be included. In the present work, the collocation method of this problem is o ered for families of languages that include the Uzbek language as well. This method concerns the so-called agglutinative languages, in which the task of recognizing stop words is much more difficult, since the stop words are "masked" in the text. In this work the unigram, the bigram and the collocation methods are applied to the "School corpus" that corresponds to the type of languages being studied.
在处理文本查询以在大型数据集中搜索信息时，过滤停用词是一项重要任务。它能够在不丢失语义的情况下减少搜索空间。那些仅具有语法作用而对信息内容无贡献的停用词，仍然增加了查询的复杂性。用于解决此问题的现有数学模型并不适用于所有自然语言家族[1]。例如，它们没有涵盖乌兹别克语所属的语言家族。在本研究中，也为包括乌兹别克语在内的语言家族提供了该问题的搭配方法。此方法涉及所谓的黏着语，在这类语言中识别停用词的任务要困难得多，因为停用词在文本中被“掩盖”了。在这项工作中，一元语法、二元语法和搭配方法被应用于与所研究语言类型相对应的“学校语料库”。
