View source for Distributional semantics

[[File:Distributional semantics.png|thumb|How words are related in a given language is demonstrated in the "semantic space", which mathematically corresponds to the vector space.]]{{Short description|Field of linguistics}}
'''Distributional semantics'''<ref>{{cite book | last1 = Lenci | first1 = Alessandro | last2 = Sahlgren | first2 = Magnus | title = Distributional Semantics | isbn = 9780511783692 | publisher = Cambridge University Press | year = 2023}}</ref> is a research area that develops and studies theories and methods for quantifying and categorizing semantic similarities between linguistic items based on their distributional properties in large samples of language data. The basic idea of distributional semantics can be summed up in the so-called [[Distributionalism|distributional]] hypothesis: ''linguistic items with similar distributions have similar meanings.''

==Distributional hypothesis==
The '''distributional hypothesis''' in [[linguistics]] is derived from the [[semantic theory]] of language usage, i.e. words that are used and occur in the same [[context (language use)|context]]s tend to purport similar meanings.<ref>{{harvnb|Harris|1954}}</ref>

The underlying idea that "a word is characterized by the company it keeps" was popularized by [[J. R. Firth|Firth]] in the 1950s.<ref>{{harvnb|Firth|1957}}</ref>

The distributional hypothesis is the basis for [[statistical semantics]]. Although the distributional hypothesis originated in linguistics,<ref>{{harvnb|Sahlgren|2008}}</ref><ref>{{harvnb|Sahlgren|2024}}</ref> it is now receiving attention in [[cognitive science]] especially regarding the context of word use.<ref>{{harvnb|McDonald|Ramscar|2001}}</ref>

In recent years, the distributional hypothesis has provided the basis for the theory of [[similarity-based generalization]] in language learning: the idea that children can figure out how to use words they've rarely encountered before by generalizing about their use from distributions of similar words.<ref>{{harvnb| Gleitman|2002}}</ref><ref>{{harvnb| Yarlett|2008}}</ref>

The distributional hypothesis suggests that the more semantically similar two words are, the more distributionally similar they will be in turn, and thus the more that they will tend to occur in similar linguistic contexts.

Whether or not this suggestion holds has significant implications for both the [[data-sparsity]] problem in computational modeling,<ref>{{cite conference |last1=Wishart |first1=Ryder |last2=Prokopidis |first2=Prokopis |title=Topic Modelling Experiments on Hellenistic Corpora |conference=Proceedings of the Workshop on Corpora in the Digital Humanities 17 |date=2017 |url=http://ceur-ws.org/Vol-1786/wishart.pdf |s2cid=9191936 }}</ref> and for the question of how children are able to learn language so rapidly given relatively impoverished input (this is also known as the problem of the [[poverty of the stimulus]]) is unclear.

==Distributional semantic modeling in vector spaces==
Distributional semantics favor the use of linear algebra as a computational tool and representational framework. The basic approach is to collect distributional information in high-dimensional vectors, and to define distributional/semantic similarity in terms of vector similarity.<ref>{{harvnb|Rieger |1991 }}</ref> Different kinds of similarities can be extracted depending on which type of distributional information is used to collect the vectors: '''topical''' similarities can be extracted by populating the vectors with information on which text regions the linguistic items occur in; '''paradigmatic''' similarities can be extracted by populating the vectors with information on which other linguistic items the items co-occur with. Note that the latter type of vectors can also be used to extract '''syntagmatic''' similarities by looking at the individual vector components.

The basic idea of a correlation between distributional and semantic similarity can be operationalized in many different ways. There is a rich variety of computational models implementing distributional semantics, including [[latent semantic analysis]] (LSA),<ref>{{harvnb|Deerwester|Dumais|Furnas|Landauer|1990}}</ref><ref>{{cite journal |last1=Landauer |first1=Thomas K. |last2=Dumais |first2=Susan T. |title=A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. |journal=Psychological Review |date=1997 |volume=104 |issue=2 |pages=211–240 |doi=10.1037/0033-295x.104.2.211 }}</ref> [[Hyperspace Analogue to Language]] (HAL), syntax- or dependency-based models,<ref>{{harvnb|Padó|Lapata|2007}}</ref> [[random indexing]], [[semantic folding]]<ref>{{cite arXiv|eprint=1511.08855|title=Semantic Folding Theory And its Application in Semantic Fingerprinting|last=De Sousa Webber|first=Francisco|date=2015|class=cs.AI}}</ref> and various variants of the [[topic model]].<ref>{{Cite journal|last1=Jordan|first1=Michael I.|last2=Ng|first2=Andrew Y.|last3=Blei|first3=David M.|date=2003|title=Latent Dirichlet Allocation|url=http://www.jmlr.org/papers/v3/blei03a.html|journal=Journal of Machine Learning Research|volume=3|issue=Jan|pages=993–1022 }}</ref>

Distributional semantic models differ primarily with respect to the following parameters:

* Context type (text regions vs. linguistic items)
* Context window (size, extension, etc.)
* Frequency weighting (e.g. [[entropy (information theory)|entropy]], [[pointwise mutual information]],<ref>{{Cite journal|last1=Church|first1=Kenneth Ward|last2=Hanks|first2=Patrick|date=1989|title=Word association norms, mutual information, and lexicography|journal=Proceedings of the 27th Annual Meeting on Association for Computational Linguistics |pages=76–83|location=Morristown, NJ, USA|publisher=Association for Computational Linguistics|doi=10.3115/981623.981633|doi-access=free}}</ref> etc.)
* Dimension reduction (e.g. [[random indexing]], [[singular value decomposition]], etc.)
* [[Similarity measure]] (e.g. [[cosine similarity]], [[Minkowski distance]], etc.)

Distributional semantic models that use linguistic items as context have also been referred to as '''word space, or vector space models'''.<ref>{{harvnb|Schütze|1993}}</ref><ref>{{harvnb|Sahlgren|2006}}</ref>

==Beyond Lexical Semantics==
While distributional semantics typically has been applied to lexical items—words and multi-word terms—with considerable success, not least due to its applicability as an input layer for neurally inspired deep learning models, [[lexical semantics]], i.e. the meaning of words, will only carry part of the semantics of an entire utterance. The meaning of a clause, e.g. ''"Tigers love rabbits."'', can only partially be understood from examining the meaning of the three lexical items it consists of. Distributional semantics can straightforwardly be extended to cover larger linguistic item such as constructions, with and without non-instantiated items, but some of the base assumptions of the model need to be adjusted somewhat. [[Construction grammar]] and its formulation of the lexical-syntactic continuum offers one approach for including more elaborate constructions in a distributional semantic model and some experiments have been implemented using the Random Indexing approach.<ref>{{cite journal |last1=Karlgren |first1=Jussi |last2=Kanerva |first2=Pentti |title=High-dimensional distributed semantic spaces for utterances |journal=Natural Language Engineering |date=July 2019 |volume=25 |issue=4 |pages=503–517 |doi=10.1017/S1351324919000226 |arxiv=2104.00424 |s2cid=201141249 }}</ref>

[[Compositional distributional semantics|Compositional distributional semantic]] models extend distributional semantic models by explicit semantic functions that use syntactically based rules to combine the semantics of participating lexical units into a ''compositional model'' to characterize the semantics of entire phrases or sentences. This work was originally proposed by Stephen Clark, [[Bob Coecke]], and [[Mehrnoosh Sadrzadeh]] of [[University of Oxford|Oxford University]] in their 2008 paper, "A Compositional Distributional Model of Meaning".<ref>{{Cite journal|last1=Clark|first1=Stephen|last2=Coecke|first2=Bob|last3=Sadrzadeh|first3=Mehrnoosh|date=2008|title=A compositional distributional model of meaning|url=http://www.cs.ox.ac.uk/people/stephen.clark/papers/qai08.pdf|journal=Proceedings of the Second Quantum Interaction Symposium|pages=133–140}}</ref> Different approaches to composition have been explored—including neural models—and are under discussion at established workshops such as [[SemEval]].<ref>{{cite web |url=http://alt.qcri.org/semeval2014/task1/|title=SemEval-2014, Task 1}}</ref>

==Applications==

Distributional semantic models have been applied successfully to the following tasks:
* finding [[semantic similarity]] between words and multi-word expressions;
* [[Keyword clustering|word clustering]] based on semantic similarity;
* automatic creation of [[thesauri]] and bilingual dictionaries;
* [[word-sense disambiguation]];
* expanding search requests using synonyms and associations;
* defining the topic of a document;
* [[document clustering]] for [[information retrieval]];
* [[data mining]] and [[named-entity recognition]];
* creating [[semantic mapping (literacy)|semantic maps]] of different subject domains;
* [[Paraphrasing (computational linguistics)|paraphrasing]];
* [[sentiment analysis]];
* modeling selectional preferences of words.

==Software==
* [https://github.com/fozziethebeat/S-Space/ S-Space]
* [https://github.com/semanticvectors/semanticvectors/ SemanticVectors]
* [http://radimrehurek.com/gensim/index.html Gensim]
* [http://www.linguatools.de/disco/disco-builder.html DISCO Builder]
* [https://github.com/Lambda-3/Indra Indra]

==See also==
*[[Conceptual space]]
*[[Co-occurrence]]
*[[Distributional–relational database]]
*[[Gensim]]
*[[Phraseme]]
*[[Random indexing]]
*[[Sentence embedding]]
*[[Statistical semantics]]
*[[Word2vec]]
*[[Word embedding]]

===People===
*[[Scott Deerwester]]
*[[Susan Dumais]]
*[[J. R. Firth]]
*[[George Furnas]]
*[[Zellig Harris]]
*[[Thomas Landauer]]
*[[Magnus Sahlgren]]

==References==

{{reflist}}

===Sources===

* {{cite journal | last = Harris | first = Z. | year = 1954 | title = Distributional structure | journal = Word | volume = 10 | issue = 23| pages = 146–162 | doi = 10.1080/00437956.1954.11659520 }}
* {{cite journal | last = Firth| first = J.R. | year = 1957 | title = A synopsis of linguistic theory 1930-1955 | journal = Studies in Linguistic Analysis | pages = 1–32 }} Reprinted in {{cite book | editor = F.R. Palmer | title = Selected Papers of J.R. Firth 1952-1959 | publisher = London: Longman | year = 1968}}
* {{cite book | last1 = Lenci | first1 = Alessandro | last2 = Sahlgren | first2 = Magnus | title = Distributional Semantics | isbn = 9780511783692 | publisher = Cambridge University Press | year = 2023}}
* {{cite journal | last = Sahlgren | first = Magnus | year = 2008 | title = The Distributional Hypothesis | url = http://soda.swedish-ict.se/3941/1/sahlgren.distr-hypo.pdf | journal = Rivista di Linguistica | volume = 20 | issue = 1 | pages = 33–53 | access-date = 2010-12-10 | archive-date = 2012-03-15 | archive-url = https://web.archive.org/web/20120315233953/http://soda.swedish-ict.se/3941/1/sahlgren.distr-hypo.pdf | url-status = dead }}
* {{cite conference | last1 = McDonald | first1 = S. | last2 = Ramscar | first2 = M. | year = 2001 | citeseerx = 10.1.1.104.7535 | title = Testing the distributional hypothesis: The influence of context on judgements of semantic similarity | book-title = Proceedings of the 23rd Annual Conference of the Cognitive Science Society | pages = 611–616 }}
* {{cite book | last = Gleitman | first = Lila R. | year = 2002 | volume = 1| pages = 209–229| doi=10.1075/cilt.228.17gle| series = Current Issues in Linguistic Theory | isbn = 978-90-272-4736-0 | chapter = Verbs of a feather flock together II | title = The Legacy of Zellig Harris }} 
* {{cite thesis | last = Yarlett | first = D. | year = 2008 | title = Language Learning Through Similarity-Based Generalization | url = http://psych.stanford.edu/~michael/papers/Draft_Yarlett_Similarity.pdf | degree = PhD | publisher = Stanford University | access-date = 2012-07-12 | archive-url = https://web.archive.org/web/20140419012951/http://psych.stanford.edu/~michael/papers/Draft_Yarlett_Similarity.pdf | archive-date = 2014-04-19 | url-status = dead }}
* {{cite report | last = Rieger |first= Burghard B. | year= 1991 | citeseerx = 10.1.1.37.7976 | title  =  On Distributed Representations in Word Semantics | url  =  http://ftp.icsi.berkeley.edu/ftp/pub/techreports/1991/tr-91-012.pdf | publisher = ICSI Berkeley 12-1991 }}
* {{cite journal | last1 = Deerwester | first1 = Scott | last2 = Dumais | first2 = Susan T. | last3 = Furnas | first3 = George W. | last4 = Landauer | first4 = Thomas K. | last5 = Harshman | first5 = Richard | url = http://lsi.research.telcordia.com/lsi/papers/JASIS90.pdf | title = Indexing by Latent Semantic Analysis | journal = Journal of the American Society for Information Science | volume = 41 | issue = 6 | pages = 391–407 | year = 1990 | doi = 10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9 | url-status = dead | archive-url = https://web.archive.org/web/20120717020428/http://lsi.research.telcordia.com/lsi/papers/JASIS90.pdf | archive-date = 2012-07-17 | citeseerx = 10.1.1.33.2447 }}
* {{cite journal | last1 = Padó | first1 = Sebastian | last2 = Lapata | first2 = Mirella | year = 2007 | title = Dependency-based construction of semantic space models | journal = Computational Linguistics | volume = 33 | issue = 2 | pages = 161–199 | doi=10.1162/coli.2007.33.2.161| s2cid = 7747235 | doi-access = free }}
* {{cite conference | last = Schütze | first = Hinrich | year = 1993 | citeseerx = 10.1.1.41.8856 | title = Word Space | book-title = Advances in Neural Information Processing Systems 5 | pages = 895–902 }}
* {{cite thesis | last = Sahlgren | first = Magnus | year = 2006 | title = The Word-Space Model | url = http://soda.swedish-ict.se/437/1/TheWordSpaceModel.pdf | degree = PhD | publisher = Stockholm University | access-date = 2012-11-26 | archive-date = 2012-06-19 | archive-url = https://web.archive.org/web/20120619002510/http://soda.swedish-ict.se/437/1/TheWordSpaceModel.pdf | url-status = dead }}
* {{cite journal |last1=Sahlgren |first1=Magnus |date=December 2024 |title=Distributional Legacy: The Unreasonable Effectiveness of Harris's Distributional Program |url=https://www.tandfonline.com/doi/full/10.1080/00437956.2024.2414515 |journal=WORD |volume=70 |issue=4 |pages=246–257 |doi=10.1080/00437956.2024.2414515 |url-access=subscription }}
* {{cite web | url=http://www.welchco.com/02/14/01/60/96/02/2901.HTM | title=A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge |author1=Thomas Landauer |author2=Susan T. Dumais | access-date=2007-07-02  }}
* {{cite conference |author1=Kevin Lund |author2=Curt Burgess |author3=Ruth Ann Atchley | year = 1995 | title = Semantic and associative priming in a high-dimensional semantic space | conference = Cognitive Science Proceedings | pages = 660–665 }}
* {{cite journal |author1=Kevin Lund |author2=Curt Burgess | year = 1996 | title = Producing high-dimensional semantic spaces from lexical co-occurrence | journal = Behavior Research Methods, Instruments, and Computers | volume = 28 | issue = 2 | pages = 203–208 | doi=10.3758/bf03204766| doi-access = free }}

==External links==
*[http://zelligharris.org Zellig S. Harris]

{{Natural language processing}}

{{DEFAULTSORT:Distributional Hypothesis}}
[[Category:Computational linguistics]]
[[Category:Semantics]]
[[Category:Language acquisition]]
[[Category:Semantic relations]]