Information-Theoretic Models of Language Processing

Human language processing is the process by which a person transforms an input linguistic signal (audio, text, etc.) into a representation of meaning during comprehension, or transforms a representation of meaning into linguistic output during production. Information theory is the mathematical theory of information and communication. We will develop simple information-theoretic models that can explain some aspects of human language comprehension, with a focus on explaining eyetracking data in which eye movements are taken as an index of the mental processes that comprehenders are performing.

Course time will be spent on lectures, discussions, exercises, and demos. You will follow along and perform small group exercises in CoLab notebooks.

This course is intended for linguists and cognitive scientists of all backgrounds. The course assumes familiarity with the Python programming language, but only minimal familiarity with statistics, probability, information theory, etc.

• On information theory

  There is a Khan Academy video course on information theory, which is highly recommended.

  James Gleick wrote a popular book about information theory. The Information: A History, A Theory, a Flood.

  The comprehensive textbook on information theory is Cover & Thomas (2006). Prof. Cover's lectures based on the book are online. If you have a strong math background, this is the book to work through.

  A more accessible introduction is given in MacKay (2003).

  On probability

  If you would like to brush up on probability theory, I recommend watching John Tsitiklis's lectures.