Noiseless Coding

Abstract

In this chapter we consider the problem of safe transmission of a message over a channel, which cannot be affected by noise. We are looking for error-free and fastest possible methods for transmitting messages. This is a rather special, but important, problem in classical information theory. We rely mainly on the following two central tools: prefix-free sets and Shannon entropy. Undoubtedly, the prefix-free sets are the easiest codes to construct, and most interesting problems on codes can be raised for prefix-free sets. Shannon entropy is a measure of the degree of ignorance concerning which possibility holds in an ensemble with a given a priori probability distribution. Later on, we shall contrast the Shannon measure with the information content of an individual (finite) object — viewed as a measure of how difficult it is to specify that object.