求条件熵和互信息的matlab实现

Assignment 1_08116649_Chaoyun_Song

EEE315 Information Theory and

Coding Assignment 1

Channel Capacity and Mutual

Information

ID : 08116649 Name: Chaoyun.Song

Assignment 1_08116649_Chaoyun_Song

1. Introduction

Shannon's information content should have some intuitive properties :1（ （）Information contained in the events ought to be defined in terms of some measure of uncertainty of the events. Less certain events ought to contain more information than more certain events. The information of unrelated events taken as a single event should equal the sum of the information of the unrelated events.

In information theory, entropy is a measure of the uncertainty associated with a random variable. In this context, the term usually refers to the Shannon entropy, which quantifies the expected value of the information contained in a message, usually in units such as bits. In this context, a 'message' means a specific realization of the random variable. Entropy is defined as:

H(x)???p(x)logp(x)it can be viewed as: a measure of the minimum cost needed to send some form of information; “the amount of surprise factor” of the information measured in bits. or how much energy it is worth spending to carry the information which translates to the minimum number of bits needed to code the information.

In probability theory and information theory, the mutual

information (sometimes known by the archaic termtransinformation) of two random variables is a quantity that measures the mutual dependence of the two random variables. The most common unit of measurement of mutual information is the bit, when logarithms to the base 2 are used. The mutual information can be defined as:

where p(x,y) is the joint probability distribution function of X and Y, and p1(x) and p2(y) are the marginal probability distribution functions of X and Y respectively.

Assignment 1_08116649_Chaoyun_Song

2. Result with Matlab stripts and functions

(1). Write a Matlab function to calculate the entropy of a source given a discrete distribution. Cacluate the entropy for the following distribution. Plot the entropy diagram for each of the distributions. A={B={

C={0.1, 0.31, 0.001, 0.009, 0.2, 0.15, 0.23} Solution: Matlab code

>> A=[1/2 1/4 1/8 1/8]; >> H1=-sum(A.*log2(A)) H1 =

1.7500

>> B=[1/4 1/4 1/4 1/4]; >> H2=-sum(B.*log2(B)) H2 = 2

>> C=[0.1, 0.31, 0.001, 0.009, 0.2, 0.15, 0.23]; >> H3= -sum(C.*log2(C))

1111,,248814}

14,

14,

14,}

Assignment 1_08116649_Chaoyun_Song

H3 =

2.2897

(2). Write a MATLAB[1] script to plot the capacity of a binary symmetric channel with cross probability p as function of p where 0

?p ? 1. For what value of p is the capacity minimized and what is

the minimum value?

For a binary symmetric channel(BSC), we know that P(0|1)=P(1|0)=p, P(0|0)=P(1|1)=1-p, which ?p? is the cross probability 0 ?p ? 1. When P(Y0)=P(Y1)=0.5 the mutual information comes to the minimum value. The capacity of this channel is like:

C= P(X0)P(0|0)log[P(0|0)/0.5]+ P(X0)P(1|0)log[P(1|0)/0.5]+ P(X1)P(0|1)log[P(0|1)/0.5]+ P(X1)P(1|1)log[P(1|1)/0.5] =plog2p+(1-p)log2(1-p)+1

Using matlab we can plot the diagram of ?p? and ?C?

Assignment 1_08116649_Chaoyun_Song

Solution： Matlab Code： >> p=0 : 0.01 : 1;

>> C=p.*log2(p)+(1-p).*log2(1-p)+1; >> plot(p,C)

From the diagram, we can see the change of ?C? with different value of ?p?. When p=1 the channel capacity is 1(bit/symbol)

When p=0.5 there are no information and the mutual information is 0 When 0.5 ?p ? 1 the diagram is same as leftside

So p=0.5 is the capacity minimized, the minimum value of C is 0.

(3). A binary non-symmetric channel is characterized by the probabilities P(0|1) =0.1 and P(1|0) = 0.2.

百度搜索“70edu”或“70教育网”即可找到本站免费阅读全部范文。收藏本站方便下次阅读，70教育网，提供经典综合文库求条件熵和互信息的matlab实现在线全文阅读。