Natural Computation and Self-Organization:
The Physics of Information Processing in Complex Systems

Jim Crutchfield
chaos@cse.ucdavis.edu; http://cse.ucdavis.edu/~chaos

Winter 2008
WWW: http://cse.ucdavis.edu/~chaos/courses/ncaso/

Homework 7

Calculating word probabilities from a model of a process: Consider the deterministic hidden Markov model of the Golden Mean process:

T⁽⁰⁾	=
T⁽¹⁾	=

Find the equilibrium distribution 〈e∣ = (p_A,p_B) for the internal states, A and B.

The probability of length L = 1 words is

p(0)	= 〈e∣T⁽⁰⁾∣n〉
p(1)	= 〈e∣T⁽¹⁾∣n〉 ,

where ∣n〉 is the column vector containing all 1s. Calculate these.

Generally, the probability of word w = s₀s₁s₂ occuring is p(w) = 〈e∣T^(w)∣n〉 = 〈e∣T^(s₀)T^(s₁)T^(s₂)∣n〉. Apply this technique to find the probability of length L = 2 words:

p(00)	= 〈e∣T⁽⁰⁾T⁽⁰⁾∣n〉
p(01)	= 〈e∣T⁽⁰⁾T⁽¹⁾∣n〉
p(10)	= 〈e∣T⁽¹⁾T⁽⁰⁾∣n〉
p(11)	= 〈e∣T⁽¹⁾T⁽¹⁾∣n〉 .

For the Period-2, Golden Mean, and Even processes:
1. Calculate word probabilities for L = 1,2,3,4.
2. Entropy growth: Find the block entropy H(L) for each L above.
3. Entropy convergence: Find the length-L entropy rate h_μ(L) for each L above.
4. Find the asymptotic entropy rate h_μ using its closed-form expression: $∑ ∑ ∑ (s) (s) hμ = - pi Ti,j log2 Ti,j . i j s$ How does this compare with the h_μ(L) estimates?
5. Find the predictability gain Δh_μ(L) for each L above.
6. Approximate the total predictability G using the estimates from the previous step.
7. Calculate the redundancy . How does this compare with the previous estimate?
8. Use the estimates of h_μ and h_μ(L) to approximate the excess entropy E up to the given L.
9. Estimate the transient information T up to the given L.
10. What do the values obtained above tell you about the processes under consideration?
The ε-machine for Golden Mean process:
1. Extend the topological reconstruction of the ε-machine for the Golden Mean process covered in the lecture to obtain the full probabilistic version. Show the parse tree and the morphs at a depth appropriate for correct reconstruction. Give the history sets for each causal state and the resulting ε-machine.
2. Calculate the entropy rate h_μ and the statistical complexity C_μ. Compare C_μ to the excess entropy E. (You calculated the latter in the previous homework.)
The ε-machine for the Even process:
1. Reconstruct the ε-machine for the Even process. The correct answer is given in the Lecture Notes. Show the parse tree and the morphs at a depth appropriate for correct reconstruction.
2. Calculate the entropy rate h_μ and the statistical complexity C_μ. Compare the later to the excess entropy E. (You calculated the latter in the previous homework, but this was only an approximation. You can look at the RURO paper on the Computational Mechanics Reader page at the course website for a more accurate value.)

Homework due one week after being assigned.