Variance and Covariance of Several Simultaneous Outputs of a Markov Chain

The partial sum of the states of a Markov chain or more generally a Markov source is asymptotically normally distributed under suitable conditions. One of these conditions is that the variance is unbounded. A simple combinatorial characterization of Markov sources which satisfy this condition is given in terms of cycles of the underlying graph of the Markov chain. Also Markov sources with higher dimensional alphabets are considered. Furthermore, the case of an unbounded covariance between two coordinates of the Markov source is combinatorically characterized. If the covariance is bounded, then the two coordinates are asymptotically independent. The results are illustrated by several examples, like the number of specific blocks in $0$-$1$-sequences and the Hamming weight of the width-$w$ non-adjacent form.


Introduction
We investigate the random vector defined as the n-th partial sum of a Markov source over a higher dimensional alphabet. Under suitable conditions, this random variable is asymptotically jointly normally distributed. Its mean and variance-covariance matrix is linear in the number of summands (cf. [6, Theorem 2.22]). On the one hand, these conditions include irreducibility and aperiodicity of the underlying graph of the Markov chain, which can be checked easily for a given Markov chain. On the other hand, we also have to check that the variance-covariance matrix is regular, which requires technical computations. In this article, we give a simple combinatorial characterization of Markov sources whose corresponding variance-covariance matrix is singular.
The covariance between two coordinates of this random vector is also of interest: If it is bounded, then these two coordinates are asymptotically independent because of the joint normal distribution. We give a combinatorial characterization of this case.
These characterizations are given in terms of subgraphs of the underlying graph of the Markov chain: For the variance-covariance matrix, we only have to consider all cycles. A regular variance-covariance For the characterization of an unbounded covariance, we have to consider functional digraphs. This result is proven using an extension of the Matrix-Tree Theorem in [5,20]. As Markov sources are closely related to automata and transducers, our results can also be used for the asymptotic analysis of sequences which can be computed by transducers. This includes the Hamming weight of many syntactically defined digit expansions as performed in [11,16,15,13,14]. Furthermore, occurrences of digits or subwords can also be computed by transducers. Their variance (and covariance) is analyzed in [12,2,19,3,22,8,10].
In [18], the variance of the output of a transducer as well as the covariance between the input and the output were analyzed. In this article, we consider the more general setting of Markov chains. The proofs are similar as those in [18], but the results are valid in a broader context and can be formulated more clearly. In contrast to [18], we allow the input sequence of the transducer to be generated by a Markov source. This allows us to model an input sequence for a transducer whose letters do not occur with equal probabilities and/or have dependencies between the letters. The precise relation between the setting of this article and that of [18] is given in Section 3.
As an example, we prove that the Hamming weight of the so-called width-w non-adjacent form is asymptotically jointly normally distributed for two different values of w ≥ 2. The width-w non-adjacent form is a binary digit expansion with digits in {0, ±1, ±3, . . . , ±(2 w−1 − 1)} and the syntactical rule that at most one of any w adjacent digits is non-zero. This digit expansion exists and is unique for every integer (cf. [21,1]). Furthermore, it has minimal Hamming weight among all digit expansions with this base and digit set.
The outline of this article is as follows: In Section 2, we define our setting and the types of graphs we use to state the combinatorial characterization of independent output sums and singular variance-covariance matrices. These characterizations are given in Section 3 and examples are given in Section 4. In Section 5, we finally prove the results of Section 3.

Preliminaries
In this article, a finite Markov chain consists of a finite state space {1, . . . , M }, a finite set of transitions E between the states, each with a positive transition probability, and a unique (i) initial state 1. We denote the transition probability for a transition e by p e . Then we have e∈E e starts in i p e = 1 for all states i. Note that for all transitions e ∈ E, we require p e > 0. Further note that there may be multiple transitions between two states but always only a finite number of them. This may be useful for different outputs later on.
The transition probabilities induce a probability distribution on the paths of length n starting in the initial state 1. Let X n be a random path of length n according to this model.
(i) This is no restriction as we can always add an additional state and the transitions starting in this state with probabilities corresponding to the non-degenerate initial distribution. The output functions are then extended by mapping these transitions to 0.
All states of the underlying digraph of the Markov chain are assumed to be accessible from the initial state. Contracting each strongly connected component of the underlying digraph gives an acyclic digraph, the so-called condensation. We assume that this condensation has only one leaf (i.e., one vertex with out-degree 0). The strongly connected component corresponding to this leaf is called final component. We assume that the period (i.e., the greatest common divisor of the lengths of all cycles) of this final component is 1. We call such Markov chains finally connected and finally aperiodic.
Additionally we use output functions k : E → R. The corresponding random variable K n is the sum of all values of k along a random path X n . We call K n the output sum of the Markov chain with respect to k. We use several output functions k 1 , . . . , k m and the corresponding random variables K Thus, our setting can be seen as a Markov source with a finite set of m-dimensional vectors as alphabet.
We are interested in the joint distribution of the random variables K n . For one coordinate, we will prove that the expected value of K will turn out to be Σn + O(1) for a matrix Σ. We call Σ the asymptotic variance-covariance matrix and its entries the asymptotic variances and covariances.
We will combinatorically characterize Markov chains with output functions such that the variancecovariance matrix is regular. Furthermore, we give a combinatorial characterization of the case that the asymptotic covariance is zero. As this is only influenced by two output functions, we restrict ourselves to K (1) n and K (2) n in this case. Remark 2.2. Markov chains with output functions are closely related to transducers with a probability distribution for the input: A transducer is defined to consist of a finite set of states, an initial state, a set of final states, an input alphabet, an output alphabet and a finite set of transitions, where a transition starts in one state, leads to another state and has an input and an output label from the corresponding alphabets. See [4, Chapter 1] for a more formal definition. An example of a transducer is given in Figure 1. We label the transitions with "input label | output label". The initial state is marked by an ingoing arrow starting at no other state and the final states are marked by outgoing arrows leading to no other state.
A Markov chain with one output function can be obtained by a transducer with additional probability distributions for the outgoing transitions of each state and by deleting the input labels of the transducer.
If we have two transducers where only the outputs of the transitions are different, we can choose probability distributions for the outgoing transitions of each state. Then we obtain a Markov chain with two output functions. Thus, we can use our results for two output functions (see Examples 4.2 and 4.3). Remark 2.3. We can additionally have final output functions f : {1, . . . , M } → R for each output function k and redefine the random variable K n as the sum of the values of the output function k along a random path X n plus the final output f of the final state of this path. We will see that this does not change the main terms of the asymptotic behavior. Thus, the results in Section 3 are still valid (see also Remark 5.5).
Remark 2.4. The Parry measure are probabilities p e such that every path of length n has the same weight up to a constant factor (cf. [24,23]). If we are interested in probabilities such that every path of length n starting in the initial state 1 has exactly the same weight, we have to use the Parry measure with additional exit weights: Each path is additionally weighted by these exit weights according to the final state of the path (cf. [17,Lemma 4.1]).
However, the sum of the weights of all paths of length n is no longer normalized: It differs from 1 by an exponentially small error term for n → ∞. This gives an approximate equidistribution of all paths of length n. As we are interested in the asymptotic behavior for n → ∞, the expected value and the variance of the corresponding measurable function K n can still be defined as usual.
If we use these exit weights w s in our setting, the main terms of the asymptotic behavior are not changed. Thus, the theorems in Section 3 are still valid (see also Remark 5.5).
These exit weights can also be used to simulate final and non-final states of a transducer by setting the weights of non-final states to 0. However, not all exit weights of the final component are allowed to be zero.
Next, we define some subgraphs of the underlying graph of the final component and extend the probabilities and the output functions to these subgraphs. Definition 2.5. We define the following types of directed graphs as subgraphs of the final component of the Markov chain.
• A rooted tree is a weakly connected digraph with one vertex which has out-degree 0, while all other vertices have out-degree 1. The vertex with out-degree 0 is called the root of the tree.
• A functional digraph is a digraph whose vertices have out-degree 1. Each component of a functional digraph consists of a directed cycle and some trees rooted at vertices of the cycle. For a functional digraph D, let C D be the set of all cycles of D.
The probabilities p e can be multiplicatively extended to a weight function for arbitrary subgraphs of the Markov chain: Let D be any subgraph of the underlying graph of the Markov chain, then define the weight of D by For a path P of length n, this is exactly the probability P(X n = P ). However, the output function k is additively extended to cycles C of the underlying graph of the Markov chain by This can further be extended to functional digraphs: Definition 2.6. Let D 1 and D 2 be the sets of all spanning subgraphs of the final component of the Markov chain M which are functional digraphs and have one and two components, respectively. For functions g and h : E → R, we define As functions g and h, we use the output functions k 1 , . . . , k m and the constant function 1(e) = 1.

Main Results
In this section, we present the combinatorial characterization of output functions of Markov chains which are asymptotically independent and of Markov chains with output functions with a singular variancecovariance matrix. The proofs can be found in Section 5.
If the underlying directed graph of the Markov chain is j-regular, every transition has probability 1/j, we only have two output functions and the first output function k 1 : E → {0, 1, . . . , j − 1} is such that the restrictions of k 1 to the outgoing transitions of one state is bijective for every state, then these results are stated in [18] (see also Remark 2.2).
The next definition describes a sequence of random variables whose difference from its expected value is bounded for all elements. holds for all n.
Next we give the combinatorial characterization of output sums with bounded variance in the case of a not necessarily independent identically distributed input sequence.

In that case, an + O(1) is the expected value of the output sum and Statement (b) holds for all states s of the final component.
If M is furthermore strongly connected, the following assertion is also equivalent: In the case that the value of the output function is 0 or 1 for each transition, there are only two trivial output functions with asymptotic variance zero. The next theorem gives a combinatorial characterization of output functions of a Markov chain which are asymptotically independent. As this characterization is given by the covariance, we can restrict ourselves to two output functions without loss of generality.
The covariance of K

Examples
In this section, we first prove the asymptotic joint normal distribution of the Hamming weights of two different digit expansions by using Theorem 2. Then we investigate the independence of length 2 blocks of 0-1-sequences by using Theorem 3. In both cases we start with two transducers to construct a Markov chain with two output functions, once as a Cartesian product, once via Remark 2.2. Example 4.1 (Width-w non-adjacent forms). Let 2 ≤ w 1 < w 2 be integers. We consider the asymptotic joint distribution of the Hamming weight of the width-w 1 non-adjacent form (w 1 -NAF) and the Hamming weight of the w 2 -NAF. The width-w non-adjacent form is a binary digit expansion with digit set {0, ±1, ±3, . . . , ±(2 w−1 − 1)} and the syntactical rule that at most one of any w adjacent digits is non-zero.
It will turn out that this distribution is normal if and only if the variance-covariance matrix is regular. Using Theorem 2, we have to find closed walks in the corresponding Markov chain such that all coefficients in (1) have to be zero.
The transducer T (w) in Figure 2 computes the Hamming weight of the w-NAF of the integer n when the input is the binary expansion of n (cf. [15]). It has w + 1 states. Next, we construct the Cartesian product of the transducers for w 1 and w 2 and choose any non-degenerate probability distribution, i.e. with all probabilities non-zero, for the outgoing transitions of a state. Thus, we obtain a Markov chain M with (w 1 + 1)(w 2 + 1) states with two different output functions h 1 and h 2 corresponding to the outputs of the transducers for w 1 and w 2 , respectively. We can now use Theorem 2 to prove that these two Hamming weights are asymptotically jointly normally distributed.
The Cartesian product of two closed walks in T (w 1 ) and T (w 2 ) with the same input sequence is a closed walk in M. We construct three different closed walks and prove that all three coefficients in (1) have to be zero. For brevity, we denote a closed walk in the Cartesian product M and its projections to T (w 1 ) and T (w 2 ) by the same letter.
First, we choose the closed walk C 1 starting in state 1 with input sequence 0. We obtain h 1 (C 1 ) = 0 in T (w 1 ), h 2 (C 1 ) = 0 in T (w 2 ) and 1(C 1 ) = 1. Second, we choose the closed walk C 2 starting in 1 with input sequence 10 w2−1 . Because w 1 < w 2 and the loop at state 1, C 2 is a closed walk in T (w 1 ) and T (w 2 ). We obtain h 1 (C 2 ) = 1 in T (w 1 ), h 2 (C 2 ) = 1 in T (w 2 ) and 1(C 2 ) = w 2 . The third choice depends on whether w 1 = w 2 − 1 or not: • w 1 = w 2 − 1: We choose the closed walk C 3 starting in 1 with input sequence 10 w1−1 10 w1−1 0 α where α = max(w 2 − 2w 1 , 0). On the one hand, this is a closed walk in T (w 1 ) consisting of two times the cycle 1 → w 1 → 1 and α times the loop at state 1. On the other hand, this is a closed walk in T (w 2 ) consisting of the cycle 1 → w 2 → 1 and the correct number of loops at state 1. We • w 1 = w 2 −1: We choose the closed walk C 3 starting in 1 with input sequence 10 w1−1 10 w1−1 10 w1−1 .
On the one hand, this is a closed walk in T (w 1 ) consisting of three times the cycle 1 → w 1 → 1. On the other hand, this is a closed walk in T (w 2 ) consisting of the closed walk 1 → w 2 → w 2 + 1 → w 2 → 1 and the correct number of loops at state 1. We obtain h 1 (C 3 ) = 3 in T (w 1 ), h 2 (C 3 ) = 2 in T (w 2 ) and 1(C 3 ) = 3w 1 . This yields a system of linear equations for the coefficients a 0 , a 1 and a 2 with coefficient matrix which only has the trivial solution. Thus, the Hamming weights of the w 1 -NAF and the w 2 -NAF are asymptotically jointly normally distributed, independently of the choice of the distributions for the Markov chain.
The next two examples investigate the asymptotic independence of length two blocks of 0-1-sequences. Example 4.2 (10-and 11-blocks). The two transducers in Figure 3 count the number of 10and 11-blocks in 0-1-sequences. After deleting the outputs, both transducers are the same. Thus, any non-degenerate probability distribution on the outgoing edges of the states gives a Markov chain with two output functions k 10 (for the 10-blocks) and k 11 (for the 11-blocks).
Because of the two loops and the cycle 0 → 1 → 0, Theorem 2 implies that the number of 10and 11-blocks is asymptotically normally distributed. The next question is: For which choices of probability distributions is the number of 10and 11-blocks asymptotically independent? All functional digraphs with one or two components are given in Figure 4. Using Theorem 3, we obtain the following system of equations for the values of the probabilities such that the numbers of 11-blocks and 10-blocks are asymptotically independent: first by definition then by (2) and finally for the independence This system has non-trivial real solutions, i.e. solutions where all probabilities are non-zero, with for all 0 < p 1→1 < 1. Then we have 2 − √ 2 < p 0→0 < 1. Thus, for these transition probabilities, the number of 10-blocks and the number of 11-blocks are asymptotically independent.
One such example of a non-trivial solution is p 1→1 = p 1→0 = 0.5, p 0→0 ≈ 0.7192 and p 0→1 ≈ 0.2808. Note that for the symmetric distributions p 0→0 = p 0→1 = p 1→1 = p 1→0 = 0.5, we obtain asymptotic dependence of the number of 10and 11-blocks. Example 4.3 (00-and 11-blocks). The two transducers in Figure 5 count the number of 00and 11-blocks in 0-1-sequences. They have the same underlying graph and the same input labels. Thus, choosing any non-degenerate probability distribution of the outgoing edges of the states yields a Markov chain with two output functions.
Because of the two loops and the cycle 0 → 1 → 0, Theorem 2 implies that the number of 00and 11-blocks is asymptotically normally distributed.
These equations have no solution with 0 < p e < 1 for all transitions e. Thus, the numbers of 00and 11-blocks are asymptotically dependent for all choices of the input distributions, as expected.

Proofs
In this section, we prove the results from Section 3. Most of the proofs follow along the same ideas as in [18]. The main differences are that one has to replace "complete transducer" by "Markov chain" and the input sum by the output sum K n . We first prove Theorem 3 with the help of two lemmas. For one of these lemmas, we use a version of the Matrix-Tree Theorem for weighted directed forests proved in [5,20]. At the end of this section, we prove Theorems 1 and 2. Let A = {i 1 , . . . , i n } and B = {j 1 , . . . , j n } with i 1 < · · · < i n and j 1 < · · · < j n . For F ∈ F A,B , we define a function g : B → A by g(j) = i if j is in the tree of F which is rooted in vertex i. We further define the function h : A → B by h(i k ) = j k for k = 1, . . . , n. The composition g • h : A → A is a permutation of A. We define sign F = sign g • h.
If |A| = |B|, then F A,B = ∅. If |A| = |B| = 1, then sign F = 1 and F A,B consists of all spanning trees rooted in a ∈ A.
Theorem (All-Minors-Matrix-Tree Theorem [5,20]). For a directed, weighted graph with loops and multiple edges, let L = (l ij ) 1≤i,j≤N be the Laplacian matrix, that is where * denotes any matrix. If the Markov chain is strongly connected, the matrices * are not present (they have 0 rows).
We first use the All-Minors-Matrix-Tree Theorem to connect the derivatives of the characteristic polynomial of the transition matrix with a sum of weighted digraphs in the next lemma.
This lemma can be proven in the same way as [18, Lemma 5.3] using the All-Minors-Matrix-Tree Theorem [5,20].
The following lemma will be used for m ≥ 2 output functions later on. and v(s 1 , . . . , s m ) are analytic functions in a small neighborhood of (0, . . . , 0).

Proof:
The moment generating function of (K  (3), we obtain F 1 (x 1 , . . . , x m , z) F 2 (x 1 , . . . , x m , z)f (x 1 , . . . , x m , z) for "polynomials" F 1 and F 2 , i.e. finite linear combinations of x α1 1 · · · x αm m z β for α i ∈ R and β a nonnegative integer. The function F 2 corresponds to the determinant of the non-final part of the Markov chain.
We obtain the coefficient of z n by singularity analysis (cf. [7]): Since the final component of M is again a Markov chain, the dominant singularity of 1/f (1, . . . , 1, z) is 1 by the theorem of Perron-Frobenius (cf. [9]). By the aperiodicity of the final component, this dominant singularity is unique and it is ρ(1, . . . , 1) = 1.
Next, we consider the non-final components of the Markov chain using the same arguments as in [18]. The corresponding non-final component M 0 is not a Markov chain as the transition matrix is not stochastic. Let M + 0 be the Markov chain that is obtained from M 0 by adding loops with the missing probabilities where necessary. The dominant eigenvalue of the transition matrix of M + 0 is 1. As the transition matrices of M 0 and M + 0 satisfy element-wise inequalities but are not equal (at (x 1 , . . . , x m ) = (1, . . . , 1)), the theorem of Perron-Frobenius (cf. [9, Theorem 8.8.1]) implies that the dominant eigenvalues of M 0 have absolute value less than 1. Thus, the dominant singularities of F 2 (1, . . . , 1, z) −1 are at |z| > 1.
Then Σ is singular if and only if the diagonal matrix D is singular. This is equivalent to holds for a j ∈ {1, . . . , m}. Now consider the output function t j1 k 1 + · · · + t jm k m . By Theorem 1, (4) is equivalent to t j1 k 1 (C) + · · · + t jm k m (C) = 0 holding for all cycles of the final component (since the expected value of this output function is O(1)).
If we shift back the output function such that the expected value is no longer bounded, we obtain an additional summand a 0 1(C).
The asymptotic joint normal distribution follows from Lemma 5.4 and the multidimensional Quasi-Power Theorem [6,Theorem 2.22].