Simulation-based optimal sensor scheduling with application to observer trajectory planning

Abstract

The sensor scheduling problem can be formulated as a controlled hidden Markov model and this paper solves the problem when the state, observation and action spaces are continuous. This general case is important as it is the natural framework for many applications. The aim is to minimise the variance of the estimation error of the hidden state w.r.t. the action sequence. We present a novel simulation-based method that uses a stochastic gradient algorithm to find optimal actions.

Introduction

Consider the following continuous state hidden Markov model (HMM):

$$X_{n+1}=f(X_n,A_{n+1},W_n),Y_n=g(X_n,A_n,V_n)\text{,}$$

where $X_n\in {\mathbb{R}}^{d_x}$ is the hidden system state, $Y_n\in {\mathbb{R}}^{d_y}$ the observation of the state and $W_n$ and $V_n$ are i.i.d. noise terms. Unlike the classical HMM model, the evolution of the state and observation processes depends on an input parameter $A_n\in {\mathbb{R}}^{d_a}$, which is the control or action. In HMM models, one is primarily concerned with the problem of estimating the hidden state, which is achieved by propagating the posterior distribution (or filtering density) ${\pi }_n(x)\mathrm{d}x=\mathbf{P}(X_n\in \mathrm{d}x|A_{1:n},Y_{1:n})$. By a judicious choice of action sequence $\{A_n\}$, the evolution of the state and observation processes can be ‘steered’ in order to yield filtering densities that give more accurate estimates of the state process. This problem is also known in the literature as the sensor scheduling problem.

Sensor scheduling has been a topic of interest to the target tracking community for the some years now (Hernandez et al., 2004, Kershaw and Evans, 1994, Logothetis et al., 1997, Meier et al., 1967, Singh et al., 2005, Tremois and Le Cadre, 1999). The classical setting is the problem of tracking a manoeuvring target over N epochs. Here $X_n$ denotes the state of the target at epoch n, $Y_n$ the observation provided by the sensor and $A_n$ some parameter of the sensor that may be adjusted to improve the ‘quality’ of the observation. For example, consider a non-moving platform with a finite number of sensors, where each has different characteristics. In this case $A_n$ denotes the choice of sensor to be used at epoch n (Lee et al., 2001, Singh et al., 2004). Alternatively, there may be only one sensor and $A_n$ could denote some tunable parameter of the sensor, as in the waveform selection problem (Kershaw & Evans, 1994), or in the case of directing an electronically scanned aperture (ESA) radar (Blackman & Popoli, 1999). In contrast, consider the scenario in which a moving platform (or observer) is to be adaptively manoeuvred to optimise the tracking performance of a manoeuvring target. In this setting, $A_n$ denotes the position of the observer at epoch n and the problem is termed the optimal observer trajectory planning (OTP) problem (Hernandez et al., 2004, Logothetis et al., 1997, Tremois and Le Cadre, 1999). In all these sensor scheduling problems, a measure of tracking performance is the mean squared tracking error over the N epochs,

$$\mathbf{E}\left\{\sum _{n=1}^N(\psi (X_n)-〈{\pi }_n,\psi 〉{)}^2\right\}\text{,}$$

where $\psi :{\mathbb{R}}^{d_x}\to \mathbb{R}$ is a suitable test function that emphasises the component (or components) of interest of the state vector we wish to track. The aim is to minimise (2) w.r.t. the choice of actions $\{A_1,\dots ,A_N\}$.

When the dynamics of the state and the observation processes are both linear and Gaussian, then the optimal solution to the sensor scheduling problem (2) (when $\psi$ gives a quadratic cost) can be computed off line (Meier et al., 1967); this is not surprising given that the Kalman filter covariance is also independent of the actual realisation of observations. In the general setting studied in this paper, the dynamics can be both non-linear and non-Gaussian, which means that the filtering density ${\pi }_n$, and integration w.r.t. it, cannot be evaluated in closed form. Hence, the tracking error performance criterion itself does not admit a closed-form expression. To further complicate matters, the actions sought are continuous valued, i.e., vectors in ${\mathbb{R}}^{d_a}$.

To address the complications to do with the non-linear and non-Gaussian dynamics, one could linearise the state and observation model (as in Logothetis et al. (1997)), i.e., using the extended Kalman filter to propagate the filtering density ${\pi }_n$. However, dealing with the tracking error performance criterion directly is the exception rather than the rule. The majority of works (Hernandez et al., 2004, Paris and Le Cadre, 2002, Tremois and Le Cadre, 1999, and references therein), while aiming to minimise mean squared tracking error, do so indirectly by defining a lower bound to the tracking error criterion and minimising the lower bound instead. The bound in question is the posterior Cramer–Rao lower bound (PCRLB), which is the inverse of the Fisher information matrix (FIM). This approach hinges on the ability to propagate recursively the FIM in closed form by a Ricatti-type equation for the non-linear and non-Gaussian filtering problem. Unfortunately, the recursion for the FIM involves evaluating the expectation of certain derivatives of the transition probability density of the state dynamics, as well as the expectation of certain derivatives of the observation likelihood (see (3) and (4) below). As these quantities cannot be evaluated in general except for the linear and Gaussian case, this assumption is either invoked or the authors resort to simulation-based approximations.

As for the complications due to continuous valued actions, the approach in the literature is to discretise ${\mathbb{R}}^{d_a}$ to a grid. There have also been studies where the continuous state HMM (1) is approximated by a discrete state HMM, and the latter solved using dynamic programming (Tremois & Le Cadre, 1999).

The aim of this paper is to solve the sensor scheduling problem with continuous action space directly, and not a surrogate problem defined through the PCRLB or otherwise. We make no assumptions of linearity or Gaussianity for analytic convenience, nor do we discretise the state, observation or action space. We avoid these restrictive modelling assumptions on the continuous state HMM by recourse to methods based on computer simulation (simulation for short). As the action policy derived will be a function of the filtering density, we will employ simulation to approximate the posterior density by a finite sum of weighted point–mass distributions (Doucet, De Freitas, & Gordon, 2001). The main advantage of simulation over other numerical integration methods is that it is typically very easy to implement. Furthermore, it follows the strong law of large numbers (Del Moral, 2004) and there is much literature on its efficient implementation for approximating posterior densities (Doucet et al., 2001).

In order to solve for the optimal sequence of continuous valued actions, we will use an iterative stochastic gradient algorithm. We derive the gradient of the performance criterion w.r.t. the action trajectory and demonstrate how low-variance estimates of it may be obtained using control variate (CV) (Glynn & Szechtman, 2002) techniques. One major advantage of a stochastic gradient-based method is that theoretical guarantees are easily obtained. Under suitable regularity assumptions, one can guarantee convergence to a local optimum of the performance criterion, while it is difficult to make similar assertions about the quality of the solutions obtained by other approximate methods proposed in the literature for sensor scheduling.

As an instance of the sensor scheduling problem, we study the OTD problem for a bearings-only application. We state theoretical results concerning the convergence of the observer trajectory identified by our simulation-based algorithm. Handling multiple observers simultaneously is easy in our proposed framework, and numerical results are presented for cooperating observers tracking a manoeuvring target.

The organisation of this paper is as follows. In Section 2, we formulate the optimal sensor scheduling problem. We also summarise some key points concerning several methods in the literature that may be used to solve this problem. In Section 3, we derive the gradient of the performance criterion being optimised, and detail the use of simulation and variance reduction techniques for its estimation. In Section 4.2, we present the main algorithm of the paper, which is a two time-scale stochastic gradient algorithm for solving the sensor scheduling problem. General convergence results for this algorithm are presented in the Appendix. In Section 5, we formulate the OTD problem as an instance of the sensor scheduling problem and apply the convergence results to this application. Numerical examples are presented in Section 6, and concluding remarks are presented in Section 7. All proofs appear in the technical report (Singh et al., 2005), which is available online or may be obtained by e-mailing any of the authors.

Notation

The notation that is used in the paper is now outlined. The norm of a scalar, vector or matrix is denoted by $|·|$. For a vector b, $\left|b\right|$ denotes the vector 2-norm $\sqrt{{\sum }_i|b(i){|}^2}$. For a matrix A, $\left|A\right|$ denotes the matrix 2-norm, ${\max }_{b:\left|b\right|\ne 0}\left|\mathit{Ab}\right|/\left|b\right|$. For convenience, we also denote a vector $b\in {\mathbb{R}}^n$ by $b=[b(i){]}_{i=1,\dots ,n}$, or the ith component of a vector by $[b{]}_i$. For scalars $a_{j,i}$, $j=1,\dots ,m$, $i=1,\dots ,n$, let $[[a_{j,i}{]}_{j=1,\dots ,m}{]}_{i=1,\dots ,n}$ denote the stacked vector $[a_{1,1},\dots ,a_{m,1},\dots ,a_{1,n},\dots ,a_{m,n}{]}^{\mathrm{T}}$. For a vector b, let diag(b) denote the diagonal matrix formed from b. For a function $f:{\mathbb{R}}^n\to \mathbb{R}$ with arguments $z\in {\mathbb{R}}^n$, we denote $(\mathrm{\partial }f/\mathrm{\partial }z(i))(z)$ by ${\nabla }_{z(i)}f(z)$ and $\nabla f(z)=[{\nabla }_{z(1)}f(z),\dots ,{\nabla }_{z(n)}f(z){]}^{\mathrm{T}}$. For the vector-valued function $F=[F_1,\dots ,F_n{]}^{\mathrm{T}}:{\mathbb{R}}^n\to {\mathbb{R}}^n$, let $\nabla F$ denote the matrix $[\nabla F_1,\dots ,\nabla F_n]$. For real-valued integrable functions f and g, let $〈f,g〉$ denote $\int f(x)g(x)\mathrm{d}x$.