On top of the challenges discussed in the POMDP project description (i.e. optimality properties over long planning horizons and efficient integration of uncertainties in the decision optimization process) another major challenge hindering efficient engineering systems decision-making in large scales is the curse of dimensionality in the state and action sets related to the number of system components that the system consists of.

That is, a system with 10 components having 5 possible damage states each (e.g. no, minor, major, severe, extensive, near-collapse damage), and 3 available actions each (e.g. do nothing, repair, replace), has nearly 10M possible states and 60K possible actions at every decision step from a system-level standpoint. This exponential scaling with the number of components renders the planning problem practically intractable by standard optimization schemes, from gradient-based and mixed-integer programming approaches to genetic algorithms and conventional Markov decision processes schemes.

Simplifying techniques to reduce this complexity mainly focus on the optimization of decision rules that are based on heuristic risk- and condition-based thresholds that activate certain decisions; block and/or periodic policies; clustering techniques that allow for policy uniformity of intra-cluster components; ranking/scoring metrics that prescribe component prioritization; and combinations thereof. Naturally, such assumptions, however, explore a limited subset of the space of all possible policies (i.e. combinations of actions with possible posterior system probability distributions), thus at best describing locally optimal, or otherwise suboptimal, policies. But how can we solve the original optimization problem?

​

Recent advances in Artificial Intelligence (AI) and machine learning provide a powerful answer to this question through Deep Reinforcement Learning (DRL). Integrating POMDPs with DRL allows us to parametrize the involved value and/or policy functions through neural networks, thus alleviating the curse of dimensionality related to joint probabilistic state representations. This parametrization is carried out by (deep) neural networks, whose parameters are trained through reinforcement learning. The latter, allows us to solve the underlying problem through real-time interaction with the belief-MDP environment, a process that gradually updates an initial policy until convergence.

​

The nonlinear approximation of the value function (e.g. expected life-cycle cost) can be achieved either directly, by parametrizing the action-value or the value function (e.g. Deep Q-Networks in the former case) or indirectly, by parametrizing the policy function (policy gradients), or by combinations of the above (actor-critic). The Deep Centralized Multi-agent Actor Critic (DCMAC) algorithm, integrates POMDPs with the actor-critic DRL framework. A critic network parametrizes the expected life-cycle cost, and a multi-agent actor outputs conditionally independent policies based on the entire system belief. In this approach, the actor is 'centralized', in the sense that parameters are shared for the hidden layers and global information is accessible to all agents, however, output actions are 'decentralized'. Drawing from the concept of point-based POMDPs, but without the need for using joint probabilistic representation, DCMAC operates directly on the belief space of the marginal component beliefs. The basic training concept is shown below.

​