On the application of deep learning with reinforcement in modern systems

Abstract

The article reveals the principles of application of deep learning with reinforcement in modern systems. It is emphasized that the function of reinforced learning includes the adaptation of the non-Markov model of decision-making to the situation that has developed due to the analysis of the prehistory of the decision-making process, which improves the quality of decisions. The principle of realization of training with reinforcement is described and the scheme of interaction of the agent with environment is schematically opened. For a detailed description, the use of a 2D pole balancing problem is proposed, which is the basis of the mathematical aspect. It is emphasized that in modern systems two schemes of reinforcement are most often used: the method of time differences and the method of Monte Carlo. The mathematical substantiation of each method is carried out separately and the architecture of a deep Q-network is offered. Model and non-model methods are described, it is emphasized that model methods are based on models of training with reinforcement, forcing the agent to try to understand the world and create a model for its presentation. Non-model methods try to capture two functions, the transition function and the reward function, from this model the agent has a link and can plan accordingly. However, it is noted that there is no need to study the model, and the agent can instead study the policy directly, using algorithms such as Q-learning or policy gradient. Deep Q-network, uses a convolutional neural network to directly interpret the graphical representation of the input state with the environment. It is substantiated that the deep Q-network can be considered as a parameterized policy network, which is constantly trained to approximate the optimal policy, and, mathematically, the deep Q-network uses the Bellman equation to minimize the loss function, which is effective in reducing time. However, the use of the neural network to approximate the value function proved to be unstable and could lead to discrepancies due to the bias resulting from correlative samples.