Twin delayed deep deterministic policy gradient-based target tracking for unmanned aerial vehicle with achievement rewarding and multistage training

Abo Mosali, Najmaddin and Shamsudin, Syariful Syafiq and Alfandi, Omar and Omar, Rosli and AL-Fadhali, Najib (2022) Twin delayed deep deterministic policy gradient-based target tracking for unmanned aerial vehicle with achievement rewarding and multistage training. IEEE Access, 10. pp. 23545-23559. ISSN 2169-3536

Text J14006_28269837c18b517045750a7bc07cf431.pdf Restricted to Registered users only Download (2MB) | Request a copy

Abstract

Target tracking using an unmanned aerial vehicle (UAV) is a challenging robotic problem. It requires handling a high level of nonlinearity and dynamics. Model-free control effectively handles the uncertain nature of the problem, and reinforcement learning (RL)-based approaches are a good candidate for solving this problem. In this article, the Twin Delayed Deep Deterministic Policy Gradient Algorithm (TD3), as recent and composite architecture of RL, was explored as a tracking agent for the UAV-based target tracking problem. Several improvements on the original TD3 were also performed. First, the proportional�differential controller was used to boost the exploration of the TD3 in training. Second, a novel reward formulation for the UAV-based target tracking enabled a careful combination of the various dynamic variables in the reward functions. This was accomplished by incorporating two exponential functions to limit the effect of velocity and acceleration to prevent the deformation in the policy function approximation. In addition, the concept of multistage training based on the dynamic variables was proposed as an opposing concept to one-stage combinatory training. Third, an enhancement of the rewarding function by including piecewise decomposition was used to enable more stable learning behaviour of the policy and move out from the linear reward to the achievement formula. The training was conducted based on fixed target tracking followed by moving target tracking. The flight testing was conducted based on three types of target trajectories: fixed, square, and blinking. The multistage training achieved the best performance with both exponential and achievement rewarding for the fixed trained agent with the fixed and square moving target and for the combined agent with both exponential and achievement rewarding for a fixed trained agent in the case of a blinking target. With respect to the traditional proportional differential controller, the maximum error reduction rate is 86%. The developed achievement rewarding and the multistage training opens the door to various applications of RL in target tracking.

Item Type:	Article
Uncontrolled Keywords:	Navigation; reinforcement learning; target tracking; twin delayed deep deterministic policy gradient; unmanned aerial vehicles.
Subjects:	T Technology > TL Motor vehicles. Aeronautics. Astronautics > TL500-777 Aeronautics. Aeronautical engineering
Divisions:	Faculty of Electrical and Electronic Engineering > Department of Electrical Engineering
Depositing User:	Mr. Abdul Rahim Mat Radzuan
Date Deposited:	12 Apr 2022 06:55
Last Modified:	12 Apr 2022 06:55
URI:	http://eprints.uthm.edu.my/id/eprint/6913

Actions (login required)

View Item