Recently, a research group led by Prof. Zhao (Port, Coastal & Offshore Engineering) made a significant advance in the interdisciplinary research field of artificial intelligence and computational fluid dynamics. The related academic papers “Sloshing suppression with active controlled baffles through deep reinforcement learning-expert demonstrations-behavior cloning process” has been published in 《Physics of Fluids》，an established academic journal in the field of fluid mechanics.

With the fast development of artificial intelligence (AI), deep reinforcement learning (DRL) has attracted more and more attention. The DRL paradigm combines the advantages of deep neural networks (DNNs) and reinforcement learning algorithms, allowing the use of high-dimensional state spaces and possessing excellent feature extraction capabilities. The interested reader can refer to the recent review articles for more details on the DRL paradigm itself. DRL has been successfully deployed to resolve tasks with unprecedented efficiency in many domains such as controlling the dynamics of complex robots, language processing, playing games, and even autonomous cars. Recently, in the field of fluid mechanics, DRL has become a new tool to explore control strategies, shown to be superior to previous techniques in some cases. The framework of DRL+CFD is shown in Fig.1.Although DRL has great potential, the literature on its application in active flow control (AFC) is still limited. Until now, only a few applications have been reported that take advantage of such learning methods.

This paper presents an effective paradigm to make full use of both Deep Reinforcement Learning (DRL) and expert knowledge to find an optimal control strategy. As shown in Fig.2, the paradigm consists of three parts: DRL, expert demonstrations, and behavior cloning. It is the first time that the proposed paradigm is used for suppressing tank sloshing with two active controlled horizontal baffles. Meanwhile, a self-developed computational fluid dynamics (CFD) solver is used to simulate the environment of tank sloshing. For direct DRL, both the proximal policy optimization agent and the twin delayed deep deterministic policy gradient agent are tested for performing learning. The strategies obtained by different algorithms may not be uniform even for the same environment. Then, we derive a simplified parametric control policy informed from direct DRL. Finally, DRL with behavior cloning is used to optimize the simplified parametric control policy.

Fig.1 The framework of DRL+CFD

Fig.2 An effective paradigm: DRL, expert demonstrations, and behavior cloning

After training, the agent can actively control the baffles and reduce sloshing by 81.48%. The Fourier analysis of the surface elevations pinpoints that the aim of the control strategy obtained by DRL with behavior cloning is to disperse the wave energy and change the sloshing frequency of the tank through fast oscillation of baffles. This provides an idea to suppress sloshing, similar to forcing waves to disassemble ahead of time. The experience and insights gained from this study indicate that the future development direction of DRL + CFD is how to couple DRL, expert demonstrations, and behavior cloning. The part of results is shown in Fig.3.

Fig.3 The time series of the surface elevations of the tank (left);

Velocity vectors and vortex contour of sloshing flow (right)

At the same time, the related academic paper PREDICTION OF WAVE TRANSMISSION OF CULVERT BREAKWATER BASED ON CNN has been published in 《Chinese Journal of Theoretical and Applied Mechanics》which is a famous academic journal in china.

With the development of machine learning technology, the traditional hydrodynamic problems ushered in a new solution concept which has attracted many attentions in the field of physics and engineering. In this paper, based on the convolutional neural network (CNN), the wave transmission characteristics of the culvert breakwater under different incident and different opening conditions are predicted. As shown in Fig.4, the corresponding training data set is generated by a CFD model for convolution neural network training. The CFD results are compared with physical results for validation. After the data mapping relationship between different working conditions and the corresponding wave transmission results are established, the wave transmission coefficient and wave characteristics of transmission wave under the new working conditions can be predicted rapidly. The results show that the trained convolutional neural network can calculate the corresponding results within 10 milliseconds with a relatively high accuracy. This study can provide a new idea for solving the problem of interaction between waves and coastal structures, and is of importance in engineering application.

Fig.4 The structure of the convolution neural network

These research achievements are supported by the National Natural Science Foundation of China; The numerical calculation is supported by high performance computing cluster in Zhoushan campus.

(Source: Division of Port, Coastal & Offshore Engineering of Ocean College)