Adaptive Operator and Scaling Factor Selection in Differential Evolution using Parametrized Reinforcement Learning

Abstract views: 2 ,

Kadek Gemilang Santiyuda
Putu Sugiartawan
Gede Agus Santiago
Ni Nengah Dita Ardriani
Moch Ilham Nur Kafiyanna

Abstract

Mutation strategy selection along with parameter settings are well known challenges in enhancing the performance of differential evolution (DE). In this paper, we propose to solve these problems as a parametrized action Markov decision process. A multi-pass deep Q-network (MP-DQN) is used as the reinforcement learning method in the parametrized action space. The architecture of MP-DQN comprises an actor network and a Q-network, both trained offline. The networks’ weights are trained based on the samples of states, actions and rewards collected on every DE iterations. We use 99 features to describe a state of DE and experiment on 4 reward definitions. A benchmark study is carried out with functions from CEC2005 to compare the performance of the proposed method to baseline DE methods without any parameter control, with random scaling factor, and to other DEs with adaptive operator selection methods, as well as to the two winners of CEC2005. The results show that DE with MP-DQN parameter control performs better than the baseline DE methods and obtains competitive results compared to the other methods.

Downloads

Download data is not yet available.
How to Cite
Santiyuda, K., Sugiartawan, P., Santiago, G., Ardriani, N. N., & Kafiyanna, M. (2025). Adaptive Operator and Scaling Factor Selection in Differential Evolution using Parametrized Reinforcement Learning. Jurnal Sistem Informasi Dan Komputer Terapan Indonesia (JSIKTI), 7(3), 40-50. https://doi.org/10.33173/jsikti.206

References

[1] M. E. Abdual-Salam, H. M. Abdul-Kader, and W. F. Abdel-Wahed, “Comparative study between Differential Evolution and Particle Swarm Optimization algorithms in training of feed-forward neural network for stock price prediction,” Proc. 7th Int. Conf. on Informatics and Systems (INFOS), pp. 1–8, 2010.
[2] A. Auger and N. Hansen, “Performance evaluation of an advanced local search evolutionary algorithm,” Proc. IEEE Congress on Evolutionary Computation, vol. 2, pp. 1777–1784, 2005.
[3] A. Auger and N. Hansen, “A restart CMA evolution strategy with increasing population size,” Proc. IEEE Congress on Evolutionary Computation, vol. 2, pp. 1769–1776, 2005.
[4] M. Baioletti, G. Di Bari, V. Poggioni, and M. Tracolli, “Can Differential Evolution Be an Efficient Engine to Optimize Neural Networks?” in Machine Learning, Optimization, and Big Data, Springer, pp. 401–413, 2018.
[5] C. J. Bester, S. D. James, and G. D. Konidaris, “Multi-pass Q-networks for deep reinforcement learning with parameterised action spaces,” arXiv preprint arXiv:1905.04388, 2019.
[6] Y. Chabane and A. Ladjici, “Differential evolution for optimal tuning of power system stabilizers to improve power systems small signal stability,” Proc. 5th Int. Conf. on Systems and Control (ICSC), pp. 84–89, 2016.
[7] F. Chen, Y. Gao, Z. Q. Chen, and S. F. Chen, “SCGA: Controlling genetic algorithms with Sarsa(0),” Proc. Int. Conf. on Computational Intelligence for Modelling, Control and Automation (CIMCA), pp. 1177–1182, 2005.
[10] A. E. Eiben et al., “Parameter Control in Evolutionary Algorithms,” in Parameter Setting in Evolutionary Algorithms, Springer, pp. 19–46, 2007.
[11] A. E. Eiben and S. K. Smit, “Evolutionary Algorithm Parameters and Methods to Tune Them,” in Autonomous Search, Springer, pp. 15–36, 2012.
[12] T. Eltaeib and A. Mahmood, “Differential evolution: A survey and analysis,” Applied Sciences, vol. 8, no. 10, 2018.
[13] Á. Fialho et al., “Dynamic Multi-Armed Bandits and Extreme Value-Based Rewards for Adaptive Operator Selection in Evolutionary Algorithms,” in Learning and Intelligent Optimization, Springer, pp. 176–190, 2009.
[14] Á. Fialho et al., “Comparison-Based Adaptive Strategy Selection with Bandits in Differential Evolution,” in Parallel Problem Solving from Nature (PPSN XI), Springer, pp. 194–203, 2010.
[15] Á. Fialho, M. Schoenauer, and M. Sebag, “Toward Comparison-Based Adaptive Operator Selection,” Proc. 12th Genetic and Evolutionary Computation Conf., pp. 767–774, 2010.
[16] D. E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning, Addison-Wesley, 1989.
[17] D. E. Goldberg, “Probability matching, the magnitude of reinforcement, and classifier system bidding,” Machine Learning, vol. 5, no. 4, pp. 407–425, 1990.
[18] W. Gong, Á. Fialho, and Z. Cai, “Adaptive Strategy Selection in Differential Evolution,” Proc. 12th Genetic and Evolutionary Computation Conf., pp. 409–416, 2010.
[19] W. Gong et al., “Adaptive strategy selection in differential evolution for numerical optimization: An empirical study,” Information Sciences, vol. 181, no. 24, pp. 5364–5386, 2011.
[21] H. van Hasselt, A. Guez, and D. Silver, “Deep Reinforcement Learning with Double Q-Learning,” Proc. 30th AAAI Conf. Artificial Intelligence, pp. 2094–2100, 2016.
[22] M. Hausknecht and P. Stone, “Deep reinforcement learning in parameterized action space,” Proc. 4th Int. Conf. on Learning Representations (ICLR), 2016.
[24] G. Karafotias, A. E. Eiben, and M. Hoogendoorn, “Generic parameter control with reinforcement learning,” Proc. GECCO 2014, pp. 1319–1326, 2014.
[25] G. Karafotias, M. Hoogendoorn, and A. E. Eiben, “Evaluating Reward Definitions for Parameter Control,” in Applications of Evolutionary Computation, Springer, pp. 667–680, 2015.
[26] J. Kennedy and R. Eberhart, “Particle swarm optimization,” Proc. IEEE Int. Conf. Neural Networks, vol. 4, pp. 1942–1948, 1995.
[29] W. Masson, P. Ranchod, and G. Konidaris, “Reinforcement Learning with Parameterized Actions,” Proc. AAAI Conf. Artificial Intelligence, pp. 1934–1940, 2016.
[30] E. Mezura-Montes, J. Velázquez-Reyes, and C. A. Coello Coello, “A Comparative Study of Differential Evolution Variants for Global Optimization,” Proc. 8th Genetic and Evolutionary Computation Conf., pp. 485–492, 2006.
[33] V. Nair and G. E. Hinton, “Rectified Linear Units Improve Restricted Boltzmann Machines,” Proc. 27th Int. Conf. Machine Learning, pp. 807–814, 2010.
[34] J. E. Pettinger and R. M. Everson, “Controlling Genetic Algorithms with Reinforcement Learning,” Proc. 4th Genetic and Evolutionary Computation Conf., pp. 692, 2002.
[35] A. K. Qin and P. N. Suganthan, “Self-adaptive differential evolution algorithm for numerical optimization,” Proc. IEEE Congress on Evolutionary Computation, vol. 2, pp. 1785–1791, 2005.
[36] Y. Sakurai et al., “A method to control parameters of evolutionary algorithms by using reinforcement learning,” Proc. 6th Int. Conf. on Signal Image Technology and Internet Based Systems (SITIS), pp. 74–79, 2010.
[37] M. Sharma, A. Komninos, and M. López-Ibáñez, “Deep reinforcement learning based parameter control in differential evolution,” Proc. GECCO, 2019.
[38] M. Sharma, M. López-Ibáñez, and D. Kazakov, “Performance Assessment of Recursive Probability Matching for Adaptive Operator Selection in Differential Evolution,” in PPSN XV, Springer, pp. 321–333, 2018.
[39] M. Sharma, M. López-Ibáñez, and D. Kazakov, “Unified Framework for the Adaptive Operator Selection of Discrete Parameters,” arXiv preprint, 2020.
[40] R. Storn, “On the usage of differential evolution for function optimization,” Proc. North American Fuzzy Information Processing, pp. 519–523, 1996.
[41] R. Storn and K. Price, “Differential Evolution—A Simple and Efficient Heuristic for global Optimization over Continuous Spaces,” J. Global Optimization, vol. 11, no. 4, pp. 341–359, 1997.
[42] B. Subudhi and D. Jena, “A differential evolution based neural network approach to nonlinear system identification,” Applied Soft Computing, vol. 11, no. 1, pp. 861–871, 2011.
[43] K. R. Sudha, “Design of differential evolution algorithm-based robust fuzzy logic power system stabiliser using minimum rule base,” IET Generation, Transmission & Distribution, vol. 6, no. 2, pp. 121–132, 2012.
[44] P. N. Suganthan et al., “Problem definitions and evaluation criteria for the CEC 2005 special session on real-parameter optimization,” KanGAL Report 2005005, 2005.
[45] T. H. Teng, S. D. Handoko, and H. C. Lau, “Self-organizing neural network for adaptive operator selection in evolutionary search,” Lecture Notes in Computer Science, vol. 10079, pp. 187–202, 2016.
[46] D. Thierens, “An Adaptive Pursuit Strategy for Allocating Operator Probabilities,” Proc. 7th Genetic and Evolutionary Computation Conf., pp. 1539–1546, 2005.
[47] J. Vesterstrom and R. Thomsen, “A comparative study of differential evolution, particle swarm optimization, and evolutionary algorithms on numerical benchmark problems,” Proc. IEEE Congress on Evolutionary Computation, vol. 2, pp. 1980–1987, 2004.
[48] J. Xiong et al., “Parameterized deep Q-networks learning: Reinforcement learning with discrete-continuous hybrid action space,” arXiv preprint arXiv:1810.06394, 2018.
[49] X.-S. Yang and S. Deb, “Cuckoo Search via Lévy Flights,” Proc. IEEE World Congress on Nature and Biologically Inspired Computing, pp. 210–214, 2009.
[50] J. Zhang and A. C. Sanderson, “JADE: Adaptive differential evolution with optional external archive,” IEEE Transactions on Evolutionary Computation, vol. 13, no. 5, pp. 945–958, 2009.
[51] Y.-X. Zhao et al., “An Improved Differential Evolution Algorithm for Maritime Collision Avoidance Route Planning,” Abstract and Applied Analysis, vol. 2014, Article ID 614569, 2014.