research-article

A tabular sarsa-based stock market agent

Authors:

Renato Arantes de Oliveira,

Heitor S. Ramos,

Daniel Hasan Dalip,

Adriano César Machado PereiraAuthors Info & Claims

ICAIF '20: Proceedings of the First ACM International Conference on AI in Finance

Article No.: 32, Pages 1 - 8

https://doi.org/10.1145/3383455.3422559

Published: 07 October 2021 Publication History

Abstract

Automated stock trading is now the de-facto way that investors have chosen to obtain high profits in the stock market while keeping risk under control. One of the approaches is to create agents employing Reinforcement Learning (RL) algorithms to learn and decide whether or not to operate in the market in order to achieve maximum profit. Automated financial trading systems can learn how to trade optimally while interacting with the market pretty much like a human investor learns how to trade. In this research, a simple RL agent was implemented using the SARSA algorithm. Next, it was tested against 10 stocks from Brazilian stock market B3 (Bolsa, Brasil, Balcão). Results from experiments showed that the agent was able to provide high profits with less risk when compared to a supervised learning agent that used a LSTM neural network.

References

[1]

Muhammad Reza Alimoradi and Ali Husseinzadeh Kashan. 2018. A league championship algorithm equipped with network structure and backward q-learning for extracting stock trading rules. Applied Soft Computing 68 (2018), 478--493.

Digital Library

[2]

Carl R Bacon. 2012. Practical risk-adjusted performance measurement. John Wiley & Sons.

[3]

Eunsuk Chong, Chulwoo Han, and Frank C Park. 2017. Deep learning networks for stock market analysis and prediction: Methodology, data representations, and case studies. Expert Systems with Applications 83 (2017), 187--205.

Digital Library

[4]

Robert W Colby and Thomas A Meyers. 1988. The encyclopedia of technical market indicators. Dow Jones-Irwin Homewood, IL,"".

[5]

Marco Corazza and Andrea Sangalli. 2015. Q-Learning and SARSA: a comparison between two intelligent stochastic control approaches for financial trading. University Ca'Foscari of Venice, Dept. of Economics Research Paper Series No 15 (2015).

[6]

Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, and Qionghai Dai. 2017. Deep Direct Reinforcement Learning for Financial Signal Representation and Trading. IEEE Transactions on Neural Networks and Learning Systems 28, 3 (March 2017), 653--664.

[7]

Alan Fan and Marimuthu Palaniswami. 2001. Stock selection using support vector machines. In IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No. 01CH37222), Vol. 3. IEEE, 1793--1798.

[8]

Damian Faustryjak, Lidia Jackowska-Strumiłło, and Michał Majchrowicz. 2018. Forward forecast of stock prices using LSTM neural networks with statistical analysis of published messages. In 2018 International Interdisciplinary PhD Workshop (IIPhDW). IEEE, 288--292.

[9]

Xiang Gao. 2018. Deep reinforcement learning for time series: playing idealized trading games. arXiv preprint arXiv:1803.03916 (2018).

[10]

Achyut Ghosh, Soumik Bose, Giridhar Maji, Narayan Debnath, and Soumya Sen. 2019. Stock Price Prediction Using LSTM on Indian Share Market. In Proceedings of 32nd International Conference on, Vol. 63. 101--110.

[11]

S. Hochreiter and J. Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (Nov. 1997), 1735--1780.

Digital Library

[12]

Yuh-Jong Hu and Shang-Jen Lin. 2019. Deep reinforcement learning for optimizing finance portfolio management. In 2019 Amity International Conference on Artificial Intelligence (AICAI). IEEE, 14--20.

[13]

Dmitry Iskrich and Dmitry Grigoriev. 2017. Generating long-term trading system rules using a genetic algorithm based on analyzing historical data. In 2017 20th Conference of Open Innovations Association (FRUCT). IEEE, 91--97.

Digital Library

[14]

WU Jia, WANG Chen, Lidong Xiong, and SUN Hongyong. 2019. Quantitative Trading on Stock Market Based on Deep Reinforcement Learning. In 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 1--8.

[15]

U JuHyok, PengYu Lu, ChungSong Kim, UnSok Ryu, and KyongSok Pak. 2020. A new LSTM based reversal point prediction method using upward/downward reversal point feature sets. Chaos, Solitons & Fractals 132 (2020), 109559.

[16]

Ha Young Kim and Chang Hyun Won. 2018. Forecasting the volatility of stock price index: A hybrid model integrating LSTM with multiple GARCH-type models. Expert Systems with Applications 103 (2018), 25--37.

[17]

Charles D Kirkpatrick II and Julie A Dahlquist. 2010. Technical analysis: the complete resource for financial market technicians. FT press.

Digital Library

[18]

Kai Lei, Bing Zhang, Yu Li, Min Yang, and Ying Shen. 2020. Time-driven feature-aware jointly deep reinforcement learning for financial signal representation and algorithmic trading. Expert Systems with Applications 140 (2020), 112872.

Digital Library

[19]

Yang Li, Wanshan Zheng, and Zibin Zheng. 2019. Deep Robust Reinforcement Learning for Practical Algorithmic Trading. IEEE Access 7 (2019), 108014--108022.

[20]

Siyuan Liu, Guangzhong Liao, and Yifan Ding. 2018. Stock transaction prediction modeling and analysis based on LSTM. In 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA). IEEE, 2787--2790.

[21]

Dome Lohpetch and David Corne. 2009. Discovering effective technical trading rules with genetic programming: Towards robustly outperforming buy-and-hold. In 2009 World Congress on Nature & Biologically Inspired Computing (NaBIC). IEEE, 439--444.

[22]

Terry Lingze Meng and Matloob Khushi. 2019. Reinforcement Learning in Financial Markets. Data 4, 3 (2019), 110.

[23]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529.

[24]

Mahdi Pakdaman Naeini, Hamidreza Taremian, and Homa Baradaran Hashemi. 2010. Stock market value prediction using neural networks. In 2010 international conference on computer information systems and industrial management applications (CISIM). IEEE, 132--136.

[25]

David MQ Nelson, Adriano CM Pereira, and Renato A de Oliveira. 2017. Stock market's price movement prediction with LSTM neural networks. In 2017 International joint conference on neural networks (IJCNN). IEEE, 1419--1426.

[26]

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. nature 529, 7587 (2016), 484.

[27]

Richard S. Sutton and Andrew G. Barto. 2018. Introduction to Reinforcement Learning (2 nd ed.). MIT Press, Cambridge, MA, USA.

Digital Library

[28]

Anderson R Tavares and Luiz Chaimowicz. 2018. Tabular Reinforcement Learning in Real-Time Strategy Games via Options. In 2018 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, 1--8.

Digital Library

[29]

Xiaofei Wang, Chenyang Wang, Xiuhua Li, Victor CM Leung, and Tarik Taleb. 2020. Federated deep reinforcement learning for internet of things with decentralized cooperative edge caching. IEEE Internet of Things Journal (2020).

[30]

Muh-Cherng Wu, Sheng-Yu Lin, and Chia-Hsin Lin. 2006. An effective application of decision tree to stock trading. Expert Systems with Applications 31, 2 (2006), 270--274.

[31]

Zhuoran Xiong, Xiao-Yang Liu, Shan Zhong, Hongyang Yang, and Anwar Walid. 2018. Practical deep reinforcement learning approach for stock trading. arXiv preprint arXiv:1811.07522 (2018).

[32]

Siyu Yao, Linkai Luo, and Hong Peng. 2018. High-Frequency Stock Trend Forecast Using LSTM Model. In 2018 13th International Conference on Computer Science & Education (ICCSE). IEEE, 1--4.

[33]

Zhenpeng Zhou, Steven Kearnes, Li Li, Richard N Zare, and Patrick Riley. 2019. Optimization of molecules via deep reinforcement learning. Scientific reports 9, 1 (2019), 1--10.

Cited By

Huang YWan XZhang LLu X(2024)A novel deep reinforcement learning framework with BiLSTM-Attention networks for algorithmic tradingExpert Systems with Applications10.1016/j.eswa.2023.122581240(122581)Online publication date: Apr-2024
https://doi.org/10.1016/j.eswa.2023.122581
Cheng XZhang JZeng YXue W(2024)MOT: A Mixture of Actors Reinforcement Learning Method by Optimal Transport for Algorithmic TradingAdvances in Knowledge Discovery and Data Mining10.1007/978-981-97-2238-9_3(30-42)Online publication date: 1-May-2024
https://doi.org/10.1007/978-981-97-2238-9_3
Sun SWang RAn B(2023)Reinforcement Learning for Quantitative TradingACM Transactions on Intelligent Systems and Technology10.1145/358256014:3(1-29)Online publication date: 24-Mar-2023
https://dl.acm.org/doi/10.1145/3582560
Show More Cited By

Index Terms

A tabular sarsa-based stock market agent
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Reinforcement learning
        Sequential decision making

Recommendations

Stock Market, Exchange Rate and Chinese Money Demand
ISME '10: Proceedings of the 2010 International Conference of Information Science and Management Engineering - Volume 02

The paper examines the long-term relationship among RMB exchange rate, stock market, interest rate, consumption, general real money balance and their dynamics from 2000 to 2009 by employing Johanson and SVAR methods. The result shows: exchange rate and ...
Stock market analysis using clustering techniques: the impact of foreign ownership on stock volatility in Vietnam
SoICT '16: Proceedings of the 7th Symposium on Information and Communication Technology

Data mining techniques have been used for various aspects of the financial market, such as prediction on stock index and price, portfolio risk management, and trend detection. In the stock market, there are a huge amount of data, including firms' ...
Stock Market Intraday Trading Using Reinforcement Learning
Multi-disciplinary Trends in Artificial Intelligence
Abstract
In this study, Reinforcement Learning (RL) techniques are used to develop trading strategies for the stock market. Conventional trading strategies rely on human intuition and the examination of historical data to make forecasts, whereas RL agents ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICAIF '20: Proceedings of the First ACM International Conference on AI in Finance

October 2020

422 pages

ISBN:9781450375849

DOI:10.1145/3383455

Conference Chair:
Tucker Balch
J.P. Morgan AI Research

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICAIF '20

Sponsor:

ACM

ICAIF '20: ACM International Conference on AI in Finance

October 15 - 16, 2020

New York, New York

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
137
Total Downloads

Downloads (Last 12 months)34
Downloads (Last 6 weeks)1

Reflects downloads up to 03 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Huang YWan XZhang LLu X(2024)A novel deep reinforcement learning framework with BiLSTM-Attention networks for algorithmic tradingExpert Systems with Applications10.1016/j.eswa.2023.122581240(122581)Online publication date: Apr-2024
https://doi.org/10.1016/j.eswa.2023.122581
Cheng XZhang JZeng YXue W(2024)MOT: A Mixture of Actors Reinforcement Learning Method by Optimal Transport for Algorithmic TradingAdvances in Knowledge Discovery and Data Mining10.1007/978-981-97-2238-9_3(30-42)Online publication date: 1-May-2024
https://doi.org/10.1007/978-981-97-2238-9_3
Sun SWang RAn B(2023)Reinforcement Learning for Quantitative TradingACM Transactions on Intelligent Systems and Technology10.1145/358256014:3(1-29)Online publication date: 24-Mar-2023
https://dl.acm.org/doi/10.1145/3582560
Do Nascimento FLima De Castro P(2023)Mitigating Risk in Machine Learning-Based Portfolio Management: A Deep Reinforcement Learning Approach2023 Congress in Computer Science, Computer Engineering, & Applied Computing (CSCE)10.1109/CSCE60160.2023.00108(628-634)Online publication date: 24-Jul-2023
https://doi.org/10.1109/CSCE60160.2023.00108
Kanzari D(2023)Context-adaptive intelligent agents behaviors: multivariate LSTM-based decision making on the cryptocurrency marketInternational Journal of Data Science and Analytics10.1007/s41060-023-00435-3Online publication date: 19-Aug-2023
https://doi.org/10.1007/s41060-023-00435-3

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents