THE MECHANISM FOR CREATION OF EVENT-DRIVEN APPLICATIONS BASED ON WOLFRAM MATHEMATICA AND APACHE KAFKA

Keywords: event-driven applications, queue manager, mathematical processor, saving resources and funds, Kafka, Mathematica

Abstract

The article is devoted to the research and development of the mechanism of interaction between Wolfram Mathematica programs and Apache Kafka queue to provide the ability to build event-driven applications based on it. The subject of the research is the practical principles of building a mechanism for interaction between Wolfram Mathematica and Apache Kafka. The purpose of the article is to develop and substantiate practical recommendations regarding the formation of a mechanism for publishing messages to the Apache Kafka queue and reading messages from it for programs of the mathematical processor Wolfram Mathematica, which will make it possible to build event-driven applications. Tasks: to determine the mechanism of such interaction, prove the choice of tools for its implementation, create and test the obtained results. The research used the following tools: Apache Kafka, Kafkacat, the method of developing the Wolfram Mathematica package. The results of the research: the mechanism of interaction between Wolfram Mathematica and Apache Kafka was determined and the corresponding toolkit was created on its basis in the form of two Mathematica packages, which are built on using Apache Kafka as a queue client and third-party Kafkacat software, respectively. It is shown that the first option is less reliable and consumes much more computer resources during operation. It has been demonstrated that the Mathematica processor is currently not suitable in its pure form for real-time data analysis. Recommendations are given regarding the use of built-in compilation functions to increase the speed of such processing. Conclusions. Practical recommendations have been developed and substantiated regarding the formation of the mechanism of interaction between the Wolfram Mathematica mathematical processor and the Apache Kafka queue manager for the possibility of working in two directions with the queue: publishing messages and reading them. A toolkit for such interaction in the form of Mathematica packages has been created, their capabilities have been demonstrated, as well as comparison with each other. The economic benefit of using the described tools is shown. Future ways of its improvement are given.

Downloads

Download data is not yet available.

Author Biography

Denis Zolotariov
  PhD (Physics and Mathematics Sciences)

References

Ed-daoudy, A., Maalmi, K. (2019), "A new Internet of Things architecture for real-time prediction of various diseases using machine learning on big data environment", Journal of Big Data, Vol. 6, No. 104. DOI: https://doi.org/10.1186/s40537-019-0271-7

Mahapatra, T. (2020), "Composing high-level stream processing pipelines", Journal of Big Data, Vol. 7, No. 81. DOI: https://doi.org/10.1186/s40537-020-00353-2

Nasiri, H., Nasehi, S., Goudarzi, M. (2019), "Evaluation of distributed stream processing frameworks for IoT applications in Smart Cities", Journal of Big Data, Vol. 6, No. 52. DOI: https://doi.org/10.1186/s40537-019-0215-2

Jung, S., Kim, Y., Hwang, E. (2018), "Real-time car tracking system based on surveillance videos", EURASIP Journal on Image and Video Processing, Vol. 2018, No. 133. DOI: https://doi.org/10.1186/s13640-018-0374-7

Ismail, A., Truong, H. L., Kastner, W. (2019), "Manufacturing process data analysis pipelines: a requirements analysis and survey", Journal of Big Data, Vol. 6, No. 1. DOI: https://doi.org/10.1186/s40537-018-0162-3

Kim, Y. K., Kim, Y., Jeong, C. S. (2018), "RIDE: real-time massive image processing platform on distributed environment", EURASIP Journal on Image and Video Processing, Vol. 2018, No. 39. DOI: https://doi.org/10.1186/s13640-018-0279-5

Kolajo, T., Daramola, O., Adebiyi, A. (2019), "Big data stream analysis: a systematic literature review", Journal of Big Data, Vol. 6, No. 47. DOI: https://doi.org/10.1186/s40537-019-0210-7

GitHub (2020), "Mathworks-ref-arch/matlab-apache-kafka: MATLAB Interface for Apache Kafka", available at : https://github.com/mathworks-ref-arch/matlab-apache-kafka (last accessed 10 December 2020).

Rehman, S., Idrees, M., Shah, R. A. et al. (2019), "Suction/injection effects on an unsteady MHD Casson thin film flow with slip and uniform thickness over a stretching sheet along variable flow properties", Boundary Value Problems, Vol. 2019, No. 26. DOI: https://doi.org/10.1186/s13661-019-1133-0

Ghorbani, M. A., Singh, V. P., Sivakumar, B. et al. (2017), "Probability distribution functions for unit hydrographs with optimization using genetic algorithm", Applied Water Science, Vol. 7, P. 663–676. DOI: https://doi.org/10.1007/s13201-015-0278-y

DeCanio, S. J. (2020), "Can an AI learn political theory?", AI Perspectives, Vol. 2, Article 3. DOI: https://doi.org/10.1186/s42467-020-00007-2

You, X., Chen, D. R. (2018), "A new sequence convergent to Euler–Mascheroni constant", Journal of Inequalities and Applications, Vol. 2018, Article 75. DOI: https://doi.org/10.1186/s13660-018-1670-6

Mathematica Stack Exchange (2020), "Networking - Connect Mathematica to message broker - Kafka, NATS or mosquitto", available at: https://mathematica.stackexchange.com/questions/199848/connect-mathematica-to-message-broker-kafka-nats-or-mosquitto (last accessed 10 December 2020).

Mathematica Stack Exchange (2020), "Networking - MQTTLink TopicSubscribe[] cannot receive messages", available at: https://mathematica.stackexchange.com/questions/211940/mqttlink-topicsubscribe-cannot-receive-messages (last accessed 10 December 2020).

Apache Kafka (2020), "Apache Kafka", available at: https://kafka.apache.org/documentation/#os (last accessed 10 December 2020).

GitHub (2020), "Edenhill/kafkacat: Generic command line non-JVM Apache Kafka producer and consumer", available at: https://github.com/edenhill/kafkacat (last accessed 10 December 2020).

Microsoft Docs (2020), "An overview on the Windows Subsystem for Linux", available at: https://docs.microsoft.com/en-us/windows/wsl/ (last accessed 10 December 2020).

Stack Overflow (2020), "Kafka bootstrap-servers vs zookeeper in kafka-console-consumer", available at: https://stackoverflow.com/questions/41774446/kafka-bootstrap-servers-vs-zookeeper-in-kafka-console-consumer (last accessed 10 December 2020).

Stack Overflow (2020), "Apache kafka - bootstrap-server vs zookeeper params in consumer console", available at: https://stackoverflow.com/questions/53954877/bootstrap-server-vs-zookeeper-params-in-consumer-console (last accessed 10 December 2020).

Zolotariov, D. (2020), "The distributed system of automated computing based on cloud infrastructure", Innovative Technologies and Scientific Solutions for Industries, No. 4 (14), P. 47–55. DOI: https://doi.org/10.30837/ITSSI.2020.14.047

Zolotariov, D. A. (2020), "Automation and optimization of scientific and engineering calculations in Wolfram Mathematica", Kharkiv : FOP Panov A. M. ISBN: 978-617-7859-36-8 [In Ukrainian].


Abstract views: 14
PDF Downloads: 34
Published
2021-03-28
How to Cite
Zolotariov, D. (2021) “THE MECHANISM FOR CREATION OF EVENT-DRIVEN APPLICATIONS BASED ON WOLFRAM MATHEMATICA AND APACHE KAFKA”, INNOVATIVE TECHNOLOGIES AND SCIENTIFIC SOLUTIONS FOR INDUSTRIES, (1 (15), pp. 53-58. doi: 10.30837/ITSSI.2021.15.053.