Paper Key : IRJ************637
Author: Dr. Hemant Gianey,Shaurya Saxena,Siddhesh Darak,Jaykumar Patel,Rahul Singh
Date Published: 27 Oct 2023
Abstract
This paper addresses the challenges of processing Sensex log data, crucial for assessing the efficiency of the Indian stock market. With 30 companies in the Sensex, the Bombay Stock Exchange (BSE) maintains a vast trade information database, often stored in large text files. To effectively tackle these issues, the paper explores the utility of MapReduce and Pig. MapReduce, a parallel and distributed algorithm executed on clusters, is employed for processing and generating extensive datasets. This framework guarantees reliability, scalability, and fault tolerance. Pig, a high-level programming language, facilitates the querying and analysis of large datasets, especially those stored in distributed file systems like Hadoop Distributed File System (HDFS). Operating at a high level of abstraction, Pig streamlines the complex data processing tasks, proving invaluable for investors and analysts seeking insights into the Indian stock market's intricacies.