Arvind Kumar Gopal
Data Engineer III
Chennai
Summary
Highly experienced, solution-oriented professional with 8 years of experience in Big Data-Hadoop Ecosystem and Data Warehousing. Possess extensive experience in Spark Scala/python, Hive etl development and various performance optimizations on Spark and Hadoop environment including Hive, Hdfs, sqoop and Unix shell scripting
Overview
8
8
years of professional experience
4
4
years of post-secondary education
Work History
Data Engineer III
Walmart Global Tech
03.2022 - Current
- Worked on project Total cost of Ownership where we will be reporting the total cost spent by the organization on project, cost center, product and application level
- Worked on automating one click deployment from GIT commit to GCP storage bucket using looper and concord workflows
- Worked on developing airflow dags to schedule the spark jobs to run on the dataproc clusters
- Worked on optimizing airflow, dataproc cluster and spark job to contribute on cloud cost optimization
Senior Hadoop Developer
Standard Chartered Bank Global Solutions
06.2021 - 03.2022
- Working on developing Spark-Scala code to create etl process on top of credit analytics data in bank for each system and store them in flattened table segregated by country wise partitions, which will be used by downstream users for analysis
Senior Software Engineer
Tiger Analytics
06.2020 - 06.2021
- Worked for leading Insurance Client, developed Pyspark etl process to form flattened table that contains all required fields and created data science model on top of it to predict customer who are most likely to surrender their policies using XGBoost Algorithm
Associate - Projects
Cognizant Technology Solutions
09.2019 - 06.2020
- Worked for leading Canada Banking Client, developed spark program that generates control metadata file (XML) from details provided in OK file, developed spark program that ingests psv data files into hive table using above formed metadata, developed spark sql program to perform transformations
System Engineer
Tata Consultancy Services
06.2016 - 09.2019
- Worked for leading US Travel Transport and Hospitality Client, developed a tool to migrate data between netezza and Hadoop , responsible for overall process of data migration to handle various scenarios like Data cleaning, Data Integration and Synonym approaches to handle data discrepancies and data corruptions.
- Worked for leading US Travel Transport and Hospitality Client, developed spark program, shell scripts and hqls to perform various etl transformations on top of clickstream data and store them as a flattened hive tables, which will be used by downstream tableau users to create dashboard for business.
Education
Bachelor of Engineering - Computer Science and Engineering
Bannari Amman Institute of Technology
Sathyamangalam 08.2012 - 04.2016
Skills
Spark - Scala
Timeline
Data Engineer III
Walmart Global Tech
03.2022 - Current
Senior Hadoop Developer
Standard Chartered Bank Global Solutions
06.2021 - 03.2022
Senior Software Engineer
Tiger Analytics
06.2020 - 06.2021
Associate - Projects
Cognizant Technology Solutions
09.2019 - 06.2020
System Engineer
Tata Consultancy Services
06.2016 - 09.2019
Bachelor of Engineering - Computer Science and Engineering
Bannari Amman Institute of Technology
08.2012 - 04.2016
Similar Profiles
Prakhar GuptaPrakhar Gupta
Software Engineer III at Walmart Global TechSoftware Engineer III at Walmart Global Tech
Jaita SahaJaita Saha
Staff, Technical Product Management at Walmart Global TechStaff, Technical Product Management at Walmart Global Tech
Vijay MishraVijay Mishra
Senior Software Engineer at Walmart Global TechSenior Software Engineer at Walmart Global Tech