Highly skilled Big Data Engineer with 2+ years of hands-on experience in installing, configuring, monitoring, and using Hadoop ecosystem components like Hadoop Map-Reduce, HDFS, HBase, python Hive, Sqoop. Proven developer in core Scala and SQL.
Experience
Big Data Developer
- Handled the development of Credit Card Fraud Detection project, partnering with the client to understand customer spending patterns, and developing ETL processes for data loading using Sqoop. Ensured accurate identification and prevention of fraudulent activities, protecting the financial interests of the company and customers.
- Contributed to the Customer Credit Card Upgradation project, migrating data and conducting data cleaning, shaping, and transformations using SQL server and ETL. Improved customer experience by seamlessly upgrading credit card accounts and ensuring accurate and up-to-date customer information.
Projects
credit card fraud detection
- HDFS
- Sqoop
- Hive
- HBase
- Spark
- MySQL
- Amazon RDS
- Kafka
- Airflow
- pyspark
- python
- Ingested transactional and batch data from client using Sqoop tool and stored it in HDFS for efficient processing. Improved data storage and access efficiency.
- Cleaned, enriched, and processed data using Hive and Spark framework, resulting in improved data quality and accuracy. Enhanced data quality and accuracy through data processing and enrichment.
- Optimized pipelines to reduce resource utilization and increased processing speed, enabling faster data analysis. Improved data processing efficiency and speed, leading to faster data analysis.
- Stored processed data in HBase, facilitating easy access for analysts to analyze and derive insights. Enabled easy data access and analysis for analysts through data storage in HBase.
- Ensured data integrity and security throughout the entire data processing pipeline, maintaining high levels of data quality and privacy. Maintained high standards of data quality and privacy through data integrity and security measures throughout the processing pipeline.
At NSS ITC, partnered with the client to handle credit card analytics, understanding customer spending behavior, categorizing customers, and improving purchasing habits. Developed ETL processes for data loading from multiple sources using Sqoop, stored and processed data using HDFS, Hive, and Spark framework, and optimized pipelines for increased processing speed and efficiency.
customer credit card upgradation
- SSIS
- Informatica
- SQL server
- Migrated the client application to a newer version and migrated the data.
- Utilized SQL server and ETL to identify active and inactive accounts.
- Conducted data cleaning, shaping, and transformations.
Upgraded the client application, migrating data from older to newer version, utilizing SQL server and ETL for data migration and applying business logic for account identification. Conducted data cleaning, shaping, and transformations, ensuring seamless application migration.