Senior Big Data Engineer

Key Responsibilities

• Design and develop scalable data pipelines and solutions using Python and PySpark.
• Utilize big data technologies such as Hadoop, Spark, Kafka, or similar tools for processing and analyzing large datasets.
• Develop and maintain ETL processes to extract, transform, and load data into data lakes or warehouses.
• Collaborate with data engineers and scientists to implement machine learning models and algorithms.
• Optimize and tune data processing workflows for performance and efficiency.
• Implement data governance and security measures to ensure data integrity and privacy.
• Create and maintain documentation for data pipelines, workflows, and processes.
• Provide technical leadership and mentorship to junior team members.

 

Skills Required

• Python: Proficiency in Python programming for data manipulation and analysis.
• PySpark: Experience with PySpark for processing large-scale data.
• Big Data Technologies: Strong understanding and practical experience with big data technologies such as Hadoop, Spark, Kafka, etc.
• ETL Processes: Knowledge of designing and implementing ETL processes for data integration.
• Data Processing: Ability to work with large datasets, perform data cleansing, transformations, and aggregations.
• Machine Learning: Familiarity with machine learning concepts and experience implementing ML models.
• Data Governance: Understanding of data governance principles and experience implementing data security measures.
• Documentation: Ability to create clear and concise documentation for data pipelines and processes.
• Collaboration: Strong teamwork and collaboration skills to work with cross-functional teams.
• Problem-Solving: Analytical and problem-solving skills to optimize data workflows and processes.
• Leadership: For senior roles, the ability to provide technical leadership, mentorship, and guidance to junior team members.
• SQL: Knowledge of SQL for querying and manipulating data in databases.
• Cloud Platforms: Experience with cloud platforms like AWS, Azure, or Google Cloud is a plus.