Senior Big Data Engineer

Job Summary

We are looking for a highly skilled Senior Spark Engineer with 5–7+ years of hands-on experience in big data development. The ideal candidate will have strong expertise in Spark (batch and streaming), PySpark, Hadoop, Hive, SQL, and data engineering, with experience in building and deploying end-to-end data pipelines. Knowledge of Kafka, Airflow, Databricks, and cloud-based big data solutions is highly desirable.

Job Roles & Responsibilities

• Design, develop, and optimize batch and streaming data pipelines using Apache Spark (Core, SQL, Streaming).
• Build scalable ETL workflows with PySpark and migrate existing pipelines to Spark-based architectures.
• Work with Hadoop, HDFS, YARN, and integrate them with streaming sources like Kafka.
• Implement and support real-time data processing using DStreams and Structured Streaming.
• Write robust code in Scala, Python, or Java for data transformations and automation.
• Use Jupyter Notebooks or similar tools for exploration, development, and debugging.
• Orchestrate and schedule jobs using Airflow or equivalent workflow engines.
• Ensure performance tuning and fault tolerance of big data systems.

Cultural Expectations

• Minimum 5 years of experience in big data engineering with expertise in Spark and Hadoop ecosystems.
• Proficient in data engineering practices including ETL development and real-time data streaming.
• Strong programming skills in Scala, Python, and Java.
• Experience working with Kafka and implementing Spark Streaming solutions.
• Familiarity with tools like Jupyter Notebooks, Airflow, and CI/CD best practices.
• Strong problem-solving skills, team collaboration, and communication abilities.
• Good to Have Skills:
• Experience with Flink and Kudu for advanced streaming use cases.
• Knowledge of CI/CD automation for data workflows.
• Exposure to Apache NiFi for data ingestion and transformation.
• Familiarity with Informatica workflow migration projects.

Hiring Process

• Live coding task to evaluate coding proficiency.
• Technical interview with senior member from the technical team.
• Final interview with Tech Lead/CTO.