AWS Data Engineer - Azure Databricks

We have Immediate Openings on AWS lead with Databricks

Location : anywhere in India

Key Responsibilities
 Design and Implementation:
 Design, build, and maintain scalable data pipelines using Databricks on AWS.
 Design & Develop custom ETL frameworks and processes to ingest and transform data from various sources.
 Implement Unity Catalog to manage and secure data assets
 Implement best practices for data governance, data quality, and data security.
 Collaboration:
 Work closely with data scientists, analysts, and business stakeholders to understand data requirements.
 Collaborate with DevOps teams to automate deployment and monitoring of data pipelines.
 Optimization and Troubleshooting:
 Optimize and troubleshoot performance issues in Databricks and AWS environments.
 Conduct root cause analysis of data pipeline failures and implement corrective actions.
 Monitoring and Maintenance:
 Monitor data pipelines and infrastructure to ensure high availability and performance.
 Maintain comprehensive documentation of data workflows and architecture.

Required Qualifications

Education :
 Bachelor's degree in Computer Science, Information Technology, or a related field (Master's degree Overall 10 years with 3+ years of experience working with Databricks and AWS.
 Experience is setting up Unity Catalog
 Proven experience in building and managing large-scale data pipelines.
 Experience with Spark, SQL, and Python.

Technical Skills
 Proficient in Databricks, AWS (S3, EC2, , RDS, Glue, etc.), and big data technologies.
 Strong programming skills in Python and SQL.
 Experience with data warehousing solutions
 Familiarity with CI/CD pipelines and tools like Jenkins or GitLab CI/CD.

Soft Skills
 Excellent problem-solving skills and attention to detail.
 Strong communication skills, both written and verbal.

(ref:hirist.tech),