Data Engineers (Python, Spark,SQL, Azure DataBricks)

Job Description: We are seeking an experienced and dynamic Senior Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, with expertise in Python, PySpark, and SQL. This role involves developing and maintaining scalable data pipelines, ensuring data integrity, and optimizing data retrieval processes. You will be working closely with our data science and analytics teams to support various data initiatives and help us make data-driven decisions.

Responsibilities:

Design, construct, install, test, and maintain highly scalable data management systems.

Develop high-performance data processing pipelines using Python and PySpark.

Write complex SQL queries for data manipulation and retrieval.

Ensure data quality and integrity in pipelines and database systems.

Collaborate with data analysts, scientists, and other stakeholders to understand data needs.

Optimize and improve existing data structures to enhance performance and. scalability.

Implement data security and privacy policies.

Stay up-to-date with advancements in data engineering technologies and best practices.

Mentor junior data engineers and participate in code reviews.

Requirements:

Bachelors or Masters degree in Computer Science, Engineering, or a related field.

Minimum of 5 years of experience in a data engineering role.

Proficiency in Python and PySpark for data processing.

Strong experience with SQL and relational databases, as well as NoSQL databases.

Experience building and optimizing big data pipelines, architectures, and data sets.

Strong analytic skills related to working with unstructured datasets.

Experience with data pipeline and workflow management tools.

Experience with cloud services (AWS, Azure, GCP) is preferred.

Knowledge of machine learning and data science is a plus.

Excellent problem-solving skills and attention to detail.

Strong communication and teamwork skills,