Lead Data Engineer

Experience Level: 4+ Years

Job Description:

We are looking for a Lead Data Engineer who is responsible for the design and development of scalable data pipelines and integrations to support continual increases in data volume and complexity. Work with analytics and business teams to understand their needs, create pipelines, improve data models that feed BI and visualization tools.

Key Responsibilities:
 Data Pipeline Development: Design, develop, and maintain robust ETL pipelines to move data from source systems to our data storage platforms. This includes data ingestion, transformation, and data quality checks to ensure accuracy and completeness.
 Data Modeling: Create and maintain data models and schemas that support business analytics, reporting, and data visualization requirements. Optimize data structures for performance and scalability.
 Data Integration: Integrate data from various internal and external sources, including databases, APIs, flat files, and streaming data. sources. Ensure seamless data flow across systems and platforms.
 Performance Optimization: Identify and address performance bottlenecks in data pipelines and database systems. Implement optimizations to improve data processing speed, scalability, and efficiency.
 Data Quality and Governance: Implement data quality checks, validation rules, and data governance policies to maintain the integrity and reliability of our data assets. Monitor data quality metrics and take proactive measures to address issues.
 Collaboration: Work closely with data analysts, data scientists, and other stakeholders to understand data requirements and deliver solutions that meet business needs. Collaborate with IT teams to ensure alignment with overall technology architecture and standards.
 Documentation and Best Practices: Document data pipelines, data models, and technical specifications. Follow best practices for software development, version control, and deployment processes.
 Continuous Improvement: Stay abreast of industry trends, emerging technologies, and best practices in data engineering. Identify opportunities for process improvement and automation to streamline data workflows and increase productivity.

Required Skills and Qualifications:
 Bachelor's degree in Computer Science, Engineering, or related field.
 4+ years of experience in data engineering or related roles.
 Proficiency in programming languages such as Python, Java, or Scala.
 Strong SQL skills and experience with relational databases (e.g., MySQL, PostgreSQL).
 Experience with data warehousing concepts and technologies (e.g., Snowflake, Redshift).
 Familiarity with big data processing frameworks (e.g., Apache Spark, Hadoop).
 Hands-on experience with ETL tools and data integration platforms.
 Knowledge of cloud platforms such as AWS, Azure, or Google Cloud Platform.
 Understanding of data modeling principles and data warehousing design patterns.
 Excellent problem-solving skills and attention to detail.
 Strong communication and collaboration skills, with the ability to work effectively in a team environment,