Big Data Engineer (Kochi)

Job Description

Position is based in Kochi.

Key Responsibilities
 Lead and manage assigned projects, providing guidance and support to others on team you are supporting.
 Designing and implementing highly performant data ingestion pipelines from multiple sources
 Evaluate and choose appropriate data engineering tools and technologies where necessary.
 Optimize data infrastructure and pipelines for maximum efficiency.
 Working with event based / streaming technologies to ingest and process data
 Build large scale data migration scripts to migrate big data from native files in OSI PI and Timescale database into Databricks Data Lake
 Develop automated CICD pipelines for optimally deploying data pipelines and data science workloads.
 Contribute to development and maintain various streamlit applications
 Collaborate closely and independently with data scientists to provide data for analytics and machine learning.
 Implement best practices for data engineering and promote a culture of continuous improvement.

Qualifications And Skills
 Strategic thinking, excellent communication skills and the ability to align data engineering with business goals.
 Advanced understanding with spark and/or Databricks(preferred), including spark cluster tuning.
 Great understanding of distributed systems and partitioning.
 Knowledge in Databricks and AWS data analytics tooling.
 Knowledge of Data Science life cycle
 Advanced knowledge in SQL and other languages such as Python, Scala, R or Java
 Experience with developing python streamlit applications.
 Visualization/reporting tools such as Tableau, Spotfire, Grafana, etc.
 Understanding of CI/CD practices and technologies. GitHub and GitHub Actions(preferred), Jenkins, TeamCity, Udeploy, etc.

Optional But Beneficial Skills
 Knowledge of Terraform
 Snowflake
 Some experience in building ML/AI models (worked with packages such as scikit learn, tensorflow, openai, etc.)
 Kafka
 Matillion

Work Environment And Conditions
 Flexible and ambition to learn new skills quickly
 Ability to work independently,