Skill diminta
AWSAccess ControlAirflowApacheAzureBigQueryCI/CDGCPGitPythonREST APISQLSnowflakeSpark
Deskripsi
What You'll Do
Design, develop, and optimize scalable ETL/ELT pipelines to ingest, transform, and process large volumes of structured and unstructured data.
- Build and maintain enterprise-grade data warehouses, data lakes, and lakehouse architectures.
- Develop reliable batch and real-time streaming data pipelines using modern data engineering technologies.
- Implement data quality validation, monitoring, governance, lineage, and observability processes.
- Optimize SQL queries, pipeline performance, storage costs, and processing efficiency.
Work closely with Data Scientists, BI Developers, Machine Learning Engineers, Product Owners, and Software Engineers to deliver data-driven solutions.
- Integrate data from APIs, enterprise systems, ERP platforms, SaaS applications, and third-party sources.
- Develop reusable data models and transformation frameworks following best engineering practices.
- Participate in cloud migration and modernization initiatives.
- Support AI and Machine Learning teams by building scalable feature engineering and model-serving data pipelines.
- Implement CI/CD, Infrastructure as Code, and automated deployment strategies for data platforms.
- Mentor junior engineers and contribute to architecture discussions.
- Required Skills & Experience
- Bachelor's degree in Computer Science, Information Technology, Engineering, Mathematics, or related discipline.
- 5+ years of professional Data Engineering experience.
- Strong SQL skills with experience optimizing complex queries.
- Strong Python programming experience.
- Hands-on experience with ETL/ELT development.
- Experience working with large-scale data processing frameworks such as Apache Spark.
- Experience using orchestration tools such as Apache Airflow.
- Strong understanding of dimensional modeling, star schema, snowflake schema, normalization, and denormalization.
- Experience with cloud data platforms on AWS, Azure, or Google Cloud Platform.
Experience with one or more modern data warehouse technologies
- Snowflake
- BigQuery
- Redshift
- Synapse
- Databricks
- Experience building REST API integrations.
- Experience using Git and CI/CD pipelines.
- Understanding of data governance, security, and access control.