Skill diminta
AccurateAirflowApacheApache KafkaBigQueryEnglishMicroservicesPostgreSQLPythonSQLSnowflakeSparkdbt
Deskripsi
- Key Responsibilities
- Design and develop microservices for ingesting data from multiple sources, both internal and external.
- Build and manage end-to-end data pipelines, covering data extraction, cleaning, transformation, and loading (ETL/ELT).
- Implement and maintain data quality monitoring processes to ensure data used by AI systems is accurate, consistent, and reliable.
- Develop data pipelines that serve as input for AI systems, enabling AI-generated outputs to be integrated into Dialog application features.
Assess business requirements related to data warehousing, including evaluating whether the existing PostgreSQL infrastructure remains sufficient or whether migration to a dedicated data warehouse (e.g., BigQuery, Snowflake, Amazon Redshift, or similar platforms) is required.
Delegate tasks to other team members when necessary while maintaining full ownership of the overall data aggregation roadmap, architecture, and quality.
Collaborate closely with AI and Product teams to ensure the data architecture and pipelines align with AI model requirements and product feature objectives.
Qualifications
- Proven experience as a
- Data Engineer
- (this is not an entry-level role; candidates are expected to have hands-on experience designing and building data pipelines independently).
- Strong proficiency in
- Python
- for data processing and pipeline development.
- Hands-on experience with at least one of the following technologies:
- SQL, Apache Spark, Apache Airflow, Apache Kafka
- , or equivalent tools such as
- dbt, Dagster, Apache Flink
- , or similar.
- Solid understanding of
- data warehousing concepts
, with the ability to analyze business requirements and provide technical recommendations, including strategic decisions around warehouse architecture and migration.
- Experience implementing and monitoring
- data quality checks
- , including data validation, anomaly detection, and other data integrity practices.
- Ability to work independently while also leading initiatives and delegating tasks to other team members when required.
- Good command of English
- , both written and spoken.
Nice to have
- Experience building and maintaining data pipelines that serve as input for
- AI/ML systems