Skill diminta
HadoopPythonSQLScalaSpark
Deskripsi
Role Summary
We are seeking a skilled ETL Developer to design, develop, and maintain high-performance ETL pipelines on Tencent Cloud platforms, especially using WeData for data integration and orchestration, and DLC (Data Lake Compute) for SQL-based lake processing. The role requires strong hands-on experience in building, scheduling, and monitoring ETL data flows with Spark underneath.
- Key Responsibilities
- Design, build, schedule, and monitor end-to-end ETL pipelines using Tencent Cloud WeData
- Develop and optimize SQL-based ETL jobs on DLC (Data Lake Compute) with Spark engine
- Perform data extraction, transformation, cleaning, and loading from various source systems into data lakehouse
- Implement incremental loading, scheduling, orchestration, and real-time monitoring of pipelines
- Troubleshoot pipeline failures and optimize performance (Spark tuning, SQL optimization)
- Collaborate with data architects and business teams to understand data requirements and deliver reliable solutions
- Person Specifications
- Required Qualifications
- Minimum 3+ years of hands-on ETL development experience (build, schedule, and monitor pipelines)
- Direct and proven experience with Tencent Cloud WeData (data integration, orchestration, scheduling)
- Direct and proven experience with Tencent Cloud DLC (Data Lake Compute) – SQL-based lake processing
- Strong proficiency in SQL and Spark (DLC uses Spark for backend ETL processing)
- Bachelor's degree in Computer Science or related fields
- Preferred Qualifications
- Experience with Hive, Hadoop, Iceberg, Lakehouse architecture
- Previous experience with Ab Initio or other legacy ETL tools
- Familiarity with other Tencent Cloud products: EMR, COS, TencentDB, CLS
- Proficiency in Python or Scala scripting for ETL automation
- Background in telecom, data warehouse, or large-scale data migration projects
- Joining: Immediate joiners are highly preferable
- Vendor submissions -
- 06 months