Skill diminta
AWSAirflowApacheCommunicationHadoopLinuxSQLSparkTroubleshooting
Deskripsi
Job Description
- Monitor and maintain ETL/ELT pipelines and Apache Spark batch processing jobs to ensure stable and reliable data operations.
- Monitor WeData or equivalent ETL platform execution, job schedulers, dependencies, execution logs, and system performance.
- Investigate, troubleshoot, and resolve ETL/Spark job failures and production incidents.
- Perform job rerun, recovery, performance tuning, and workaround implementation based on operational procedures.
- Conduct data validation to ensure data quality and pipeline integrity.
- Perform Root Cause Analysis (RCA) for recurring or critical production issues and recommend preventive actions.
- Prepare daily monitoring reports, incident reports, and technical documentation.
- Escalate complex issues to L3 Platform Engineers or Tencent Cloud support when required.
- Collaborate with cross-functional teams to ensure production stability and continuous service improvement.
- Provide technical guidance and support to junior team members (for Senior level).
Requirements
- Bachelor's Degree in Computer Science, Information Technology, Engineering, or a related field.
- Minimum
- 2 - 5 years of experience
- as a Data Engineer, ETL Support, Data Analyst, or Fresh Graduate with internship/project experience in Big Data or Data Engineering.
- Strong understanding of ETL/ELT concepts and data pipeline architecture.
Experience with Apache Spark, including Spark SQL, batch processing, and job troubleshooting (advanced proficiency required for Senior level).
- Familiarity with Tencent WeData or similar ETL platforms such as Airflow, Informatica, Talend, Ab Initio, AWS Glue, or Alibaba DataWorks.
- Good SQL skills for data validation, troubleshooting, and query optimization (advanced SQL required for Senior level).
- Familiarity with the Hadoop ecosystem (Hive, HDFS, YARN) is an advantage.
- Basic knowledge of Linux commands and Shell scripting.
- Understanding of job monitoring, execution logs, schedulers, dependencies, and recovery processes.
- Strong analytical, troubleshooting, and problem-solving skills.
- Good communication skills and the ability to work collaboratively in a production support environment.
- Available to join as soon as possible (ASAP).