Skill diminta
AWSCI/CDCommunicationDockerGitGitHub ActionsGitLab CIGrafanaJenkinsKubernetesPostgreSQLPrometheusPythonSQLSQL ServerTroubleshooting
Deskripsi
- Deskripsi pekerjaan Data Engineer Dewasz Laundry & Drycleaning Services
- Data Engineer
Job Description
Position Overview
We are seeking a Data Engineer for IlmuData to support the development and operations of enterprise data pipelines. The role focuses on Change Data Capture (CDC) ingestion from source systems into Oracle On-Premises ODS using Debezium, Solace messaging, and Python-based consumer applications running on Kubernetes.
Responsibilities
- Monitor and maintain Debezium CDC connectors that capture database changes.
- Troubleshoot and optimize Python consumer applications processing Solace messages.
- Support and troubleshoot transformed data loading into Oracle On-Premises ODS.
- Investigate and resolve pipeline failures and performance bottlenecks.
- Support Kubernetes deployments, scaling, and health monitoring for consumer services.
- Maintain operational runbooks and technical documentation.
Requirements
- Technical Skills
- Primary Technologies
- Debezium (1+ years)
- CDC connector configuration
- Source connectors (Oracle, PostgreSQL, SQL Server)
- Python (2+ years)
- Data pipeline development
- Consumer applications
- Asynchronous programming
- Error handling
- Kubernetes (1+ years)
- Pod management
- Deployments and services
- ConfigMaps and Secrets
- Resource management
- Troubleshooting
- RDBMS (2+ years)
- SQL query development
- Data loading (SQL*Loader, INSERT/UPDATE scripts)
- Performance tuning
- Data Engineering Fundamentals
- ETL/ELT pipeline design and optimization
- Schema evolution management
- SQL DML proficiency
- Python DataFrame processing
- DevOps & Tools
- Docker containerization and image building
- CI/CD pipelines (Jenkins, GitLab CI, GitHub Actions, or equivalent)
- Monitoring tools (Prometheus, Grafana, ELK Stack, or equivalent)
- Git version control
- Nice to Have
- Oracle RDBMS (PL/SQL and SQL Loader)
- Solace message broker experience
- AWS cloud data services
- High-throughput data pipeline performance tuning
- Data governance and data lineage tools exposure
Qualifications
- Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent professional experience).
- 5+ years of experience in data engineering or backend development.
- Experience supporting production systems in regulated environments.
- Strong analytical, problem-solving, and communication skills.
- Ability to work independently and collaborate with cross-functional teams.
Benefits
- Opportunity to work on enterprise-scale data platforms.
- Continuous learning environment with exposure to modern data technologies.
- Collaborative and cross-functional team culture.
- Work Environment
- On-premises and hybrid infrastructure.
- On-call responsibilities for critical incidents.
- Continuous learning and frequent technology updates.