Job description for Data Engineer (Data Lakehouse architectures) at SotaTek
JOB DESCRIPTION
- Design, develop, and maintain data pipelines to support business operations.
- Process and transform data to meet both functional and non-functional business requirements.
- Identify, design, and implement process improvements to automate manual tasks and optimize data delivery.
- Design and develop data warehouse (DW) solutions, including data modeling and ETL processes.
- Develop real-time data ingestion and streaming solutions.
- Design and implement cloud-based data platforms and architectures on AWS or GCP.
- Design and develop container-based services using Kubernetes (Amazon EKS/Google GKE).
- Design, develop, and maintain MLOps platforms and machine learning deployment pipelines..
JOB REQUIREMENTS
- No spoken English proficiency required.
- 3–5 years of hands-on experience designing and building large-scale data pipelines (processing over 100GB/day) in enterprise environments.
Proven experience with AWS Data Services, including Amazon S3, AWS Glue (Crawlers & Jobs), Amazon EMR, Amazon MSK (Kafka), AWS Step Functions, AWS Lake Formation, and Amazon Redshift Spectrum.
Strong expertise in Apache Spark (PySpark or Scala) for developing batch and streaming ETL/ELT workloads on Amazon EMR or AWS Glue.
Solid understanding of and hands-on experience implementing Data Lakehouse architectures using open table formats such as Apache Iceberg or Delta Lake, including schema evolution, time travel, and ACID transactions.
Strong Python programming skills with experience developing AWS Glue Jobs, AWS Lambda functions, pipeline automation, and unit testing for data transformations.
Experience building real-time streaming pipelines using Apache Kafka or Amazon MSK, with a solid understanding of Kafka Consumer Groups, offset management, and Schema Registry.
Hands-on experience designing and implementing data pipeline orchestration using Apache Airflow or AWS Step Functions, including DAG design, dependency management, retry mechanisms, and SLA monitoring.
Experience integrating data from multiple sources, including relational databases (Oracle, MySQL, PostgreSQL), NoSQL databases (MongoDB, Redis), SaaS APIs (REST/SOAP), and SFTP-based file ingestion.
Good understanding of Data Quality concepts, with experience implementing data quality checks (completeness, uniqueness, timeliness) using Oracle EDQ, Great Expectations, or Deequ.
Familiarity with Data Governance practices, including metadata management, data lineage, and data tagging, with hands-on experience using Apache Atlas or AWS Glue Data Catalog.
Experience with Oracle GoldenGate CDC is a strong advantage.
Experience working in Agile/Scrum environments and collaborating closely with Data Architects, Cloud Engineers, DBAs, and Analytics Engineers.
- Proactive in creating and maintaining technical documentation.
- Note: Open to level Fresher. Must have: Good at English
COMPENSATION & BENEFITS
🎯Flexible working regime and health care
- Flexible timekeeping (from 8:00 - 9:00 to 17:30 - 18:30)
- Minimum 14 paid leaves per annum for all employees after probation
- 01-day remote work per month
- A flexitime allowance of 90-180 minutes per month for employees
- 01 hour paid leave per day for women having children under 12 months
- Social insurance, health insurance, unemployment insurance and Bao Viet care insurance
🎯Transparent and fair benefits
- Saturday & Sunday OFF, Overtime pay is 150%, 200%, 300% as per labor law;
- Work performance review 2 times/ year (in April and October)
- Bonus Policy: 13th-month salary and Performance-based bonus
- Men’s Day, Women’s Day, Children’s Day, Mid-Autumn Festival and other benefits under the provisions of the company
🎯Dynamic environment and open culture
- Year-end party, sports day, yearly company trip and quarterly team building,...with a generous budget
- Socialize with colleagues through monthly Happy Hour
- Monthly allowance when joining clubs: Soccer, Swimming, Yoga, Music,...
- Nice & modern working space with young, dynamic & friendly colleagues and free coffee, tea, drinks,...
- Flat, open and sharing culture with friendly management team; outsourcing company with product mindset
🎯Strong learning culture
- Free training courses for technical and soft skills (presentation skills, communication skills, foreign language courses,...)
- Account to log in to our online learning system, which contains thousands of valuable lectures (LMS)
- Participate in workshops, seminars, tech talk,... with sharing from experts inside and outside the company
- Working opportunities with technical gurus who built and operated world-class applications with millions of users.