Data Architect or AWS Data Engineer

Key Responsibilities

Own and operate the complete AWS data hub: Bronze (S3, raw partitioned data), Silver (Apache Iceberg on S3, ACID transactions, domain tables), and Gold (Amazon Athena views with business rules applied – Sales Performance, Loyalty Health, Campaign ROI, Staff Performance, Financial View, Unified Customer).

Design, build, and own the Customer Data Platform (CDP): architect the unified customer profile store on AWS, implement identity resolution logic (matching customers across clients’s main chain, Beauty, Fashion, and Sleep concepts by email, phone, loyalty member ID, and device identifiers), and maintain a golden customer record per individual across all touchpoints.

Manage CDP data ingestion: ingest customer event data from all source systems – POS transactions, e-commerce orders, loyalty events from Eagle Eye, CRM interactions, and web/app behavioural data – into the CDP in near-real-time or batch as appropriate per source.

Manage CDP outbound syncs: push enriched customer profiles, segment memberships, and loyalty tier data from the CDP to Eagle Eye (loyalty execution), the marketing platform (campaign targeting), and the AI/Agent Engineer's Gold layer views (personalisation and agent use cases).

Manage all data ingestion pipelines: AWS Glue Python ETL jobs, AWS DMS replication, Lambda event-driven ingestion, and scheduled batch pulls, covering all client’s source systems.

Govern Lake Formation security: row-level and column-level access controls per consumer role; monthly access reviews; PDPA and UU PDP compliance for all customer PII fields.

• Maintain the AWS Glue Data Catalog: schema registration, schema evolution management, partition projections, and table properties.
• Orchestrate pipelines via AWS Step Functions: scheduling, retry logic, failure alerting, and inter-job dependencies.
• Maintain and extend business intelligence dashboards and reports connected directly to the Gold Athena layer.

Collaborate with the AI/Agent Engineer: ensure Gold layer views and CDP customer profiles are structured for effective MCP natural language query generation and personalisation agent use cases.

Manage the Azure-to-AWS data bridge for client’s hybrid cloud architecture: design and operate pipelines that replicate Microsoft system data (D365 F&SCM transactions, Dataverse/Power Pages records, SharePoint document metadata) from Azure into the AWS Bronze layer.

Govern cross-cloud data consistency: ensure that key business entities remain consistent across AWS and Azure; define and monitor reconciliation checks that flag divergence between the two platforms.

• Shadow CloudMile during ECI build phase; produce client’s internal architecture runbooks before handover.
• Required Qualifications & Experience
• Minimum 5 years of data engineering experience; at least 3 years on AWS data platforms in production, not training environments.
• Demonstrable production experience with AWS Glue, Amazon S3, Amazon Athena, and AWS Lake Formation.
• Hands-on experience with Apache Iceberg or Apache Hudi on S3: ACID transactions, schema evolution, time travel.

Experience designing or building a Customer Data Platform or unified customer profile store: identity resolution, profile merging, and customer event ingestion from multiple source systems.

• Advanced SQL: complex Athena queries, window functions, CTEs, partition optimisation.
• English proficiency at B2 minimum (IELTS 6.0); mandatory assessment at interview.

Strong independence: able to diagnose pipeline failures, identify data quality risks proactively, and own the platform without a team around them.

Technical Proficiency

AWS: S3, AWS Glue (Python ETL, Data Catalog, Data Quality), AWS DMS, AWS Step Functions, Amazon Athena, AWS Lake Formation, Amazon CloudWatch, CloudTrail, AWS Lambda, Amazon DynamoDB.

CDP architecture: unified customer profile design, identity resolution logic, customer event schema design, real-time and batch ingestion patterns, and outbound sync to loyalty and marketing systems.

• Apache Iceberg: snapshot management, schema evolution, compaction, time travel.
• Python (intermediate–advanced): ETL scripting, schema standardisation, deduplication,
• identity resolution.

Azure Data Factory (ADF): sufficient working knowledge to design and operate data pipelines bridging Microsoft Azure sources into AWS S3 (leeway given for learning if experience is rare).

• BI reporting tools: experience with at least one cloud-connected BI platform (e.g., Amazon QuickSight, Power BI, or equivalent).
• Git / Azure DevOps.