Data engineer

Als Data Engineer ontwerp, bouw en integreer je data-infrastructuur en -pijplijnen vanuit diverse bronnen naar het Big Data platform, met focus op eenvoud en efficiëntie.

Je realiseert end-to-end BI-oplossingen: van het verzamelen en structureren van data tot het beschikbaar maken ervan voor visualisatie en rapportage (self-service, PowerBI).

Je ontwikkelt en onderhoudt ETL/ELT-processen, streaming-analytics en API-koppelingen, en draagt bij aan AI/ML use-cases door modellen te bouwen en resultaten te ontsluiten voor de business.

Je werkt in DevOps-teams binnen de MS Azure-stack (Synapse, Databricks, Event Hub, Function Apps e.d.) en automatiseert deployment, monitoring en proactieve alerting volgens Vattenfall-standaarden.

Je vertaalt user stories naar technische taken, beoordeelt prioriteiten op klantwaarde en stabiliteit, en adviseert stakeholders over data-oplossingen en architectuurkeuzes.

Vereist profiel

- HBO/WO werk- en denkniveau met aantoonbare trackrecord in data engineering.

- Gedegen ervaring met MS Azure-ecosysteem: Synapse, SQL Pools/Databases, Databricks, Event Hub, Stream Analytics, Function Apps.

- Sterk in programmeertalen en tooling: Python (PySpark), SQL, Terraform, Bash; vertrouwd met CI/CD (Azure DevOps/Git), testing en versiebeheer.

- Ervaring met datawarehousing, datamodellering, data lakes, data quality checks, validatie frameworks en beveiligingsprincipes (RBAC, IAM, Key Vault, GDPR).

- Bekend met containerisatie & Kubernetes, infrastructuur als code (Terraform/ARM) en geautomatiseerde deployments inclusief rollbacks.

- Vaardig in het opzetten van monitoring en capacity-parameters voor productieomgevingen; kennis van on-premise, cloud en hybride architecturen (TOGAF-kennis is een pré).

- Sterke communicatieve vaardigheden in het Engels (praktisch Engels is de voertaal); ervaring met agile werken en samenwerken met architects, data scientists en business stakeholders.

- Ervaring in de energiesector (SAP ERP, EAM, SCADA, regelgeving en security) is een pré; woon- of reisbereidheid naar Amsterdam (aanwezigheid op dinsdag en donderdag).

Competenties

- Create the data infrastructure in a simple and efficient manner by designing, building and integrating data from various sources.

- Build data pipelines from existing data sources and other (on-prem) systems into the Big Data platform.

- Create consistent end-to-end BI solutions with a data lifecycle starting with collecting the needed data until visualizing them.

- Create AI/ML models in BigData use cases and help the business to leverage the results.

- Create user stories with a good understanding of business processes, product and customer journey.

- Rate user stories for customer value and challenge the product owner on prioritization of user stories.

- Translate user stories into technical stories.

- Design, develop and optimize data storage and data processing solutions.

- Be proficient in one or more programming languages like Python and SQL, including test criteria.

- Create and execute script-based deployments, including rollbacks.

- Maintain and set up automated deployment using the VF standard toolset.

- Set up automated monitoring and proactive alerting.

- Provide parameters to size the underlying application infrastructure.

- Analyze business needs and translate them into IT specifications.

- Provide expert advice internally in the team and to various other departments.

- Proactively adapt value adding solutions to the needs of other departments.

- Design, build and maintain data lake solutions to share data within Vattenfall and with external stakeholders.

- Implement traditional database and data warehouse concepts to structure the data in the cloud, on-premise and hybrid to provide self-service reporting solutions.

- Build and maintain ETL/ELT tracks and APIs.

- Work with streaming data analytics together with business stakeholders.

- Design robust data models, data warehouses, and data lakes using modern cloud and distributed platforms.

- Define and implement data quality checks, validation frameworks, and security best practices to ensure data integrity.

- Implement security controls, networking, and governance policies using Azure Key Vault and role-based access control.

- Automate the deployment of data infrastructure and pipelines.

- Have working experience with Analytics and BI tools and services, preferably in the Microsoft Azure stack.

- Have working experience with Synapse workspaces and dedicated SQL pools.

- Have working experience with SQL Database and Analysis Services.

- Have working experience with Function Apps, Event Grid and Event Hub.

- Have working experience with MQTT and stream analytics.

- Have working experience with Databricks.

- Have a technical background with an understanding of how analytics solutions are built using Python, SQL and PySpark.

- Be familiar with CI/CD, testing and version control, for example Azure DevOps, ARM and Git.

- Be familiar with Terraform and Bash.

- Be familiar with data warehousing and data modelling concepts.

- Design, implement and maintain Azure infrastructure using Infrastructure as Code with Terraform.

- Implement RBAC, IAM and security best practices in Azure environments.

- Collaborate with developers to enable cloud native deployments.

- Support containerized workloads using Kubernetes.

- Have an understanding of IT architecture guidelines and principles such as TOGAF and on-premise, cloud and hybrid architectural patterns.

- Have experience in working agile.

- Be proficient in spoken and written English.

- Ideally have a good understanding of utility business processes and their IT landscape, such as SAP ERP, IAM, EAM, SCADA and GDPR.