Lead Data Engineer

Who we are CreateFuture is fast becoming the UK’s most recognisable digital consultancy, with years of experience building digital products and services for major organisations whilst putting our people first. We have offices in the centre of Edinburgh, Leeds, Manchester, and London as well as remote employees located throughout the country.    We are a team of creators - whether that’s code, project plans, go to market strategies, culture initiatives, marketing campaigns, large language models or people policies. And together, with our clients, we create the future. This has seen us collaborate and partner across a multitude of industries and sectors, with the likes of PayPal, adidas, Natwest, FanDuel and Money Saving Expert, to name just a few.   Our reputation as a partner determined to deliver high-quality, robust and thoughtful products has enabled us to scale to over 500 people in the last couple of years, and it is our amazing people - along with the safe, supportive and friendly culture we have built - that makes CreateFuture a great place to work. Don’t just take our word for it though, we have been recognised by Best Workplaces UK multiple years in a row - across a number of categories - and our employee exit rate is astonishingly low.  Join us on our journey… Let’s create something awesome, together, today. About the role and team: ABOUT THE ROLE AND TEAM We're looking for a Principal Data Engineer to join our Data Practice and own the technical direction of our iGaming data engineering capability. You'll work with some of the world's largest betting and gaming operators — organisations processing hundreds of millions of events per day across sportsbook, casino, exchange, and lottery platforms. A Principal at Create Future operates above the delivery line as well as within it. You'll define how we solve the hardest data engineering problems in iGaming: petabyte-scale migrations from legacy on-premises estates, sub-millisecond event pipelines, settlement reconciliation at bookmaker scale, and the data foundations that keep operators compliant with gambling regulators globally. You'll influence the shape of engagements from pre-sales through to delivery, grow the engineering talent around you, and be a recognisable voice for Create Future in the iGaming market.   WHAT YOU'LL BE DOING TECHNICAL LEADERSHIP * Defining and owning the data engineering architecture patterns Create Future uses across iGaming engagements — migration frameworks, streaming architectures, data integrity standards, and Lakehouse design * Acting as the most senior technical authority on client engagements — making critical architecture decisions, resolving design disputes, and setting the bar for quality * Reviewing and signing off technical designs produced by Lead and Senior engineers across active engagements * Identifying and driving the adoption of new tooling, patterns, and AWS services that improve delivery outcomes across the iGaming practice   DELIVERY * Leading the technical design and execution of large-scale data migrations — moving high-volume transactional and event data from on-premises or legacy cloud estates to AWS, without loss of fidelity or operational continuity * Architecting real-time and near-real-time data pipelines for sportsbook and in-play use cases — odds feeds, market events, bet placement streams, and settlement pipelines — where latency and correctness are non-negotiable * Designing and enforcing data integrity frameworks across betting, trading, and customer data domains — including reconciliation between source and target systems, anomaly detection, and audit trail requirements * Overseeing the implementation of regulatory and compliance reporting pipelines that meet the requirements of the UK Gambling Commission and international licensing bodies * Working with AWS services including S3, Glue, Lambda, Athena, Redshift, RDS, Kinesis, MSK, Lake Formation, Step Functions, and MWAA — selecting the right tool for each problem rather than defaulting to a stack   PRACTICE AND PEOPLE * Mentoring and growing Lead and Senior Data Engineers — through code review, pairing, architecture sessions, and career conversations * Building the iGaming data engineering playbook: reusable patterns, accelerators, and reference architectures that reduce time-to-value on new engagements * Contributing to hiring — defining what great looks like for data engineering roles and playing an active part in assessment * Representing Create Future externally — at industry events, in technical blog posts, and in client-facing settings — as a credible voice on iGaming data engineering   COMMERCIAL * Shaping and leading technical responses to iGaming opportunities — scoping complex engagements, writing architecture proposals, and presenting to senior client stakeholders * Identifying expansion opportunities within existing client relationships — spotting data engineering problems we can help solve and framing them as propositions * Partnering with AWS on joint go-to-market activity where relevant, leveraging Create Future's Advanced Partner status   WHAT WE'RE LOOKING FOR DEEP IGAMING DOMAIN EXPERTISE * Extensive experience working in or closely with sportsbook, exchange, or casino platforms — you understand the data models, operational pressures, and regulatory constraints from the inside * You've led or made decisive contributions to large-scale data migrations in iGaming — moving live betting, customer, or trading data at scale while maintaining integrity and uptime * You've designed or owned low-latency data pipelines where correctness and throughput are both critical — odds ingestion, in-play event streams, bet settlement, or equivalent * You understand what data integrity means in a regulated gambling context: reconciliation, audit trails, lineage, and the consequences of getting it wrong   AWS DATA ENGINEERING MASTERY * AWS is your primary cloud and you can design across the full data engineering stack — from ingestion and streaming through to warehousing, serving, and cost optimisation * You have deep experience with streaming architectures — Kinesis, Kafka/MSK, Flink, or Spark Streaming — and know when each is the right choice * You can model the cost implications of architecture decisions and advise clients on trade-offs between throughput, latency, durability, and spend * AWS certification at professional or specialty level is strongly preferred   ENGINEERING EXCELLENCE * Mastery of Python, SQL, and Spark — and strong opinions on how they should be used in production data pipelines * Deep experience with dbt, Airflow, or equivalent orchestration — including how to build pipelines that are testable, observable, and operationally sound * Strong understanding of data quality engineering — not just validation rules, but how to build systems that detect and surface integrity issues before they reach consumers * Comfortable with infrastructure-as-code (Terraform, AWS CDK) and treating data platform infrastructure as a first-class engineering concern   CONSULTANCY AND LEADERSHIP * You've operated at Principal or equivalent level — making architectural decisions that affect multiple teams, and being accountable for their outcomes * You're comfortable in C-suite and VP-level client conversations, translating technical complexity into business risk and opportunity * You can hold a room on a difficult technical topic and reach a decision — not just present options * You've grown engineering talent around you and take it seriously as a leadership responsibility   TECH SNAPSHOT Primary cloud AWS — Glue, Lambda, Athena, S3, Redshift, RDS, Kinesis, MSK, Lake Formation, Step Functions, MWAA Streaming AWS Kinesis, Apache Kafka / MSK, Apache Flink, Spark Streaming Languages Python, SQL, Spark Pipeline tools dbt, Airflow / AWS MWAA, AWS Glue Platforms Databricks, Snowflake, Amazon Redshift Data quality Great Expectations, dbt tests, reconciliation frameworks, data observability tooling IaC / DevOps Terraform, AWS CDK Certification AWS Professional or Specialty level strongly preferred WHAT WE’LL OFFER YOU: We trust people to do their best work. That means flexibility over rigid rules, impact over activity, and real investment in your growth both professionally and personally. You’ll be part of a supportive, and friendly culture, surrounded by smart, curious people who care deeply about what they do. We offer flexible working, including hybrid and remote options. Our office hubs are located in Edinburgh, Leeds, Manchester, London and Bulgaria, with occasional travel to client sites or CreateFuture offices when needed. We trust you to manage your time balancing collaboration with client time and focused work. What matters is the impact you have, not how busy you look. OUR HIRING PROCESS We try to keep our hiring process clear, fair and respectful of your time. We aim to get back to everyone who applies and we will be upfront about where you are in the process. It usually looks like this: * Call with our Talent Acquisition Team  * Role specific capability interview  Depending on the role, we might also ask you to do a short presentation, a practical or technical task or have a values focused conversation. We will explain what is involved before anything happens. INCLUSION AT CREATEFUTURE  We believe diverse teams build better workplaces and better products. We want CreateFuture to be a place where people feel able to be themselves and do their best work. If you need any adjustments or support during the application process, just. We will do what we can to help. We look forward to your application!

Aker Systems was founded in 2017 by a team of experienced technology professionals who recognised an opportunity to provide highly secure enterprise data platforms to large organisations.  We build and operate ground-breaking, ultra-secure, high performance, cloud-based data infrastructure for the enterprise. Our proprietary technology solutions drive performance and reduce costs while helping our clients to improve the management and sharing of data across their organisations. In 2024, Aker Systems won the Breakthrough Culture Awards highlighting growth companies putting culture first. In 2020 Aker Systems was recognised as a ‘One to Watch’ on the Sunday Times Tech Track.  The Company was also recognised at the Thames Valley Tech Awards 2020; winning the Thames Valley Tech Company of the year, the Emerging Tech Company and High Growth Tech Business categories.  We encourage people of all different backgrounds and identities to apply.  We are committed to maintaining an inclusive, and supportive place for you to do your very best work. We are seeking an experienced and hands-on Lead Data Engineer to design, build and evolve secure, scalable data platforms for clients across the UK Public Sector and Utilities sectors. You will work closely with architects, engineers and client stakeholders to deliver modern cloud-based data solutions that support operational, analytical and AI-driven use cases. The role combines technical leadership with hands-on engineering, helping clients modernise their data estates while ensuring solutions are secure, reliable and scalable. Working across multiple projects, you will lead the development of batch and real-time data pipelines, support architectural decision-making, and help establish engineering best practices across delivery teams. Strong experience with real-time streaming technologies is essential. Key Responsibilities * Lead the design, development and deployment of secure, scalable data pipelines and data platform components. * Design and implement batch and real-time streaming solutions using modern event-driven architectures. * Own technical design decisions across projects, balancing scalability, security, performance and delivery requirements. * Lead technical workshops, discovery sessions and architecture discussions with client stakeholders. * Assess existing client data platforms and design practical approaches for modernisation, migration and integration. * Build and maintain data ingestion, transformation and integration capabilities across a variety of data sources. * Develop cloud-native data platforms using AWS services and open-source technologies. * Ensure data quality, governance, lineage and security requirements are embedded throughout the data lifecycle. * Drive engineering best practices including code reviews, automated testing, CI/CD and Infrastructure as Code. * Support multiple client engagements, helping delivery teams maintain consistent engineering quality and standards. * Mentor and support engineers across project teams, providing technical guidance and leadership. * Perform data modelling, performance tuning and root cause analysis to resolve complex data and platform issues. Essential Skills & Experience Real-Time Streaming & Data Engineering * Strong commercial experience designing and implementing real-time streaming data platforms. * Hands-on experience with technologies such as Apache Kafka, AWS Kinesis, Apache Flink, Spark Streaming, Apache Pulsar, or similar event-driven technologies. * Experience building high-volume, low-latency, resilient data pipelines in production environments. * Strong understanding of modern data engineering patterns, including batch, streaming and hybrid architectures. Data Engineering & Databases * Extensive experience developing and operating enterprise-scale data platforms. * Strong SQL skills and experience designing data models for analytical and operational workloads. * Proficiency in Python or other modern programming languages used for data engineering. * Experience configuring, tuning and optimising relational and NoSQL databases. Cloud & Software Engineering * Strong experience working with AWS cloud services, including EC2, RDS, Lambda, Athena, Glue, S3 and IAM. * Experience working within Agile delivery environments. * Strong understanding of Test-Driven Development (TDD) and automated testing practices. * Experience implementing CI/CD pipelines and engineering automation. * Strong understanding of Infrastructure as Code principles. * Experience using version control systems such as Git. Regulated Environments * Experience delivering technology solutions within UK Public Sector, Utilities, Energy, Water, Financial Services or other highly regulated industries. * Understanding of security, governance and compliance considerations within regulated environments. Desirable Skills * Experience with Terraform. * Experience with Docker and Kubernetes. * Knowledge of Ansible and Packer. * Linux and Windows administration experience. * Experience supporting AI, machine learning or advanced analytics platforms. * Experience working on programmes requiring UK Government security clearance. What We're Looking For * Strong technical leadership and problem-solving skills. * Ability to communicate effectively with both technical and non-technical stakeholders. * Experience working in complex, regulated environments. * A collaborative approach and willingness to support and mentor others. * A focus on delivering practical solutions that create value for clients. * A passion for building reliable, scalable and secure data platforms. Education Bachelor’s degree in computer science, Software Engineering, Data Engineering, Mathematics or a related technical discipline, or equivalent practical experience. Aker Systems Attributes At Aker we work as a team; we are collaborative, hardworking, open, and delivery obsessed. There is no blame culture here: try things, and take responsibility for the outcomes. You are always part of the wider Aker. We help our colleagues and take pride in successfully achieving difficult tasks. We run towards problems and help to solve them. Communicate always, do so accurately and in a timely fashion. In return, we offer a competitive salary, 25 days holiday plus bank holidays, company paid medical insurance and life assurance, pension scheme, annual training allowance, wellbeing allowance, virtual GP, Employee Assistance plan and more. Equal Opportunities Aker Systems fosters a diverse environment that encourages openness in its communications and is committed to providing equal employment opportunity for all people regardless of race, religion, gender or sexual orientation, age, marital status, national origin, citizenship status, disability, veteran status or other personal characteristics. We embrace differences of opinion and diversity because they help challenge us and find new groundbreaking technical solutions.