Lead Data Engineer

Lead Agentic Software Engineer  This is a great opportunity to work at the frontier of AI-augmented software delivery, on challenging technical problems from a strong position of influence.   At Scott Logic, we already have a reputation for world-class software engineering. Our clients are some of the world’s leading organisations and have long trusted us to build reliable, high-performance, business-critical solutions. They now expect us to help them unlock the productivity benefits of agentic AI development – how to best adopt the tools, but also crucially ensuring they have the right foundations and guardrails in place and supporting the wider shift in day-to-day workflows.  You will typically be leading a small, tight-knit multidisciplinary team, working with our clients to solve tough software problems, as well as having a real impact on how they evolve their development practices. You’ll foster stakeholder relationships and advise on key technology and software delivery decisions. There's also plenty of opportunity to contribute more widely across Scott Logic, collaborating and sharing knowledge with colleagues as we collectively learn and adapt, and helping to shape upcoming projects. 

Aker Systems was founded in 2017 by a team of experienced technology professionals who recognised an opportunity to provide highly secure enterprise data platforms to large organisations.  We build and operate ground-breaking, ultra-secure, high performance, cloud-based data infrastructure for the enterprise. Our proprietary technology solutions drive performance and reduce costs while helping our clients to improve the management and sharing of data across their organisations. In 2024, Aker Systems won the Breakthrough Culture Awards highlighting growth companies putting culture first. In 2020 Aker Systems was recognised as a ‘One to Watch’ on the Sunday Times Tech Track.  The Company was also recognised at the Thames Valley Tech Awards 2020; winning the Thames Valley Tech Company of the year, the Emerging Tech Company and High Growth Tech Business categories.  We encourage people of all different backgrounds and identities to apply.  We are committed to maintaining an inclusive, and supportive place for you to do your very best work. We are seeking an experienced and hands-on Lead Data Engineer to design, build and evolve secure, scalable data platforms for clients across the UK Public Sector and Utilities sectors. You will work closely with architects, engineers and client stakeholders to deliver modern cloud-based data solutions that support operational, analytical and AI-driven use cases. The role combines technical leadership with hands-on engineering, helping clients modernise their data estates while ensuring solutions are secure, reliable and scalable. Working across multiple projects, you will lead the development of batch and real-time data pipelines, support architectural decision-making, and help establish engineering best practices across delivery teams. Strong experience with real-time streaming technologies is essential. Key Responsibilities * Lead the design, development and deployment of secure, scalable data pipelines and data platform components. * Design and implement batch and real-time streaming solutions using modern event-driven architectures. * Own technical design decisions across projects, balancing scalability, security, performance and delivery requirements. * Lead technical workshops, discovery sessions and architecture discussions with client stakeholders. * Assess existing client data platforms and design practical approaches for modernisation, migration and integration. * Build and maintain data ingestion, transformation and integration capabilities across a variety of data sources. * Develop cloud-native data platforms using AWS services and open-source technologies. * Ensure data quality, governance, lineage and security requirements are embedded throughout the data lifecycle. * Drive engineering best practices including code reviews, automated testing, CI/CD and Infrastructure as Code. * Support multiple client engagements, helping delivery teams maintain consistent engineering quality and standards. * Mentor and support engineers across project teams, providing technical guidance and leadership. * Perform data modelling, performance tuning and root cause analysis to resolve complex data and platform issues. Essential Skills & Experience Real-Time Streaming & Data Engineering * Strong commercial experience designing and implementing real-time streaming data platforms. * Hands-on experience with technologies such as Apache Kafka, AWS Kinesis, Apache Flink, Spark Streaming, Apache Pulsar, or similar event-driven technologies. * Experience building high-volume, low-latency, resilient data pipelines in production environments. * Strong understanding of modern data engineering patterns, including batch, streaming and hybrid architectures. Data Engineering & Databases * Extensive experience developing and operating enterprise-scale data platforms. * Strong SQL skills and experience designing data models for analytical and operational workloads. * Proficiency in Python or other modern programming languages used for data engineering. * Experience configuring, tuning and optimising relational and NoSQL databases. Cloud & Software Engineering * Strong experience working with AWS cloud services, including EC2, RDS, Lambda, Athena, Glue, S3 and IAM. * Experience working within Agile delivery environments. * Strong understanding of Test-Driven Development (TDD) and automated testing practices. * Experience implementing CI/CD pipelines and engineering automation. * Strong understanding of Infrastructure as Code principles. * Experience using version control systems such as Git. Regulated Environments * Experience delivering technology solutions within UK Public Sector, Utilities, Energy, Water, Financial Services or other highly regulated industries. * Understanding of security, governance and compliance considerations within regulated environments. Desirable Skills * Experience with Terraform. * Experience with Docker and Kubernetes. * Knowledge of Ansible and Packer. * Linux and Windows administration experience. * Experience supporting AI, machine learning or advanced analytics platforms. * Experience working on programmes requiring UK Government security clearance. What We're Looking For * Strong technical leadership and problem-solving skills. * Ability to communicate effectively with both technical and non-technical stakeholders. * Experience working in complex, regulated environments. * A collaborative approach and willingness to support and mentor others. * A focus on delivering practical solutions that create value for clients. * A passion for building reliable, scalable and secure data platforms. Education Bachelor’s degree in computer science, Software Engineering, Data Engineering, Mathematics or a related technical discipline, or equivalent practical experience. Aker Systems Attributes At Aker we work as a team; we are collaborative, hardworking, open, and delivery obsessed. There is no blame culture here: try things, and take responsibility for the outcomes. You are always part of the wider Aker. We help our colleagues and take pride in successfully achieving difficult tasks. We run towards problems and help to solve them. Communicate always, do so accurately and in a timely fashion. In return, we offer a competitive salary, 25 days holiday plus bank holidays, company paid medical insurance and life assurance, pension scheme, annual training allowance, wellbeing allowance, virtual GP, Employee Assistance plan and more. Equal Opportunities Aker Systems fosters a diverse environment that encourages openness in its communications and is committed to providing equal employment opportunity for all people regardless of race, religion, gender or sexual orientation, age, marital status, national origin, citizenship status, disability, veteran status or other personal characteristics. We embrace differences of opinion and diversity because they help challenge us and find new groundbreaking technical solutions.