Data Streaming Engineer

Synopsis

The Manager Software Engineering (Integration Engineer) plays a pivotal role in driving the development and support of high-performance, event-driven applications and data streaming solutions. Reporting to the Senior Manager Software Engineering Chapter, this role is responsible for ensuring that all software engineering deliverables adhere to the standards and best practices defined by the Software Engineering chapter. 

The purpose of this role is to lead the design, implementation, and optimization of real-time data processing pipelines using technologies such as Apache Kafka, Apache Flink, and Confluent Stream Processing. The Manager will collaborate closely with cross-functional teams—including data engineers, architects, and business stakeholders to translate complex business requirements into scalable, secure, and resilient streaming solutions.

Accountabilities

• Provide Software Engineering delivery, support capabilities and guidance and technical assistance to the customers i.e. guide teams.
• Design, develop, and deploy event-driven applications using Confluent Stream Processing, Apache Flink, and Kafka.
• Build real-time data ingestion, transformation, and processing pipelines that meet performance, scalability, and reliability requirements.
• Collaborate with data engineers, architects, and business stakeholders to define streaming use cases and implement solutions.
• Optimize CEP applications for throughput, latency, and fault tolerance.
• Implement monitoring, alerting, and performance tuning for streaming applications.
• Ensure best practices for data governance, schema management, and security in Kafka/streaming ecosystems.
• Provide support for production systems, troubleshooting issues, and ensuring high availability of event-processing pipelines.
• Lead a team of Software Associates/Engineers
• Contribute to the low-level architecture/technical design aligned to the high-level solution and business requirements
• Work within the agile development team to analyse and decompose stories into tasks for simpler implementation
• Proactively identify new frameworks/tools/processes to enhance the efficiency of the development team
• Responsible for the end-to-end quality of your own deliverables

Education & Experience

• Bachelor’s Degree in related field such as Computer Science, Computer Engineering or Software Engineering.
• 5+ years of experience in real-time event processing or streaming data engineering.
• Hands-on expertise in Confluent Platform (Kafka Streams, kSQL, Schema Registry, Connectors).
• Strong development experience with Apache Flink for stream processing (stateful computations, windowing, event-time processing).
• Proficiency in programming languages such as Java, Scala, or Python for building Flink/Kafka applications.
• Solid understanding of distributed systems, event-driven architecture, and microservices.
• Experience with cloud-based streaming solutions (AWS Kinesis, Azure Event Hubs, GCP Pub/Sub) is a plus.
• Familiarity with DevOps practices, CI/CD pipelines, containerization (Docker/Kubernetes).
• Strong problem-solving skills, with the ability to work in a fast-paced, agile environment.
• Knowledge of complex event pattern detection and advanced analytics over streaming data.