Liam Murphy | Senior Kafka Engineer at StreamFlow Systems

Summary

Senior Kafka EngineerStreamFlow Systems

San Francisco, CAAug 2022Present

Architected and deployed a multi-tenant Kafka cluster on Kubernetes, scaling to process over 10TB of data daily with 99.99% uptime, reducing operational costs by 15%.
Led the migration of 5+ critical real-time microservices from legacy messaging systems to Kafka Streams, improving data processing latency by an average of 40ms.

Kafka EngineerDataNexus Inc.

San Francisco, CAAug 2019Jul 2022

Designed and implemented robust data ingestion pipelines using Kafka Connect to integrate data from 10+ relational databases, handling over 5 billion events per day.
Optimized Kafka topic configurations and broker settings across a 60-node cluster, resulting in a 25% reduction in message latency and a 10% increase in throughput.

Associate Data EngineerTech Solutions Co.

Mountain View, CAAug 2017Jul 2019

Assisted in the deployment and configuration of initial Kafka clusters for log aggregation and event processing, supporting 50+ internal applications.
Developed Python scripts for monitoring Kafka consumer lag and partition health, improving operational visibility for data pipelines by 30%.
Contributed to the development of data warehousing solutions, processing daily ETL jobs for over 100GB of customer data, enhancing reporting capabilities.

Developed a reusable Kafka Connect framework for Change Data Capture (CDC) from various relational databases (PostgreSQL, MySQL).
Configured custom transformations and message routing using Kafka Streams to enrich and distribute CDC events to downstream consumers efficiently.
Achieved near real-time data synchronization for analytical databases, reducing reporting delays by 90% and improving data freshness.

Real-time Anomaly Detection with Kafka Streams

Sep 2021Mar 2022

Implemented a real-time anomaly detection system using Kafka Streams to monitor application logs and metric streams for unusual patterns.
Utilized K-Means clustering and statistical process control techniques to identify deviations, generating alerts for proactive issue resolution.
Processed over 10,000 events/second with sub-second latency, significantly improving the detection of critical system failures.

Created a boilerplate project demonstrating event-driven architecture principles using Spring Boot, Apache Kafka, and Avro for schema evolution.
Designed and implemented idempotent consumers and transactional producers to ensure data consistency across multiple microservices.
Provided a foundation for rapid development of new event-driven services, reducing initial setup time for new projects by an estimated 40%.

University of California, BerkeleyBachelor of Science in Computer Science

Berkeley, CAAug 2013May 2017

Graduated with honors, achieving a GPA of 3.8/4.0.
Completed capstone project: 'Distributed Stream Processing Framework for Real-time Analytics' using Apache Storm.
Awarded Dean's List recognition for multiple semesters for outstanding academic performance.

Data Streaming & Messaging

Apache KafkaConfluent PlatformKafka StreamsksqlDBApache FlinkApache Pulsar

Cloud Platforms & Orchestration

AWS (EC2, S3, EKS, MSK)KubernetesDockerHelmTerraformAzure (Event Hubs)

Programming Languages

JavaScalaPythonGoBash ScriptingSQL

Databases & Storage

Apache CassandraPostgreSQLMongoDBElasticsearchRedisHDFS

DevOps & Monitoring

JenkinsGitLab CI/CDPrometheusGrafanaELK StackAnsible