Summary
Experienced and highly skilled Kafka Engineer with 7 years of expertise in designing, implementing, and optimizing large-scale distributed data streaming platforms. Proven track record in building resilient, high-throughput real-time data pipelines using Apache Kafka, Confluent Platform, and related ecosystem tools. Adept at driving performance improvements, ensuring data reliability, and leveraging cloud infrastructure to support critical business operations.
Experience
Senior Kafka EngineerStreamFlow Systems
- Architected and deployed a multi-tenant Kafka cluster on Kubernetes, scaling to process over 10TB of data daily with 99.99% uptime, reducing operational costs by 15%.
- Led the migration of 5+ critical real-time microservices from legacy messaging systems to Kafka Streams, improving data processing latency by an average of 40ms.
Kafka EngineerDataNexus Inc.
- Designed and implemented robust data ingestion pipelines using Kafka Connect to integrate data from 10+ relational databases, handling over 5 billion events per day.
- Optimized Kafka topic configurations and broker settings across a 60-node cluster, resulting in a 25% reduction in message latency and a 10% increase in throughput.
Associate Data EngineerTech Solutions Co.
- Assisted in the deployment and configuration of initial Kafka clusters for log aggregation and event processing, supporting 50+ internal applications.
- Developed Python scripts for monitoring Kafka consumer lag and partition health, improving operational visibility for data pipelines by 30%.
- Contributed to the development of data warehousing solutions, processing daily ETL jobs for over 100GB of customer data, enhancing reporting capabilities.
Projects
Kafka Connect CDC Framework
- Developed a reusable Kafka Connect framework for Change Data Capture (CDC) from various relational databases (PostgreSQL, MySQL).
- Configured custom transformations and message routing using Kafka Streams to enrich and distribute CDC events to downstream consumers efficiently.
- Achieved near real-time data synchronization for analytical databases, reducing reporting delays by 90% and improving data freshness.
Real-time Anomaly Detection with Kafka Streams
- Implemented a real-time anomaly detection system using Kafka Streams to monitor application logs and metric streams for unusual patterns.
- Utilized K-Means clustering and statistical process control techniques to identify deviations, generating alerts for proactive issue resolution.
- Processed over 10,000 events/second with sub-second latency, significantly improving the detection of critical system failures.
Event-Driven Microservices Boilerplate
- Created a boilerplate project demonstrating event-driven architecture principles using Spring Boot, Apache Kafka, and Avro for schema evolution.
- Designed and implemented idempotent consumers and transactional producers to ensure data consistency across multiple microservices.
- Provided a foundation for rapid development of new event-driven services, reducing initial setup time for new projects by an estimated 40%.
Education
University of California, BerkeleyBachelor of Science in Computer Science
- Graduated with honors, achieving a GPA of 3.8/4.0.
- Completed capstone project: 'Distributed Stream Processing Framework for Real-time Analytics' using Apache Storm.
- Awarded Dean's List recognition for multiple semesters for outstanding academic performance.
Skills
Data Streaming & Messaging
Apache KafkaConfluent PlatformKafka StreamsksqlDBApache FlinkApache Pulsar
Cloud Platforms & Orchestration
AWS (EC2, S3, EKS, MSK)KubernetesDockerHelmTerraformAzure (Event Hubs)
Programming Languages
JavaScalaPythonGoBash ScriptingSQL
Databases & Storage
Apache CassandraPostgreSQLMongoDBElasticsearchRedisHDFS
DevOps & Monitoring
JenkinsGitLab CI/CDPrometheusGrafanaELK StackAnsible