Kafka Admin

Job Overview:

• We are in search of a skilled Kafka Administrator to oversee and enhance enterprise Kafka environments. This includes managing cluster configurations, topic and partition oversight, broker administration, monitoring consumer lag, ensuring security, performance tuning, executing upgrades, and resolving production issues.
• The perfect candidate will possess practical experience with Apache Kafka, KRaft, Linux, monitoring tools, automation frameworks, and cloud infrastructure to maintain highly available and dependable messaging platforms.

Core Responsibilities:

• Provision, configure, and manage Kafka clusters, whether using Apache Kafka KRaft or ZooKeeper-backed legacy components like Confluent Platform, Schema Registry, Kafka Connect, ksqlDB, REST Proxy, Control Center, and Confluent Cloud resources.
• Guarantee high availability (HA) and resilience in multi-AZ region deployments by managing partition replication and ensuring minimum in-sync replicas (ISR) and rack awareness.
• Implement and maintain client and broker quotas, throttling, and strategies for back pressure.
• Plan and execute upgrades and patches with minimal to zero downtime while keeping current versions in line with end-of-life (EOL) and security advisories.
• Manage backup and restore workflows for metadata, including cluster configurations, Schema Registry, and Connect configurations.
• Implement authentication methods such as mTLS, SASL SCRAM, OAUTHBEARER with Identity Providers (IdP), or Kerberos for legacy systems, along with authorization via Kafka ACLs and Confluent RBAC.
• Enforce encryption both in transit and at rest, along with secret management strategies using Vault, Azure Key Vault (AKV), AWS Systems Manager (SSM), secure endpoint exposure, and private connectivity like VPC peering and PrivateLink.
• Establish governance for topic naming conventions, retention policies, schema evolution rules, backward compatibility, and personally identifiable information (PII) handling.
• Maintain audit trails for brokers, Schema Registry, Control Center, and Confluent Cloud audit logs, along with compliance documentation for SOX, GDPR, HIPAA, and PCI where applicable.
• Coordinate vulnerability management, including CVE tracking, hardening baselines, and alignment with CIS and STIG standards.
• Deploy and maintain metrics pipelines utilizing JMX, Prometheus, Grafana, ELK, OpenSearch, and tracing via OpenTelemetry as needed.
• Define and monitor Service Level Objectives (SLOs) and Service Level Indicators (SLIs), focusing on metrics such as production consumption latency, end-to-end lag, broker CPU usage, heap size, file handles, network throughput, and garbage collection pauses.
• Analyze broker hotspots, partition skewing, large message handling, compaction, and retention settings, optimizing garbage collection and Java Virtual Machine (JVM) settings for sustained throughput.
• Implement consumer lag monitoring and alerting with actionable runbooks to mitigate data loss or breaches of service level agreements (SLAs).
• Model workload growth considering topic partition scaling, storage IOPS, network egress/ingress, and broker sizing.
• For Confluent Cloud, optimize the environment, cluster configurations, topic hierarchy, throughput tiers, retention costs, and data flow egress while determining the appropriate size for Dedicated versus Standard clusters.
• Design partitioning strategies to achieve throughput targets while balancing consumer concurrency and preventing small partition anti-patterns.
• Automate the lifecycle of clusters and topics using Terraform providers for Kafka, Confluent, Helm, and GitOps workflows.

Technical Qualifications:

• 7 to 9 years of experience in platform SRE or DevOps, with at least 7 years focused on Kafka and Confluent administration in production environments.
• Extensive hands-on experience with Apache Kafka internals, including brokers, controllers, KRaft, partitions, replication, log segments, compaction, and retention.
• Familiarity with the Confluent Platform, including Schema Registry, Kafka Connect, ksqlDB, Control Center, Confluent REST APIs, and administration of clusters, RBAC, audit logs, and networking.
• Proficient with Kafka CLI tools and AdminClient, possessing deep knowledge of ACLs, quotas, transactions, idempotent producers, and exactly-once semantics scenarios.
• Experience with Prometheus, Grafana, JMX exporters, and alerting systems such as Alertmanager.
• Expertise in JVM tuning, Linux performance optimization, NUMA management, file descriptors, page cache, disk RAID, XFS, and network tuning (TCP buffers, MTU, jumbo frames) where applicable.
• Strong Linux administration skills, including systemd, journald, kernel parameters, virtualization, and containers (Docker, Kubernetes).
• Knowledge of cloud networking concepts including VPC, VNET peering, transit gateways, PrivateLink, and Private Service Connect in multi-region architectures.
• Preferred certifications include Confluent Certified Administrator for Apache Kafka (CCAAK), Certified Kubernetes Administrator (CKA), Certified Kubernetes Application Developer (CKAD), HashiCorp Terraform Associate, and major cloud certifications in Azure, AWS, or GCP.
• Experience with Confluent for Kubernetes (CFK) for operator-based deployments is a plus.

Additional Information:

• Infosys is a global leader in next-generation digital services and consulting.
• We assist clients in over 50 countries with their digital transformation journeys.
• With more than four decades of experience in managing global enterprises, we expertly guide our clients throughout their digital evolution.
• Our approach empowers enterprises with an AI-driven core that prioritizes the execution of change effectively.
• We also enable businesses to scale agile digital solutions to achieve exceptional performance and customer satisfaction.
• Our continuous learning agenda fosters improvement by building and sharing digital skills, expertise, and innovative ideas from our ecosystem.
• Infosys is committed to providing equal employment opportunities to all applicants and employees, irrespective of race, color, sex, gender identity, sexual orientation, religious practices, national origin, pregnancy-related conditions, veteran status, or disability.

Desired Skills:

• Technology -> Cloud Platform -> AWS
• Data Analytics -> Amazon Managed Streaming for Apache Kafka
• Technology -> Integration -> Confluent Kafka