Kafka Admin

Job Overview:

• We are seeking a skilled Kafka Administrator to oversee and enhance enterprise Kafka ecosystems, which includes cluster setup, topic and partition oversight, broker management, monitoring consumer lag, ensuring security, optimizing performance, executing upgrades, and resolving production issues.
• The successful candidate will possess hands-on expertise with Apache Kafka, KRaft, Linux, monitoring solutions, automation frameworks, and cloud infrastructures to guarantee dependable and highly available messaging systems.

Core Responsibilities:

• Provision, configure, and oversee Kafka clusters, utilizing either Apache Kafka KRaft or ZooKeeper for legacy systems, along with components from the Confluent Platform such as Schema Registry, Kafka Connect, ksqlDB, REST Proxy, Control Center, and resources from Confluent Cloud.
• Guarantee high availability (HA) and resilience in deployments across multiple availability zones, managing partition replication and in-sync replicas (ISR) with rack awareness.
• Establish and uphold client and broker quotas, as well as implement throttling and back pressure strategies.
• Plan and execute upgrades and patches with minimal to zero downtime, ensuring that versions remain current in relation to end-of-life (EOL) and security advisories.
• Oversee backup and restore processes for metadata, including cluster configurations, Schema Registry, and Connect configurations.
• Implement authentication mechanisms such as mTLS, SASL, SCRAM, OAUTHBEARER with Identity Provider (IdP) or Kerberos for legacy systems, and manage authorization through Kafka ACLs and Confluent RBAC.
• Enforce data encryption both in transit and at rest, along with secret management through Vault, Azure Key Vault (AKV), and Systems Manager (SSM), ensuring secure endpoint exposure and private connectivity using VPC peering and PrivateLink.
• Establish governance for topic naming conventions, retention policies, schema evolution rules, and handling of Personally Identifiable Information (PII).
• Maintain comprehensive audit trails including broker, Schema Registry, Control Center, and Confluent Cloud audit logs, in alignment with compliance requirements such as SOX, GDPR, HIPAA, and PCI as applicable.
• Manage vulnerability assessments, keeping an eye on CVEs, and align with hardening baselines and CIS STIG standards.
• Deploy and sustain metrics pipelines using tools like JMX, Prometheus, Grafana, and logging solutions such as ELK or OpenSearch, along with tracing through OpenTelemetry where applicable.
• Define and monitor service level objectives (SLOs) and service level indicators (SLIs), tracking metrics such as produce/consume latency, end-to-end lag, broker CPU usage, heap size, file handles, network throughput, and garbage collection (GC) pauses.
• Analyze broker hotspots, partition skew, large message management, compaction, retention settings, and tune GC and JVM parameters to ensure sustained throughput.
• Implement consumer lag monitoring and alerting systems along with actionable runbooks to prevent data loss or SLA violations.
• Model workload growth, scaling topics and partitions, managing storage IOPS, network ingress/egress, and broker sizing.
• For Confluent Cloud, optimize the environment by managing cluster, topic structure, throughput tiers, retention costs, and data flow egress, ensuring appropriate sizing of Dedicated versus Standard clusters.
• Design partitioning strategies that achieve throughput objectives while balancing consumer concurrency and preventing anti-patterns associated with small partitions.
• Automate the lifecycle of clusters and topics using Terraform providers for Kafka, Confluent, Helm, and GitOps workflows.

Technical Qualifications:

• 5-7 years of experience in platform SRE or DevOps, with a minimum of 5 years specifically focused on Kafka Confluent administration in production environments.
• In-depth knowledge of Apache Kafka internals, including brokers, controllers, KRaft, partitions, replication, log segments, compaction, and retention.
• Experience with the Confluent Platform, including Schema Registry, Kafka Connect, ksqlDB, Control Center, Confluent REST APIs, and management of clusters and RBAC in Confluent Cloud.
• Proficient in using Kafka CLI tools and AdminClient, with a strong understanding of ACLs, quotas, transactions, idempotent producers, and scenarios involving exactly-once semantics.
• Familiarity with monitoring tools such as Prometheus, Grafana, JMX exporters, and alerting systems like Alertmanager.
• Experience with JVM tuning, Linux performance optimization, including NUMA, file descriptors, page cache, disk RAID, XFS, and network tuning techniques such as TCP buffers and MTU settings.
• Strong Linux administration skills, including systemd, journald, kernel parameters, virtualization, and container management with Docker and Kubernetes.
• Knowledge of cloud networking, including VPC, VNET peering, transit gateways, PrivateLink, and multi-region architecture.
• Preferred qualifications include certification as a Confluent Certified Administrator for Apache Kafka (CCAAK), CKA, CKAD, and major cloud certifications from Azure, AWS, or GCP.
• Experience with Confluent for Kubernetes (CFK) for operator-based deployments is also desirable.

Company Overview:

• Infosys stands as a global leader in next-generation digital services and consulting.
• We assist clients in over 50 countries in navigating their digital transformation journeys.
• With a rich history spanning over four decades in managing global enterprise systems, we expertly guide our clients through their digital transitions.
• Our approach involves empowering enterprises with an AI-driven core that prioritizes effective change execution.
• We also enable businesses with agile digital solutions at scale, achieving unprecedented performance and customer satisfaction.
• Our ongoing commitment to learning fosters continuous improvement through developing and sharing digital skills, expertise, and innovative ideas from our ecosystem.
• Infosys is proud to provide equal employment opportunities, ensuring that all applicants and employees are considered without regard to race, color, gender identity, sexual orientation, religious beliefs, national origin, pregnancy, or disability status.

Desired Skills:

• Technology->Integration->Confluent Kafka
• Technology->Cloud Platform->AWS
• Data Analytics->Amazon Managed Streaming for Apache Kafka