Kafka Admin

Job Overview:

• We are seeking a seasoned Kafka Administrator to oversee and support enterprise-level Kafka environments. This role includes responsibilities such as cluster configuration, topic and partition management, broker administration, consumer lag monitoring, security measures, performance optimization, upgrades, and troubleshooting in production settings.
• The preferred candidate will possess hands-on expertise with Apache Kafka, KRaft, Linux, monitoring tools, automation frameworks, and cloud infrastructures, ensuring the delivery of highly available and dependable messaging platforms.

Primary Duties:

• Provision, configure, and manage Kafka clusters, whether they are based on Apache Kafka KRaft or legacy ZooKeeper, including components like the Confluent Platform, Schema Registry, Kafka Connect, ksqlDB, REST Proxy, Control Center, and Confluent Cloud resources.
• Guarantee high availability (HA) and resilience across multi-AZ region deployments, overseeing partition replication, minimum in-sync replicas (ISR), and rack awareness.
• Implement and uphold client quotas, broker quotas, throttling mechanisms, and back pressure strategies.
• Plan and carry out upgrades and patching processes with minimal to zero downtime, ensuring version compliance against end-of-life (EOL) policies and security advisories.
• Manage backup and restore workflows for metadata, including cluster configurations, Schema Registry, and Connect configurations.
• Establish authentication protocols such as mTLS, SASL, SCRAM, and OAUTHBEARER with identity providers or Kerberos for legacy systems, along with authorization via Kafka ACLs and Confluent RBAC.
• Implement encryption for data both in transit and at rest, manage secrets with tools like Vault, Azure Key Vault (AKV), AWS Systems Manager (SSM), and ensure secure endpoint exposure and private connectivity via VPC peering and PrivateLink.
• Enforce naming conventions for topics, retention policies, schema evolution guidelines, and handling of personally identifiable information (PII).
• Maintain audit trails for brokers, Schema Registry, Control Center, and Confluent Cloud audit logs, ensuring compliance with standards such as SOX, GDPR, HIPAA, and PCI where applicable.
• Oversee vulnerability management, including CVE monitoring, hardening baselines, and alignment with CIS and STIG standards.
• Deploy and sustain metrics pipelines using JMX, Prometheus, Grafana, and logging solutions like ELK or OpenSearch, including tracing with OpenTelemetry where appropriate.
• Define and monitor service level objectives (SLOs) and service level indicators (SLIs), focusing on metrics such as produce/consume latency, end-to-end lag, broker CPU usage, heap space, file handles, network throughput, and garbage collection pauses.
• Analyze broker hotspots, partition skew, large message handling, compaction, and retention settings, while fine-tuning garbage collection and JVM settings for sustained performance.
• Implement consumer lag monitoring and alerting systems with actionable runbooks to avert data loss or breaches of service level agreements (SLAs).
• Model workload growth by assessing topic partition scaling, storage IOPS, network egress/ingress, and broker sizing.
• For Confluent Cloud, optimize the environment by managing cluster and topic hierarchy, throughput tiers, retention costs, and data flow egress, ensuring appropriate sizing of dedicated versus standard clusters.
• Design partitioning strategies to achieve throughput objectives while balancing consumer concurrency and avoiding anti-patterns associated with small partitions.
• Automate the lifecycle of clusters and topics utilizing Terraform providers for Kafka, Confluent, Helm, and GitOps workflows.

Technical Qualifications:

• 9 to 11 years in platform Site Reliability Engineering (SRE) or DevOps, with 9 years specifically focused on Kafka and Confluent administration in production environments.
• Strong practical knowledge of Apache Kafka internals, including brokers, controllers, KRaft, partitions, replication, log segments, compaction, and retention mechanisms.
• Experience with the Confluent Platform, including Schema Registry, Kafka Connect, ksqlDB, Control Center, Confluent REST APIs, and administration of Confluent Cloud environments and clusters with RBAC and audit logs.
• Expertise with Kafka CLI tools and AdminClient, along with in-depth understanding of ACLs, quotas, transactions, idempotent producers, and exactly-once processing scenarios.
• Familiarity with Prometheus, Grafana, JMX exporters, and alerting systems such as Alertmanager.
• Proficient in JVM tuning, Linux performance optimization, NUMA configurations, file descriptor management, page cache, disk RAID, XFS, and network tuning including TCP buffers and MTU settings where applicable.
• Solid Linux administration skills, including experience with systemd, journald, kernel parameters, virtualization, and containerization using Docker and Kubernetes.
• Experience with cloud networking concepts such as VPC, VNET peering, transit gateways, PrivateLink, and Private Service Connect in multi-region architectures.
• Preferred certifications include Confluent Certified Administrator for Apache Kafka (CCAAK), Certified Kubernetes Administrator (CKA), Certified Kubernetes Application Developer (CKAD), HashiCorp Terraform Associate, and major cloud certifications for Azure, AWS, or GCP.
• Experience with Confluent for Kubernetes (CFK) in operator-based deployments is a plus.

Additional Information:

• Infosys stands at the forefront of next-generation digital services and consulting.
• We empower clients across more than 50 countries to navigate their digital transformation journeys.
• With over 40 years of expertise in managing global enterprise systems, we expertly guide our clients through their digital evolution.
• We achieve this by equipping the enterprise with an AI-driven core that prioritizes change execution.
• Additionally, we empower businesses with agile digital solutions at scale, delivering exceptional performance and enhancing customer satisfaction.
• Our commitment to continuous improvement is driven by an always-on learning agenda, fostering the development and transfer of digital skills, expertise, and innovative ideas from our ecosystem.
• Infosys is an equal opportunity employer, welcoming applicants and employees without discrimination based on race, color, sex, gender identity, sexual orientation, religious beliefs, national origin, pregnancy, childbirth or related medical conditions, status as a protected veteran or family member of a protected veteran, or disability.

Desired Skills:

• Technology->Cloud Platform->AWS
• Data Analytics->Amazon Managed Streaming for Apache Kafka
• Technology->Integration->Confluent Kafka