Kafka Admin

Job Overview:

• We are seeking a skilled Kafka Administrator to oversee and support our enterprise Kafka environments. This role includes responsibilities such as managing cluster configurations, handling topic and partition management, administering brokers, monitoring consumer lag, ensuring security, optimizing performance, executing upgrades, and troubleshooting production issues.
• The ideal candidate will possess practical experience with Apache Kafka, KRaft, Linux, monitoring tools, automation frameworks, and cloud infrastructure, ensuring the delivery of highly available and dependable messaging solutions.

Primary Duties:

• Provision, configure, and manage Kafka clusters, including Apache Kafka KRaft or ZooKeeper for legacy systems, as well as Confluent Platform components like Schema Registry, Kafka Connect, ksqlDB, REST Proxy, Control Center, and Confluent Cloud resources.
• Guarantee high availability (HA) and resilience across multi-AZ regional deployments, managing partition replication, minimum in-sync replicas (ISR), and rack awareness.
• Implement and uphold client quotas, broker quotas, throttling, and back pressure strategies.
• Plan and carry out upgrades and patching with minimal or zero downtime, ensuring version currency against end-of-life and security advisories.
• Manage backup and restore workflows for metadata, including cluster configurations, Schema Registry, and Connect configurations.
• Establish authentication protocols such as mTLS, SASL, SCRAM, OAUTHBEARER with Identity Providers, or Kerberos for legacy systems, alongside authorization using Kafka ACLs and Confluent RBAC.
• Enforce encryption both in transit and at rest, including secret management through Vault, Azure Key Vault (AKV), Systems Manager (SSM), secure endpoint exposure, and private connectivity options like VPC peering and PrivateLink.
• Oversee topic naming conventions, retention policies, schema evolution rules, and ensure compliance with PII handling.
• Maintain audit trails for brokers, Schema Registry, Control Center, and Confluent Cloud audit logs, and ensure compliance with regulations such as SOX, GDPR, HIPAA, and PCI where applicable.
• Coordinate vulnerability management, monitor CVEs, and align hardening baselines with CIS and STIG standards.
• Deploy and sustain metrics pipelines using JMX, Prometheus, Grafana, and log management tools such as ELK and OpenSearch, as well as tracing with OpenTelemetry when appropriate.
• Define and monitor Service Level Objectives (SLOs) and Service Level Indicators (SLIs), measuring metrics such as produce/consume latency, end-to-end lag, broker CPU usage, heap memory, file handles, network throughput, and garbage collection pauses.
• Analyze broker hotspots, partition skews, large message handling, compaction, and retention settings, optimizing garbage collection and JVM configurations for consistent throughput.
• Implement consumer lag monitoring and alerting mechanisms with actionable runbooks to prevent data loss or breaches of service level agreements (SLAs).
• Model workload growth, including topic partition scaling, storage IOPS, network egress/ingress, and broker sizing.
• For Confluent Cloud, optimize environment setup, cluster configurations, topic hierarchies, throughput tiers, retention costs, and data flow egress, ensuring the right sizing between Dedicated and Standard clusters.
• Design partitioning strategies aimed at achieving throughput goals while balancing consumer concurrency and avoiding anti-patterns associated with small partitions.
• Automate lifecycle management of clusters and topics utilizing Terraform providers for Kafka, Confluent, Helm, and GitOps workflows.

Technical Qualifications:

• 2 to 5 years of experience in platform SRE or DevOps, with at least 2 years focused on Kafka and Confluent administration in a production environment.
• Strong hands-on expertise with Apache Kafka internals, including brokers, controllers, KRaft, partitions, replication, log segments, compaction, and retention mechanisms.
• Experience with the Confluent Platform, including Schema Registry, Kafka Connect, ksqlDB, Control Center, Confluent REST APIs, and managing Confluent Cloud environments, clusters, RBAC, audit logs, and networking.
• Proficient in using Kafka CLI tools and AdminClient, with thorough knowledge of ACLs, quotas, transactions, idempotent producers, and exactly-once semantics scenarios.
• Familiarity with Prometheus, Grafana, JMX exporters, and alerting systems such as Alertmanager.
• Experience in JVM tuning and Linux performance optimization, including NUMA, file descriptors, page cache, disk RAID, XFS, and network tuning with TCP buffers and MTU settings.
• Proficient in Linux administration, including systemd, journald, kernel parameters, virtualization, containers, Docker, and Kubernetes.
• Knowledge of cloud networking concepts, including VPC, VNET peering, transit gateways, PrivateLink, and Private Service Connect, along with multi-region architecture experience.
• Preferred qualifications include being a Confluent Certified Administrator for Apache Kafka (CCAAK), Certified Kubernetes Administrator (CKA), Certified Kubernetes Application Developer (CKAD), and holding major cloud certifications from Azure, AWS, or GCP.
• Experience with Confluent for Kubernetes (CFK) for operator-based deployments is a plus.

Additional Information:

• Infosys is a worldwide leader in next-generation digital services and consulting.
• We assist clients in over 50 countries as they navigate their digital transformation journey.
• With over forty years of experience managing global enterprises, we adeptly guide clients through their digital journey.
• Our approach involves empowering enterprises with an AI-driven core, aiding in the prioritization of change execution.
• We also enable businesses to achieve agile digital transformation at scale, resulting in exceptional performance and customer satisfaction.
• Our commitment to continuous learning drives ongoing improvement by building and sharing digital skills, expertise, and innovations from our ecosystem.
• Infosys is committed to providing equal employment opportunities to all applicants and employees, regardless of race, color, gender, gender identity, sexual orientation, religious beliefs, national origin, pregnancy or related medical conditions, veteran status, disability, or familial status related to protected veterans.

Desired Skills:

Technology->Integration->Confluent Kafka, Technology->Cloud Platform->AWS Data Analytics->Amazon Managed Streaming for Apache Kafka