Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a service that is fully managed, facilitating the development and execution of applications processing streaming data through Apache Kafka. Control-plane operations, including creation, update, and deletion of clusters, are offered by Amazon MSK. The service permits the utilization of Apache Kafka data-plane operations, encompassing data production and consumption. It operates on open-source versions of Apache Kafka, ensuring compatibility with existing applications, tooling, and plugins from both partners and the Apache Kafka community, eliminating the need for alterations in the application code.
In terms of reliability, Amazon MSK is designed to automatically detect and recover from prevalent cluster failure scenarios, ensuring that producer and consumer applications persist in their data writing and reading activities with minimal disruption. Moreover, it aims to optimize data replication processes by attempting to reuse the storage of replaced brokers, thereby minimizing the volume of data that needs to be replicated by Apache Kafka.
Types
There are 2 types of Kafka clusters that AWS allows to create: Provisioned and Serverless.
From the point of view of an attacker you need to know that:
Serverless cannot be directly public (it can only run in a VPN without any publicly exposed IP). However, Provisioned can be configured to get a public IP (by default it doesn't) and configure the security group to expose the relevant ports.
Serverlessonly support IAM as authentication method. Provisioned support SASL/SCRAM (password) authentication, IAM authentication, AWS Certificate Manager (ACM) authentication and Unauthenticated access.
Note that it's not possible to expose publicly a Provisioned Kafka if unauthenticated access is enabled
Enumeration
#Get clustersawskafkalist-clustersawskafkalist-clusters-v2# Check the supported authenticationawskafkalist-clusters|jq-r".ClusterInfoList[].ClientAuthentication"# Get Zookeeper endpointsaws kafka list-clusters | jq -r ".ClusterInfoList[].ZookeeperConnectString, .ClusterInfoList[].ZookeeperConnectStringTls"
# Get nodes and node enspointsawskafkakafkalist-nodes--cluster-arn<cluster-arn>aws kafka kafka list-nodes --cluster-arn <cluster-arn> | jq -r ".NodeInfoList[].BrokerNodeInfo.Endpoints" # Get endpoints
# Get used kafka configsawskafkalist-configurations#Get Kafka config fileawskafkadescribe-configuration--arn<config-arn># Get version of configawskafkadescribe-configuration-revision--arn<config-arn>--revision<version># Get content of config version# If using SCRAN authentication, get used AWS secret name (not secret value)awskafkalist-scram-secrets--cluster-arn<cluster-arn>
Kafka IAM Access (in serverless)
# Guide from https://docs.aws.amazon.com/msk/latest/developerguide/create-serverless-cluster.html# Download Kafkawgethttps://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgztar-xzfkafka_2.12-2.8.1.tgz# In kafka_2.12-2.8.1/libs download the MSK IAM JAR file. cdkafka_2.12-2.8.1/libswgethttps://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar# Create file client.properties in kafka_2.12-2.8.1/binsecurity.protocol=SASL_SSLsasl.mechanism=AWS_MSK_IAMsasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler# Export endpoints addressexport BS=boot-ok2ngypz.c2.kafka-serverless.us-east-1.amazonaws.com:9098## Make sure you will be able to access the port 9098 from the EC2 instance (check VPS, subnets and SG)# Create a topic called msk-serverless-tutorialkafka_2.12-2.8.1/bin/kafka-topics.sh --bootstrap-server $BS --command-config client.properties --create --topic msk-serverless-tutorial --partitions 6
# Send message of every new linekafka_2.12-2.8.1/bin/kafka-console-producer.sh --broker-list $BS --producer.config client.properties --topic msk-serverless-tutorial
# Read messageskafka_2.12-2.8.1/bin/kafka-console-consumer.sh --bootstrap-server $BS --consumer.config client.properties --topic msk-serverless-tutorial --from-beginning
If you are going to have access to the VPC where a Provisioned Kafka is, you could enable unauthorised access, if SASL/SCRAM authentication, read the password from the secret, give some other controlled user IAM permissions (if IAM or serverless used) or persist with certificates.