Amazon Managed Streaming for Apache Kafka (Amazon MSK) ni huduma ambayo imepangiliwa kabisa, ikirahisisha maendeleo na utekelezaji wa programu zinazoprocess data ya mtiririko kupitia Apache Kafka. Operesheni za kiwango cha udhibiti, ikiwa ni pamoja na uundaji, uppdatering, na kufuta makundi, zinatolewa na Amazon MSK. Huduma inaruhusu matumizi ya operesheni za kiwango cha data za Apache Kafka, zikiingiza uzalishaji na utumiaji wa data. Inafanya kazi kwenye toleo za chanzo wazi za Apache Kafka, ikahakikisha utangamano na programu zilizopo, zana, na programu-jalizi kutoka kwa washirika na jamii ya Apache Kafka, ikiondoa haja ya marekebisho katika nambari ya programu.
Kuhusu uaminifu, Amazon MSK imeundwa kugundua na kurejesha moja kwa moja kutoka kwa hali za kawaida za kushindwa kwa kundi, ikahakikisha kuwa programu za uzalishaji na utumiaji wa data zinaendelea na shughuli zao za kuandika na kusoma data bila kuvurugika sana. Zaidi ya hayo, lengo lake ni kuboresha mchakato wa kuzidisha data kwa kujaribu kutumia upya uhifadhi wa wakala waliobadilishwa, hivyo kupunguza kiasi cha data inayohitaji kuzidishwa na Apache Kafka.
Aina
Kuna aina 2 za makundi ya Kafka ambayo AWS inaruhusu kuunda: Imepangiliwa na Bila seva.
Kutoka mtazamo wa mshambuliaji unahitaji kujua kwamba:
Bila seva hawezi kuwa wazi moja kwa moja (inaweza tu kukimbia kwenye VPN bila anwani ya IP inayoweza kuonekana hadharani). Hata hivyo, Imepangiliwa inaweza kusanidiwa kupata anwani ya IP ya umma (kwa chaguo-msingi haina) na kusanidi kikundi cha usalama ku kufunua bandari muhimu.
Bila seva inasaidia IAM pekee kama njia ya uthibitishaji. Imepangiliwa inasaidia uthibitishaji wa SASL/SCRAM (nywila), uthibitishaji wa IAM, Meneja wa Cheti cha AWS (ACM) uthibitishaji na ufikiaji usiothibitishwa.
Tafadhali kumbuka kwamba haiwezekani kufunua hadharani Kafka ya Imepangiliwa ikiwa ufikiaji usiothibitishwa umewezeshwa
Uchambuzi
#Get clustersawskafkalist-clustersawskafkalist-clusters-v2# Check the supported authenticationawskafkalist-clusters|jq-r".ClusterInfoList[].ClientAuthentication"# Get Zookeeper endpointsaws kafka list-clusters | jq -r ".ClusterInfoList[].ZookeeperConnectString, .ClusterInfoList[].ZookeeperConnectStringTls"
# Get nodes and node enspointsawskafkakafkalist-nodes--cluster-arn<cluster-arn>aws kafka kafka list-nodes --cluster-arn <cluster-arn> | jq -r ".NodeInfoList[].BrokerNodeInfo.Endpoints" # Get endpoints
# Get used kafka configsawskafkalist-configurations#Get Kafka config fileawskafkadescribe-configuration--arn<config-arn># Get version of configawskafkadescribe-configuration-revision--arn<config-arn>--revision<version># Get content of config version# If using SCRAN authentication, get used AWS secret name (not secret value)awskafkalist-scram-secrets--cluster-arn<cluster-arn>
Upatikanaji wa IAM wa Kafka (katika serverless)
# Guide from https://docs.aws.amazon.com/msk/latest/developerguide/create-serverless-cluster.html# Download Kafkawgethttps://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgztar-xzfkafka_2.12-2.8.1.tgz# In kafka_2.12-2.8.1/libs download the MSK IAM JAR file.cdkafka_2.12-2.8.1/libswgethttps://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar# Create file client.properties in kafka_2.12-2.8.1/binsecurity.protocol=SASL_SSLsasl.mechanism=AWS_MSK_IAMsasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler# Export endpoints addressexport BS=boot-ok2ngypz.c2.kafka-serverless.us-east-1.amazonaws.com:9098## Make sure you will be able to access the port 9098 from the EC2 instance (check VPS, subnets and SG)# Create a topic called msk-serverless-tutorialkafka_2.12-2.8.1/bin/kafka-topics.sh --bootstrap-server $BS --command-config client.properties --create --topic msk-serverless-tutorial --partitions 6
# Send message of every new linekafka_2.12-2.8.1/bin/kafka-console-producer.sh --broker-list $BS --producer.config client.properties --topic msk-serverless-tutorial
# Read messageskafka_2.12-2.8.1/bin/kafka-console-consumer.sh --bootstrap-server $BS --consumer.config client.properties --topic msk-serverless-tutorial --from-beginning
Ikiwa utakuwa na ufikiaji wa VPC ambapo Kafka iliyopangwa iko, unaweza kuwezesha ufikiaji usiothibitishwa, ikiwa uthibitishaji wa SASL/SCRAM, soma nenosiri kutoka kwa siri, toa ruhusa zingine za IAM kwa mtumiaji aliye na udhibiti (ikiwa IAM au serverless inatumika) au thibitisha na vyeti.