- #APPLE JAVA AND KAFKA SUNNYVALE HOW TO#
- #APPLE JAVA AND KAFKA SUNNYVALE INSTALL#
- #APPLE JAVA AND KAFKA SUNNYVALE PASSWORD#
Use SCP to copy the file to your HDInsight cluster: scp. On your development environment, change to the Streaming directory and use the following to create a jar for this project: mvn clean package NOTE: The streaming example expects that you have already setup the test topic from the previous section. This returns a list of the random sentences, along with a count of how many are read. Use the producer-consumer to read the records that were just written: java -jar kafka-producer-consumer.jar consumer test $KAFKABROKERS Use the producer-consumer example to write records to the topic: java -jar kafka-producer-consumer.jar producer test $KAFKABROKERSĪ counter displays how many records have been written. Use the following to create this topic: /usr/hdp/current/kafka-broker/bin/kafka-topics.sh -create -replication-factor 2 -partitions 8 -topic test -zookeeper $KAFKAZKHOSTS In this example, the list of hosts is trimmed to two entries. A connection to one broker or Zookeeper node can be used to learn about the others. IMPORTANT: You don't have to provide all broker or Zookeeper nodes. You should always retrieve the Zookeeper and Broker information before working with Kafka. NOTE: This information may change as you perform scaling operations on the cluster, as this adds and removes worker nodes. The following is an example of the contents of $KAFKABROKERS: .:9092.:9092 The following is an example of the contents of $KAFKAZKHOSTS: .:2181.:2181 Use the following to verify that the environment variables have been correctly populated: echo '$KAFKAZKHOSTS='$KAFKAZKHOSTS
#APPLE JAVA AND KAFKA SUNNYVALE INSTALL#
sudo apt -y install jqĮxport KAFKAZKHOSTS=`curl -sS -u admin:$PASSWORD -G | jq -r '.HostRoles.host_name):2181"] | join(",")' | cut -d',' -f1,2`Įxport KAFKABROKERS=`curl -sS -u admin:$PASSWORD -G | jq -r '.HostRoles.host_name):9092"] | join(",")' | cut -d',' -f1,2`
Replace KAFKANAME with the name of the Kafka on HDInsight cluster.
#APPLE JAVA AND KAFKA SUNNYVALE PASSWORD#
Replace PASSWORD with the login (admin) password for the cluster.
Note that JQ is also installed, as it makes it easier to parse the JSON returned from Ambari. You need this information when working with Kafka. Use SSH to connect to the cluster: ssh the following commands in the SSH session to get the Zookeeper hosts and Kafka brokers for the cluster. When prompted enter the password for the SSH user. target/ SSHUSER with the SSH user for your cluster, and replace CLUSTERNAME with the name of your cluster. Use SCP to upload the file to the Kafka cluster: scp.
This was tested with Oracle Java 8, but should work under things like OpenJDK as well.Īssuming Java and Maven are both in the path, and everything is configured fine for JAVA_HOME, use the following commands to build the consumer and producer example: cd Producer-ConsumerĪ file named is now available in the target directory. To run the consumer and producer example, use the following steps:įork/Clone the repository to your development environment.
NOTE: This both projects assume Kafka 0.10.0, which is available with Kafka on HDInsight cluster version 3.6. Streaming: This contains an application that uses the Kafka streaming API (in Kafka 0.10.0 or higher) that reads data from the test topic, splits the data into words, and writes a count of words into the wordcounts topic. Producer-Consumer: This contains a producer and consumer that use a Kafka topic named test. There are two projects included in this repository:
#APPLE JAVA AND KAFKA SUNNYVALE HOW TO#
The examples in this repository demonstrate how to use the Kafka Consumer, Producer, and Streaming APIs with a Kafka on HDInsight cluster.
0 kommentar(er)
