[聚合文章] Storm 与 Kafka 的整合之二：Kafka

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. - - Official website

为什么要有 Kafka?

分布式

具备经济、快速、可靠、易扩充、数据共享、设备共享、通讯方便、灵活等分布式所具备的特性

高吞吐量

同时为发布者和订阅者提高吞吐量

高可靠性

支持多个订阅者，当订阅失败的时候，能够自动均衡订阅者

离线 & 实时性

能将消息持久化，进行批量处理

Kafka 工作机制

一些主要概念

• Topic（主题）

  A topic is a category or feed name to which messages are published.

• Producers（发布者）

  Producers publish data to the topics of their choice. The producer is responsible for choosing which message to assign to which partition within the topic.

• Consumers（订阅者）

  Messaging traditionally has two models: queuing and publish-subscribe.

  In a queue, a pool of consumers may read from a server and each message goes to one of them;

  in publish-subscribe the message is broadcast to all consumers.

示意图

实用技巧

zkCli

Command	Comment
get /consumers/\ /owners	查看 topic实时消费的 group id
get /consumers/\ /offsets/\ /\	查看 offset情况 (ctime: 创建时间; mtime: 修改时间)

kafka-run-class

删除 topic

$ bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --zookeeper <zk host>:2181,<zk host>:2181,<zk host>:2181 --topic <topic>

更多资源，欢迎加入，一起交流学习

QQ group: ( 人工智能 1020982 (高级) & 1217710 (初级) | BigData 253793003 )