Introducing Decentralized Pub/Sub Based on NKN

#1

by Yilun Zhang, CTO of NKN.org

What is Pub/Sub

One basic function of NKN client (e.g. https://github.com/nknorg/nkn-client-js) is to provide a decentralized message system , which includes unicast, multicast and anycast. That’s pretty much enough if the message sender knows who are the receivers. However, in lots of common scenarios, the receiver should be logically decoupled from the sender. For example, when I send a message to a chatroom, I don’t want to know exactly who is in the chatroom, I just want whoever in the chatroom to receive my message. That’s where pub/sub comes into place.

Basically, pub/sub (short for publish/subscribe) is a model that decouples message sender (publisher) and receiver (subscribers) . Publishers publish message to topic (we only consider topic based pub/sub here) without having to know who subscribe to the topic and will receive the message. Subscribers subscribe to topics and will receive messages sent to subscribed topics. Pub/sub is a basic building block for modern applications , and has been widely used from infrastructure level (e.g. load balancing) to application level (e.g. chat room/messenger).

The following diagram from Google Cloud ( https://cloud.google.com/pubsub/docs/overview) shows the relationship between publisher, subscriber and topic.

Challenges of Decentralized Pub/Sub

Cloud providers like Google Cloud, AWS provide cloud based pub/sub, but their centralized nature makes them hard (if not impossible) to be used in decentralized applications.

On the other hand, building decentralized pub/sub is also challenging, as most existing decentralized systems (e.g. Ethereum) are not well suited for real time messages — if sending a single message would cost over a cent in USD and almost a minute to deliver, not to mention the scalability issue, how could you expect it to be practically useful?

To be more general, there are a few challenges in building decentralized pub/sub based on existing decentralized system (most likely blockchain based):

  • Message needs to be delivered in real time
  • Message delivery needs to be affordable
  • Message throughput needs to be horizontal scalable

If you come from a non-blockchain background, you are probably laughing at how trivial the above “challenges” are. Well the reality is, if we rely on on-chain transaction to transmit information, it’s still very hard to solve the above problems.

One solution to these problems is to use off-chain message delivery . That’s why we believe NKN is perfect for being the infrastructure of a decentralized pub/sub system : message delivery in NKN is instant (millisecond level end to end latency), free , and horizontal scalable (more nodes → higher throughput) because it’s purely off-chain.

Building Decentralized Pub/Sub

To build a pub/sub system, we need to solve two basic problems: how to store and retrieve topic-subscriber info , and how to deliver messages . While NKN naturally solves the second one, we still need to decide where the subscriber information is stored.

After quite a lot of discussion, we decide to store the topic-subscriber information on-chain . As the result, subscribe needs to be done in a transaction, which will be reliable but not horizontal scalable. Luckily, subscribe should be a much less frequent action compared to publish (which will be off-chain and horizontal scalable), so it shouldn’t be the bottleneck.

After some work and tests, we are now happy to say that our pub/sub is working quite nicely . Because publishing is mainly sending off-chain messages, it is integrated into NKN client (e.g. https://github.com/nknorg/nkn-client-js). Subscribing, on the other hand, is integrated into NKN wallet (e.g. https://github.com/nknorg/nkn-wallet-js) as it needs to sign and send transactions. Both are integrated into NKN SDK (e.g. https://github.com/nknorg/nkn-sdk-go) which has both NKN client and NKN wallet.

Using Pub/Sub

Details of how to use pub/sub can be found in the documentation of various NKN client/wallet/SDK implementations. The API is quite straightforward in general. For example, in the JavaScript implementation, subscribing to a topic is as simple as

wallet.subscribe(topic, bucket, duration)

The reason that we have bucket concept is to avoid (accidental) message flooding in the case of massive subscribers, and can be hidden by higher level APIs like SubscribeToFirstAvailableBucket . Similarly, publishing to a topic is as simple as

client.publish(topic, bucket, message)

and bucket count of a topic can be get by APIs like GetTopicBucketsCount . Clients subscribed to the topic can listen for messages with

client.on(‘message’, (src, payload, payloadType) => {
});

Use Cases and Summary

Pub/sub has been widely used in many systems and applications. In additional to the existing use cases, I’d like to dive into a new class that is more suitable and unique to decentralized applications .

Most centralized applications are not open sourced, and protocol is often bound with the application only. Decentralized applications, on the other hand, are most likely open sourced, and protocol is decoupled from the implementation to allow cross implementation communication. This greatly reduces the friction to design and implement cross application protocols. If multiple applications want to share the same information flow, a decentralized, application neutral and language neutral pub/sub platform would be essential.

Here are some examples:

  • Different service providers want to share the same service discovery mechanism
  • Multiple applications want to share the same rating system
  • Applications would like to pass data to downstreaming applications sharing the protocol

Our decentralized pub/sub could be used to achieve these goals. To be more general, this represents one of the most interesting properties (in my opinion) of decentralized applications — the separation of application, protocol, and data . NKN’s technology is uniquely positioned and purposely built to capitalize upon this opportunity. We will soon release something built on our decentralized pub/sub framework, so please stay tuned!

2 Likes
:de: Vorstellung eines dezentralen Pub/Sub auf Basis von NKN
#2

What is Pub/sub

https://cloud.google.com/pubsub/docs/overview

Pub/Sub brings the flexibility and reliability of enterprise message-oriented middleware to the cloud. At the same time, Cloud Pub/Sub is a scalable, durable event ingestion and delivery system that serves as a foundation for modern stream analytics pipelines. By providing many-to-many, asynchronous messaging that decouples senders and receivers, it allows for secure and highly available communication among independently written applications. Cloud Pub/Sub delivers low-latency, durable messaging that helps developers quickly integrate systems hosted on the Google Cloud Platform and externally.

Core concepts

  • Topic: A named resource to which messages are sent by publishers.
  • Subscription: A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application. For more details about subscriptions and message delivery semantics, see the Subscriber Guide.
  • Message: The combination of data and (optional) attributes that a publisher sends to a topic and is eventually delivered to subscribers.
  • Message attribute: A key-value pair that a publisher can define for a message. For example, keyiana.org/language_tag and value en could be added to messages to mark them as readable by an English-speaking subscriber.

Advantages

https://en.wikipedia.org/wiki/Publish–subscribe_pattern#Advantages

Loose coupling

Publishers are loosely coupled to subscribers, and need not even know of their existence. With the topic being the focus, publishers and subscribers are allowed to remain ignorant of system topology. Each can continue to operate as per normal independently of the other. In the traditional tightly coupled client–server paradigm, the client cannot post messages to the server while the server process is not running, nor can the server receive messages unless the client is running. Many pub/sub systems decouple not only the locations of the publishers and subscribers but also decouple them temporally. A common strategy used by middleware analysts with such pub/sub systems is to take down a publisher to allow the subscriber to work through the backlog (a form of bandwidth throttling).

Scalability

Pub/sub provides the opportunity for better scalability than traditional client-server, through parallel operation, message caching, tree-based or network-based routing, etc. However, in certain types of tightly coupled, high-volume enterprise environments, as systems scale up to become data centers with thousands of servers sharing the pub/sub infrastructure, current vendor systems often lose this benefit; scalability for pub/sub products under high load in these contexts is a research challenge.

Outside of the enterprise environment, on the other hand, the pub/sub paradigm has proven its scalability to volumes far beyond those of a single data center, providing Internet-wide distributed messaging through web syndication protocols such as RSS and Atom. These syndication protocols accept higher latency and lack of delivery guarantees in exchange for the ability for even a low-end web server to syndicate messages to (potentially) millions of separate subscriber nodes.

1 Like