aboutsummaryrefslogtreecommitdiff
path: root/content/posts/2020-03-22-simple-sse-based-pubsub-server.md
diff options
context:
space:
mode:
Diffstat (limited to 'content/posts/2020-03-22-simple-sse-based-pubsub-server.md')
-rw-r--r--content/posts/2020-03-22-simple-sse-based-pubsub-server.md109
1 files changed, 81 insertions, 28 deletions
diff --git a/content/posts/2020-03-22-simple-sse-based-pubsub-server.md b/content/posts/2020-03-22-simple-sse-based-pubsub-server.md
index 939a62d..77c42a9 100644
--- a/content/posts/2020-03-22-simple-sse-based-pubsub-server.md
+++ b/content/posts/2020-03-22-simple-sse-based-pubsub-server.md
@@ -1,25 +1,41 @@
1--- 1---
2title: Simple Server-Sent Events based PubSub Server 2title: Simple Server-Sent Events based PubSub Server
3url: simple-server-sent-events-based-pubsub-server.html 3url: simple-server-sent-events-based-pubsub-server.html
4date: 2020-03-22 4date: 2020-03-22T12:00:00+02:00
5draft: false 5draft: false
6--- 6---
7 7
8## Before we continue ... 8## Before we continue ...
9 9
10Publisher Subscriber model is nothing new and there are many amazing solutions out there, so writing a new one would be a waste of time if other solutions wouldn't have quite complex install procedures and weren't so hard to maintain. But to be fair, comparing this simple server with something like [Kafka](https://kafka.apache.org/) or [RabbitMQ](https://www.rabbitmq.com/) is laughable at the least. Those solutions are enterprise grade and have many mechanisms there to ensure messages aren't lost and much more. Regardless of these drawbacks, this method has been tested on a large website and worked until now without any problems. So now, that we got that cleared up, let's continue. 10Publisher Subscriber model is nothing new and there are many amazing solutions
11 11out there, so writing a new one would be a waste of time if other solutions
12***Wiki definition:** Publish/subscribe messaging, or pub/sub messaging, is a form of asynchronous service-to-service communication used in serverless and microservices architectures. In a pub/sub model, any message published to a topic is immediately received by all the subscribers to the topic.* 12wouldn't have quite complex install procedures and weren't so hard to maintain.
13But to be fair, comparing this simple server with something like
14[Kafka](https://kafka.apache.org/) or [RabbitMQ](https://www.rabbitmq.com/) is
15laughable at the least. Those solutions are enterprise grade and have many
16mechanisms there to ensure messages aren't lost and much more. Regardless of
17these drawbacks, this method has been tested on a large website and worked
18until now without any problems. So now, that we got that cleared up, let's
19continue.
20
21***Wiki definition:** Publish/subscribe messaging, or pub/sub messaging, is a
22form of asynchronous service-to-service communication used in serverless and
23microservices architectures. In a pub/sub model, any message published to a
24topic is immediately received by all the subscribers to the topic.*
13 25
14## General goals 26## General goals
15 27
16- provide a simple server that relays messages to all the connected clients, 28- provide a simple server that relays messages to all the connected clients,
17- messages can be posted on specific topics, 29- messages can be posted on specific topics,
18- messages get sent via [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) to all the subscribers. 30- messages get sent via [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events)
31 to all the subscribers.
19 32
20## How exactly does the pub/sub model work? 33## How exactly does the pub/sub model work?
21 34
22The easiest way to explain this is with diagram bellow. Basic function is simple. We have subscribers that receive messages, and we have publishers that create and post messages. Similar model is also well know pattern that works on a premise of consumers and producers, and they take similar roles. 35The easiest way to explain this is with diagram bellow. Basic function is
36simple. We have subscribers that receive messages, and we have publishers that
37create and post messages. Similar model is also well know pattern that works
38on a premise of consumers and producers, and they take similar roles.
23 39
24![How PubSub works](/assets/simple-pubsub-server/pubsub-overview.png) 40![How PubSub works](/assets/simple-pubsub-server/pubsub-overview.png)
25 41
@@ -29,16 +45,23 @@ The easiest way to explain this is with diagram bellow. Basic function is simple
29- consumer is receiving messages from subscribed topic, 45- consumer is receiving messages from subscribed topic,
30- servers is also known as Broker, 46- servers is also known as Broker,
31- broker does not store messages or tracks success, 47- broker does not store messages or tracks success,
32- broker uses [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) method for delivering messages, 48- broker uses [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))
33- if consumer wants to receive messages from a topic, producer and consumer topics must match, 49 method for delivering messages,
50- if consumer wants to receive messages from a topic, producer and consumer topics
51 must match,
34- consumer can subscribe to multiple topics, 52- consumer can subscribe to multiple topics,
35- producer can publish to multiple topics, 53- producer can publish to multiple topics,
36- each message has a messageId. 54- each message has a messageId.
37 55
38**Known drawbacks:** 56**Known drawbacks:**
39 57
40- messages will not be stored in a persistent queue or unreceived messages like [DeadLetterQueue](https://en.wikipedia.org/wiki/Dead_letter_queue) so old messages could be lost on server restart, 58- messages will not be stored in a persistent queue or unreceived messages like
41- [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) opens a long-running connection between the client and the server so make sure if your setup is load balanced that the load balancer in this case can have long opened connection, 59 [DeadLetterQueue](https://en.wikipedia.org/wiki/Dead_letter_queue) so old
60 messages could be lost on server restart,
61- [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events)
62 opens a long-running connection between the client and the server so make
63 sure if your setup is load balanced that the load balancer in this case can
64 have long opened connection,
42- no system moderation due to the dynamic nature of creating queues. 65- no system moderation due to the dynamic nature of creating queues.
43 66
44## Server-Sent Events 67## Server-Sent Events
@@ -49,13 +72,16 @@ Read more about it on [official specification page](https://html.spec.whatwg.org
49 72
50![Browser support](/assets/simple-pubsub-server/caniuse.png) 73![Browser support](/assets/simple-pubsub-server/caniuse.png)
51 74
52Check [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) for latest information about browser support. 75Check [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource)
76for latest information about browser support.
53 77
54### Known issues 78### Known issues
55 79
56- Firefox 52 and below do not support EventSource in web/shared workers 80- Firefox 52 and below do not support EventSource in web/shared workers
57- In Firefox prior to version 36 server-sent events do not reconnect automatically in case of a connection interrupt (bug) 81- In Firefox prior to version 36 server-sent events do not reconnect
58- Reportedly, CORS in EventSource is currently supported in Firefox 10+, Opera 12+, Chrome 26+, Safari 7.0+. 82 automatically in case of a connection interrupt (bug)
83- Reportedly, CORS in EventSource is currently supported in Firefox 10+,
84 Opera 12+, Chrome 26+, Safari 7.0+.
59- Antivirus software may block the event streaming data chunks. 85- Antivirus software may block the event streaming data chunks.
60 86
61Source: [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) 87Source: [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource)
@@ -78,7 +104,8 @@ data: this is line two
78<blank line> 104<blank line>
79``` 105```
80 106
81And you can specify your own event types (the above messages will all trigger the message event): 107And you can specify your own event types (the above messages will all trigger
108the message event):
82 109
83```bash 110```bash
84id: 36 111id: 36
@@ -89,7 +116,8 @@ data: 103.34
89 116
90### Server requirements 117### Server requirements
91 118
92The important thing is how you send headers and which headers are sent by the server that triggers browser to threat response as a EventStream. 119The important thing is how you send headers and which headers are sent by the
120server that triggers browser to threat response as a EventStream.
93 121
94Headers responsible for this are: 122Headers responsible for this are:
95 123
@@ -101,15 +129,23 @@ Connection: keep-alive
101 129
102### Debugging with Google Chrome 130### Debugging with Google Chrome
103 131
104Google Chrome provides build-in debugging and exploration tool for [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) which is quite nice and available from Developer Tools under Network tab. 132Google Chrome provides build-in debugging and exploration tool for
133[Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events)
134which is quite nice and available from Developer Tools under Network tab.
105 135
106> You can debug only client side events that get received and not the server ones. For debugging server events add `console.log` to `server.js` code and print out events. 136> You can debug only client side events that get received and not the server
137> ones. For debugging server events add `console.log` to `server.js` code and
138> print out events.
107 139
108![Google Chrome Developer Tools EventStream](/assets/simple-pubsub-server/chrome-debugging.png) 140![Google Chrome Developer Tools EventStream](/assets/simple-pubsub-server/chrome-debugging.png)
109 141
110## Server implementation 142## Server implementation
111 143
112For the sake of this example we will use [Node.js](https://nodejs.org/en/) with [Express](https://expressjs.com) as our router since this is the easiest way to get started and we will use already written SSE library for node [sse-pubsub](https://www.npmjs.com/package/sse-pubsub) so we don't reinvent the wheel. 144For the sake of this example we will use [Node.js](https://nodejs.org/en/)
145with [Express](https://expressjs.com) as our router since this is the easiest
146way to get started and we will use already written SSE library for node
147[sse-pubsub](https://www.npmjs.com/package/sse-pubsub) so we don't reinvent
148the wheel.
113 149
114```bash 150```bash
115npm init --yes 151npm init --yes
@@ -216,11 +252,17 @@ app.listen(port, () => {
216 252
217### Our custom message format 253### Our custom message format
218 254
219Each message posted on a server must be in a specific format that out server accepts. Having structure like this allows us to have multiple separated type of events on each topic. 255Each message posted on a server must be in a specific format that out server
256accepts. Having structure like this allows us to have multiple separated type
257of events on each topic.
220 258
221With this we can separate streams and only receive events that belong to the topic. 259With this we can separate streams and only receive events that belong to the
260topic.
222 261
223One example would be, that we have index page and we want to receive messages about new upvotes or new subscribers but we don't want to follow events for other pages. This reduces clutter and overall network. And structure is much nicer and maintanable. 262One example would be, that we have index page and we want to receive messages
263about new upvotes or new subscribers but we don't want to follow events for
264other pages. This reduces clutter and overall network. And structure is much
265nicer and maintanable.
224 266
225```json 267```json
226{ 268{
@@ -236,11 +278,15 @@ One example would be, that we have index page and we want to receive messages ab
236 278
237<video src="/assets/simple-pubsub-server/clients.m4v" controls></video> 279<video src="/assets/simple-pubsub-server/clients.m4v" controls></video>
238 280
239You can download [the code](../simple-pubsub-server/sse-pubsub-server.zip) and follow along. 281You can download [the code](../simple-pubsub-server/sse-pubsub-server.zip)
282and follow along.
240 283
241### Publisher 284### Publisher
242 285
243As talked about above publisher is the one that send messages to the broker/server. Message inside the payload can be whatever you want (string, object, array). I would however personally avoid send large chunks of data like blobs and such. 286As talked about above publisher is the one that send messages to the
287broker/server. Message inside the payload can be whatever you want (string,
288object, array). I would however personally avoid send large chunks of data
289like blobs and such.
244 290
245```html 291```html
246<!DOCTYPE html> 292<!DOCTYPE html>
@@ -309,16 +355,23 @@ As talked about above publisher is the one that send messages to the broker/serv
309 </body> 355 </body>
310 356
311</html> 357</html>
312
313``` 358```
314 359
315### Subscriber 360### Subscriber
316 361
317Subscriber is responsible for receiving new messages that come from server via publisher. The code bellow is very rudimentary but works and follows the implementation guidelines for EventSource. 362Subscriber is responsible for receiving new messages that come from server via
363publisher. The code bellow is very rudimentary but works and follows the
364implementation guidelines for EventSource.
318 365
319You can use either Developer Tools Console to see incoming messages or you can defer to Debugging with Google Chrome section above to see all EventStream messages. 366You can use either Developer Tools Console to see incoming messages or you can
367defer to Debugging with Google Chrome section above to see all EventStream
368messages.
320 369
321> Don't be alarmed if the subscriber gets disconnected from the server every so often. The code we have here resets connection every 15s but it automatically get reconnected and fetches all messages up to last received message id. This setting can be adjusted in `server.js` file; search for the `maxStreamDuration` variable. 370> Don't be alarmed if the subscriber gets disconnected from the server every
371> so often. The code we have here resets connection every 15s but it
372> automatically get reconnected and fetches all messages up to last received
373> message id. This setting can be adjusted in `server.js` file; search for
374> the `maxStreamDuration` variable.
322 375
323```html 376```html
324<!DOCTYPE html> 377<!DOCTYPE html>
@@ -383,7 +436,6 @@ You can use either Developer Tools Console to see incoming messages or you can d
383 </body> 436 </body>
384 437
385</html> 438</html>
386
387``` 439```
388 440
389## Reading further 441## Reading further
@@ -394,3 +446,4 @@ You can use either Developer Tools Console to see incoming messages or you can d
394- [An HTTP/2 extension for bidirectional messaging communication](https://tools.ietf.org/id/draft-xie-bidirectional-messaging-01.html) 446- [An HTTP/2 extension for bidirectional messaging communication](https://tools.ietf.org/id/draft-xie-bidirectional-messaging-01.html)
395- [Introduction to HTTP/2](https://developers.google.com/web/fundamentals/performance/http2) 447- [Introduction to HTTP/2](https://developers.google.com/web/fundamentals/performance/http2)
396- [The WebSocket API (WebSockets)](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) 448- [The WebSocket API (WebSockets)](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API)
449