aboutsummaryrefslogtreecommitdiff
path: root/content/posts/2020-03-22-simple-sse-based-pubsub-server.md
diff options
context:
space:
mode:
Diffstat (limited to 'content/posts/2020-03-22-simple-sse-based-pubsub-server.md')
-rw-r--r--content/posts/2020-03-22-simple-sse-based-pubsub-server.md132
1 files changed, 68 insertions, 64 deletions
diff --git a/content/posts/2020-03-22-simple-sse-based-pubsub-server.md b/content/posts/2020-03-22-simple-sse-based-pubsub-server.md
index 77c42a9..60745d0 100644
--- a/content/posts/2020-03-22-simple-sse-based-pubsub-server.md
+++ b/content/posts/2020-03-22-simple-sse-based-pubsub-server.md
@@ -7,35 +7,35 @@ draft: false
7 7
8## Before we continue ... 8## Before we continue ...
9 9
10Publisher Subscriber model is nothing new and there are many amazing solutions 10Publisher Subscriber model is nothing new and there are many amazing solutions
11out there, so writing a new one would be a waste of time if other solutions 11out there, so writing a new one would be a waste of time if other solutions
12wouldn't have quite complex install procedures and weren't so hard to maintain. 12wouldn't have quite complex install procedures and weren't so hard to maintain.
13But to be fair, comparing this simple server with something like 13But to be fair, comparing this simple server with something like
14[Kafka](https://kafka.apache.org/) or [RabbitMQ](https://www.rabbitmq.com/) is 14[Kafka](https://kafka.apache.org/) or [RabbitMQ](https://www.rabbitmq.com/) is
15laughable at the least. Those solutions are enterprise grade and have many 15laughable at the least. Those solutions are enterprise grade and have many
16mechanisms there to ensure messages aren't lost and much more. Regardless of 16mechanisms there to ensure messages aren't lost and much more. Regardless of
17these drawbacks, this method has been tested on a large website and worked 17these drawbacks, this method has been tested on a large website and worked until
18until now without any problems. So now, that we got that cleared up, let's 18now without any problems. So now, that we got that cleared up, let's continue.
19continue. 19
20 20***Wiki definition:** Publish/subscribe messaging, or pub/sub messaging, is a
21***Wiki definition:** Publish/subscribe messaging, or pub/sub messaging, is a 21form of asynchronous service-to-service communication used in serverless and
22form of asynchronous service-to-service communication used in serverless and 22microservices architectures. In a pub/sub model, any message published to a
23microservices architectures. In a pub/sub model, any message published to a
24topic is immediately received by all the subscribers to the topic.* 23topic is immediately received by all the subscribers to the topic.*
25 24
26## General goals 25## General goals
27 26
28- provide a simple server that relays messages to all the connected clients, 27- provide a simple server that relays messages to all the connected clients,
29- messages can be posted on specific topics, 28- messages can be posted on specific topics,
30- messages get sent via [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) 29- messages get sent via [Server-Sent
30 Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events)
31 to all the subscribers. 31 to all the subscribers.
32 32
33## How exactly does the pub/sub model work? 33## How exactly does the pub/sub model work?
34 34
35The easiest way to explain this is with diagram bellow. Basic function is 35The easiest way to explain this is with diagram bellow. Basic function is
36simple. We have subscribers that receive messages, and we have publishers that 36simple. We have subscribers that receive messages, and we have publishers that
37create and post messages. Similar model is also well know pattern that works 37create and post messages. Similar model is also well know pattern that works on
38on a premise of consumers and producers, and they take similar roles. 38a premise of consumers and producers, and they take similar roles.
39 39
40![How PubSub works](/assets/simple-pubsub-server/pubsub-overview.png) 40![How PubSub works](/assets/simple-pubsub-server/pubsub-overview.png)
41 41
@@ -45,43 +45,47 @@ on a premise of consumers and producers, and they take similar roles.
45- consumer is receiving messages from subscribed topic, 45- consumer is receiving messages from subscribed topic,
46- servers is also known as Broker, 46- servers is also known as Broker,
47- broker does not store messages or tracks success, 47- broker does not store messages or tracks success,
48- broker uses [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) 48- broker uses
49 method for delivering messages, 49 [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) method
50- if consumer wants to receive messages from a topic, producer and consumer topics 50 for delivering messages,
51 must match, 51- if consumer wants to receive messages from a topic, producer and consumer
52 topics must match,
52- consumer can subscribe to multiple topics, 53- consumer can subscribe to multiple topics,
53- producer can publish to multiple topics, 54- producer can publish to multiple topics,
54- each message has a messageId. 55- each message has a messageId.
55 56
56**Known drawbacks:** 57**Known drawbacks:**
57 58
58- messages will not be stored in a persistent queue or unreceived messages like 59- messages will not be stored in a persistent queue or unreceived messages like
59 [DeadLetterQueue](https://en.wikipedia.org/wiki/Dead_letter_queue) so old 60 [DeadLetterQueue](https://en.wikipedia.org/wiki/Dead_letter_queue) so old
60 messages could be lost on server restart, 61 messages could be lost on server restart,
61- [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) 62- [Server-Sent
62 opens a long-running connection between the client and the server so make 63 Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events)
63 sure if your setup is load balanced that the load balancer in this case can 64 opens a long-running connection between the client and the server so make sure
64 have long opened connection, 65 if your setup is load balanced that the load balancer in this case can have
66 long opened connection,
65- no system moderation due to the dynamic nature of creating queues. 67- no system moderation due to the dynamic nature of creating queues.
66 68
67## Server-Sent Events 69## Server-Sent Events
68 70
69Read more about it on [official specification page](https://html.spec.whatwg.org/multipage/server-sent-events.html). 71Read more about it on [official specification
72page](https://html.spec.whatwg.org/multipage/server-sent-events.html).
70 73
71### Current browser support 74### Current browser support
72 75
73![Browser support](/assets/simple-pubsub-server/caniuse.png) 76![Browser support](/assets/simple-pubsub-server/caniuse.png)
74 77
75Check [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) 78Check
79[https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource)
76for latest information about browser support. 80for latest information about browser support.
77 81
78### Known issues 82### Known issues
79 83
80- Firefox 52 and below do not support EventSource in web/shared workers 84- Firefox 52 and below do not support EventSource in web/shared workers
81- In Firefox prior to version 36 server-sent events do not reconnect 85- In Firefox prior to version 36 server-sent events do not reconnect
82 automatically in case of a connection interrupt (bug) 86 automatically in case of a connection interrupt (bug)
83- Reportedly, CORS in EventSource is currently supported in Firefox 10+, 87- Reportedly, CORS in EventSource is currently supported in Firefox 10+, Opera
84 Opera 12+, Chrome 26+, Safari 7.0+. 88 12+, Chrome 26+, Safari 7.0+.
85- Antivirus software may block the event streaming data chunks. 89- Antivirus software may block the event streaming data chunks.
86 90
87Source: [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) 91Source: [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource)
@@ -104,7 +108,7 @@ data: this is line two
104<blank line> 108<blank line>
105``` 109```
106 110
107And you can specify your own event types (the above messages will all trigger 111And you can specify your own event types (the above messages will all trigger
108the message event): 112the message event):
109 113
110```bash 114```bash
@@ -116,7 +120,7 @@ data: 103.34
116 120
117### Server requirements 121### Server requirements
118 122
119The important thing is how you send headers and which headers are sent by the 123The important thing is how you send headers and which headers are sent by the
120server that triggers browser to threat response as a EventStream. 124server that triggers browser to threat response as a EventStream.
121 125
122Headers responsible for this are: 126Headers responsible for this are:
@@ -129,23 +133,23 @@ Connection: keep-alive
129 133
130### Debugging with Google Chrome 134### Debugging with Google Chrome
131 135
132Google Chrome provides build-in debugging and exploration tool for 136Google Chrome provides build-in debugging and exploration tool for [Server-Sent
133[Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) 137Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events)
134which is quite nice and available from Developer Tools under Network tab. 138which is quite nice and available from Developer Tools under Network tab.
135 139
136> You can debug only client side events that get received and not the server 140> You can debug only client side events that get received and not the server
137> ones. For debugging server events add `console.log` to `server.js` code and 141> ones. For debugging server events add `console.log` to `server.js` code and
138> print out events. 142> print out events.
139 143
140![Google Chrome Developer Tools EventStream](/assets/simple-pubsub-server/chrome-debugging.png) 144![Google Chrome Developer Tools EventStream](/assets/simple-pubsub-server/chrome-debugging.png)
141 145
142## Server implementation 146## Server implementation
143 147
144For the sake of this example we will use [Node.js](https://nodejs.org/en/) 148For the sake of this example we will use [Node.js](https://nodejs.org/en/) with
145with [Express](https://expressjs.com) as our router since this is the easiest 149[Express](https://expressjs.com) as our router since this is the easiest way to
146way to get started and we will use already written SSE library for node 150get started and we will use already written SSE library for node
147[sse-pubsub](https://www.npmjs.com/package/sse-pubsub) so we don't reinvent 151[sse-pubsub](https://www.npmjs.com/package/sse-pubsub) so we don't reinvent the
148the wheel. 152wheel.
149 153
150```bash 154```bash
151npm init --yes 155npm init --yes
@@ -252,16 +256,16 @@ app.listen(port, () => {
252 256
253### Our custom message format 257### Our custom message format
254 258
255Each message posted on a server must be in a specific format that out server 259Each message posted on a server must be in a specific format that out server
256accepts. Having structure like this allows us to have multiple separated type 260accepts. Having structure like this allows us to have multiple separated type of
257of events on each topic. 261events on each topic.
258 262
259With this we can separate streams and only receive events that belong to the 263With this we can separate streams and only receive events that belong to the
260topic. 264topic.
261 265
262One example would be, that we have index page and we want to receive messages 266One example would be, that we have index page and we want to receive messages
263about new upvotes or new subscribers but we don't want to follow events for 267about new upvotes or new subscribers but we don't want to follow events for
264other pages. This reduces clutter and overall network. And structure is much 268other pages. This reduces clutter and overall network. And structure is much
265nicer and maintanable. 269nicer and maintanable.
266 270
267```json 271```json
@@ -278,15 +282,15 @@ nicer and maintanable.
278 282
279<video src="/assets/simple-pubsub-server/clients.m4v" controls></video> 283<video src="/assets/simple-pubsub-server/clients.m4v" controls></video>
280 284
281You can download [the code](../simple-pubsub-server/sse-pubsub-server.zip) 285You can download [the code](../simple-pubsub-server/sse-pubsub-server.zip) and
282and follow along. 286follow along.
283 287
284### Publisher 288### Publisher
285 289
286As talked about above publisher is the one that send messages to the 290As talked about above publisher is the one that send messages to the
287broker/server. Message inside the payload can be whatever you want (string, 291broker/server. Message inside the payload can be whatever you want (string,
288object, array). I would however personally avoid send large chunks of data 292object, array). I would however personally avoid send large chunks of data like
289like blobs and such. 293blobs and such.
290 294
291```html 295```html
292<!DOCTYPE html> 296<!DOCTYPE html>
@@ -359,19 +363,19 @@ like blobs and such.
359 363
360### Subscriber 364### Subscriber
361 365
362Subscriber is responsible for receiving new messages that come from server via 366Subscriber is responsible for receiving new messages that come from server via
363publisher. The code bellow is very rudimentary but works and follows the 367publisher. The code bellow is very rudimentary but works and follows the
364implementation guidelines for EventSource. 368implementation guidelines for EventSource.
365 369
366You can use either Developer Tools Console to see incoming messages or you can 370You can use either Developer Tools Console to see incoming messages or you can
367defer to Debugging with Google Chrome section above to see all EventStream 371defer to Debugging with Google Chrome section above to see all EventStream
368messages. 372messages.
369 373
370> Don't be alarmed if the subscriber gets disconnected from the server every 374> Don't be alarmed if the subscriber gets disconnected from the server every so
371> so often. The code we have here resets connection every 15s but it 375> often. The code we have here resets connection every 15s but it automatically
372> automatically get reconnected and fetches all messages up to last received 376> get reconnected and fetches all messages up to last received message id. This
373> message id. This setting can be adjusted in `server.js` file; search for 377> setting can be adjusted in `server.js` file; search for the
374> the `maxStreamDuration` variable. 378> `maxStreamDuration` variable.
375 379
376```html 380```html
377<!DOCTYPE html> 381<!DOCTYPE html>