diff options
Diffstat (limited to 'content/posts/2020-03-22-simple-sse-based-pubsub-server.md')
| -rw-r--r-- | content/posts/2020-03-22-simple-sse-based-pubsub-server.md | 132 |
1 files changed, 68 insertions, 64 deletions
diff --git a/content/posts/2020-03-22-simple-sse-based-pubsub-server.md b/content/posts/2020-03-22-simple-sse-based-pubsub-server.md index 77c42a9..60745d0 100644 --- a/content/posts/2020-03-22-simple-sse-based-pubsub-server.md +++ b/content/posts/2020-03-22-simple-sse-based-pubsub-server.md | |||
| @@ -7,35 +7,35 @@ draft: false | |||
| 7 | 7 | ||
| 8 | ## Before we continue ... | 8 | ## Before we continue ... |
| 9 | 9 | ||
| 10 | Publisher Subscriber model is nothing new and there are many amazing solutions | 10 | Publisher Subscriber model is nothing new and there are many amazing solutions |
| 11 | out there, so writing a new one would be a waste of time if other solutions | 11 | out there, so writing a new one would be a waste of time if other solutions |
| 12 | wouldn't have quite complex install procedures and weren't so hard to maintain. | 12 | wouldn't have quite complex install procedures and weren't so hard to maintain. |
| 13 | But to be fair, comparing this simple server with something like | 13 | But to be fair, comparing this simple server with something like |
| 14 | [Kafka](https://kafka.apache.org/) or [RabbitMQ](https://www.rabbitmq.com/) is | 14 | [Kafka](https://kafka.apache.org/) or [RabbitMQ](https://www.rabbitmq.com/) is |
| 15 | laughable at the least. Those solutions are enterprise grade and have many | 15 | laughable at the least. Those solutions are enterprise grade and have many |
| 16 | mechanisms there to ensure messages aren't lost and much more. Regardless of | 16 | mechanisms there to ensure messages aren't lost and much more. Regardless of |
| 17 | these drawbacks, this method has been tested on a large website and worked | 17 | these drawbacks, this method has been tested on a large website and worked until |
| 18 | until now without any problems. So now, that we got that cleared up, let's | 18 | now without any problems. So now, that we got that cleared up, let's continue. |
| 19 | continue. | 19 | |
| 20 | 20 | ***Wiki definition:** Publish/subscribe messaging, or pub/sub messaging, is a | |
| 21 | ***Wiki definition:** Publish/subscribe messaging, or pub/sub messaging, is a | 21 | form of asynchronous service-to-service communication used in serverless and |
| 22 | form of asynchronous service-to-service communication used in serverless and | 22 | microservices architectures. In a pub/sub model, any message published to a |
| 23 | microservices architectures. In a pub/sub model, any message published to a | ||
| 24 | topic is immediately received by all the subscribers to the topic.* | 23 | topic is immediately received by all the subscribers to the topic.* |
| 25 | 24 | ||
| 26 | ## General goals | 25 | ## General goals |
| 27 | 26 | ||
| 28 | - provide a simple server that relays messages to all the connected clients, | 27 | - provide a simple server that relays messages to all the connected clients, |
| 29 | - messages can be posted on specific topics, | 28 | - messages can be posted on specific topics, |
| 30 | - messages get sent via [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) | 29 | - messages get sent via [Server-Sent |
| 30 | Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) | ||
| 31 | to all the subscribers. | 31 | to all the subscribers. |
| 32 | 32 | ||
| 33 | ## How exactly does the pub/sub model work? | 33 | ## How exactly does the pub/sub model work? |
| 34 | 34 | ||
| 35 | The easiest way to explain this is with diagram bellow. Basic function is | 35 | The easiest way to explain this is with diagram bellow. Basic function is |
| 36 | simple. We have subscribers that receive messages, and we have publishers that | 36 | simple. We have subscribers that receive messages, and we have publishers that |
| 37 | create and post messages. Similar model is also well know pattern that works | 37 | create and post messages. Similar model is also well know pattern that works on |
| 38 | on a premise of consumers and producers, and they take similar roles. | 38 | a premise of consumers and producers, and they take similar roles. |
| 39 | 39 | ||
| 40 |  | 40 |  |
| 41 | 41 | ||
| @@ -45,43 +45,47 @@ on a premise of consumers and producers, and they take similar roles. | |||
| 45 | - consumer is receiving messages from subscribed topic, | 45 | - consumer is receiving messages from subscribed topic, |
| 46 | - servers is also known as Broker, | 46 | - servers is also known as Broker, |
| 47 | - broker does not store messages or tracks success, | 47 | - broker does not store messages or tracks success, |
| 48 | - broker uses [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) | 48 | - broker uses |
| 49 | method for delivering messages, | 49 | [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) method |
| 50 | - if consumer wants to receive messages from a topic, producer and consumer topics | 50 | for delivering messages, |
| 51 | must match, | 51 | - if consumer wants to receive messages from a topic, producer and consumer |
| 52 | topics must match, | ||
| 52 | - consumer can subscribe to multiple topics, | 53 | - consumer can subscribe to multiple topics, |
| 53 | - producer can publish to multiple topics, | 54 | - producer can publish to multiple topics, |
| 54 | - each message has a messageId. | 55 | - each message has a messageId. |
| 55 | 56 | ||
| 56 | **Known drawbacks:** | 57 | **Known drawbacks:** |
| 57 | 58 | ||
| 58 | - messages will not be stored in a persistent queue or unreceived messages like | 59 | - messages will not be stored in a persistent queue or unreceived messages like |
| 59 | [DeadLetterQueue](https://en.wikipedia.org/wiki/Dead_letter_queue) so old | 60 | [DeadLetterQueue](https://en.wikipedia.org/wiki/Dead_letter_queue) so old |
| 60 | messages could be lost on server restart, | 61 | messages could be lost on server restart, |
| 61 | - [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) | 62 | - [Server-Sent |
| 62 | opens a long-running connection between the client and the server so make | 63 | Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) |
| 63 | sure if your setup is load balanced that the load balancer in this case can | 64 | opens a long-running connection between the client and the server so make sure |
| 64 | have long opened connection, | 65 | if your setup is load balanced that the load balancer in this case can have |
| 66 | long opened connection, | ||
| 65 | - no system moderation due to the dynamic nature of creating queues. | 67 | - no system moderation due to the dynamic nature of creating queues. |
| 66 | 68 | ||
| 67 | ## Server-Sent Events | 69 | ## Server-Sent Events |
| 68 | 70 | ||
| 69 | Read more about it on [official specification page](https://html.spec.whatwg.org/multipage/server-sent-events.html). | 71 | Read more about it on [official specification |
| 72 | page](https://html.spec.whatwg.org/multipage/server-sent-events.html). | ||
| 70 | 73 | ||
| 71 | ### Current browser support | 74 | ### Current browser support |
| 72 | 75 | ||
| 73 |  | 76 |  |
| 74 | 77 | ||
| 75 | Check [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) | 78 | Check |
| 79 | [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) | ||
| 76 | for latest information about browser support. | 80 | for latest information about browser support. |
| 77 | 81 | ||
| 78 | ### Known issues | 82 | ### Known issues |
| 79 | 83 | ||
| 80 | - Firefox 52 and below do not support EventSource in web/shared workers | 84 | - Firefox 52 and below do not support EventSource in web/shared workers |
| 81 | - In Firefox prior to version 36 server-sent events do not reconnect | 85 | - In Firefox prior to version 36 server-sent events do not reconnect |
| 82 | automatically in case of a connection interrupt (bug) | 86 | automatically in case of a connection interrupt (bug) |
| 83 | - Reportedly, CORS in EventSource is currently supported in Firefox 10+, | 87 | - Reportedly, CORS in EventSource is currently supported in Firefox 10+, Opera |
| 84 | Opera 12+, Chrome 26+, Safari 7.0+. | 88 | 12+, Chrome 26+, Safari 7.0+. |
| 85 | - Antivirus software may block the event streaming data chunks. | 89 | - Antivirus software may block the event streaming data chunks. |
| 86 | 90 | ||
| 87 | Source: [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) | 91 | Source: [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) |
| @@ -104,7 +108,7 @@ data: this is line two | |||
| 104 | <blank line> | 108 | <blank line> |
| 105 | ``` | 109 | ``` |
| 106 | 110 | ||
| 107 | And you can specify your own event types (the above messages will all trigger | 111 | And you can specify your own event types (the above messages will all trigger |
| 108 | the message event): | 112 | the message event): |
| 109 | 113 | ||
| 110 | ```bash | 114 | ```bash |
| @@ -116,7 +120,7 @@ data: 103.34 | |||
| 116 | 120 | ||
| 117 | ### Server requirements | 121 | ### Server requirements |
| 118 | 122 | ||
| 119 | The important thing is how you send headers and which headers are sent by the | 123 | The important thing is how you send headers and which headers are sent by the |
| 120 | server that triggers browser to threat response as a EventStream. | 124 | server that triggers browser to threat response as a EventStream. |
| 121 | 125 | ||
| 122 | Headers responsible for this are: | 126 | Headers responsible for this are: |
| @@ -129,23 +133,23 @@ Connection: keep-alive | |||
| 129 | 133 | ||
| 130 | ### Debugging with Google Chrome | 134 | ### Debugging with Google Chrome |
| 131 | 135 | ||
| 132 | Google Chrome provides build-in debugging and exploration tool for | 136 | Google Chrome provides build-in debugging and exploration tool for [Server-Sent |
| 133 | [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) | 137 | Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) |
| 134 | which is quite nice and available from Developer Tools under Network tab. | 138 | which is quite nice and available from Developer Tools under Network tab. |
| 135 | 139 | ||
| 136 | > You can debug only client side events that get received and not the server | 140 | > You can debug only client side events that get received and not the server |
| 137 | > ones. For debugging server events add `console.log` to `server.js` code and | 141 | > ones. For debugging server events add `console.log` to `server.js` code and |
| 138 | > print out events. | 142 | > print out events. |
| 139 | 143 | ||
| 140 |  | 144 |  |
| 141 | 145 | ||
| 142 | ## Server implementation | 146 | ## Server implementation |
| 143 | 147 | ||
| 144 | For the sake of this example we will use [Node.js](https://nodejs.org/en/) | 148 | For the sake of this example we will use [Node.js](https://nodejs.org/en/) with |
| 145 | with [Express](https://expressjs.com) as our router since this is the easiest | 149 | [Express](https://expressjs.com) as our router since this is the easiest way to |
| 146 | way to get started and we will use already written SSE library for node | 150 | get started and we will use already written SSE library for node |
| 147 | [sse-pubsub](https://www.npmjs.com/package/sse-pubsub) so we don't reinvent | 151 | [sse-pubsub](https://www.npmjs.com/package/sse-pubsub) so we don't reinvent the |
| 148 | the wheel. | 152 | wheel. |
| 149 | 153 | ||
| 150 | ```bash | 154 | ```bash |
| 151 | npm init --yes | 155 | npm init --yes |
| @@ -252,16 +256,16 @@ app.listen(port, () => { | |||
| 252 | 256 | ||
| 253 | ### Our custom message format | 257 | ### Our custom message format |
| 254 | 258 | ||
| 255 | Each message posted on a server must be in a specific format that out server | 259 | Each message posted on a server must be in a specific format that out server |
| 256 | accepts. Having structure like this allows us to have multiple separated type | 260 | accepts. Having structure like this allows us to have multiple separated type of |
| 257 | of events on each topic. | 261 | events on each topic. |
| 258 | 262 | ||
| 259 | With this we can separate streams and only receive events that belong to the | 263 | With this we can separate streams and only receive events that belong to the |
| 260 | topic. | 264 | topic. |
| 261 | 265 | ||
| 262 | One example would be, that we have index page and we want to receive messages | 266 | One example would be, that we have index page and we want to receive messages |
| 263 | about new upvotes or new subscribers but we don't want to follow events for | 267 | about new upvotes or new subscribers but we don't want to follow events for |
| 264 | other pages. This reduces clutter and overall network. And structure is much | 268 | other pages. This reduces clutter and overall network. And structure is much |
| 265 | nicer and maintanable. | 269 | nicer and maintanable. |
| 266 | 270 | ||
| 267 | ```json | 271 | ```json |
| @@ -278,15 +282,15 @@ nicer and maintanable. | |||
| 278 | 282 | ||
| 279 | <video src="/assets/simple-pubsub-server/clients.m4v" controls></video> | 283 | <video src="/assets/simple-pubsub-server/clients.m4v" controls></video> |
| 280 | 284 | ||
| 281 | You can download [the code](../simple-pubsub-server/sse-pubsub-server.zip) | 285 | You can download [the code](../simple-pubsub-server/sse-pubsub-server.zip) and |
| 282 | and follow along. | 286 | follow along. |
| 283 | 287 | ||
| 284 | ### Publisher | 288 | ### Publisher |
| 285 | 289 | ||
| 286 | As talked about above publisher is the one that send messages to the | 290 | As talked about above publisher is the one that send messages to the |
| 287 | broker/server. Message inside the payload can be whatever you want (string, | 291 | broker/server. Message inside the payload can be whatever you want (string, |
| 288 | object, array). I would however personally avoid send large chunks of data | 292 | object, array). I would however personally avoid send large chunks of data like |
| 289 | like blobs and such. | 293 | blobs and such. |
| 290 | 294 | ||
| 291 | ```html | 295 | ```html |
| 292 | <!DOCTYPE html> | 296 | <!DOCTYPE html> |
| @@ -359,19 +363,19 @@ like blobs and such. | |||
| 359 | 363 | ||
| 360 | ### Subscriber | 364 | ### Subscriber |
| 361 | 365 | ||
| 362 | Subscriber is responsible for receiving new messages that come from server via | 366 | Subscriber is responsible for receiving new messages that come from server via |
| 363 | publisher. The code bellow is very rudimentary but works and follows the | 367 | publisher. The code bellow is very rudimentary but works and follows the |
| 364 | implementation guidelines for EventSource. | 368 | implementation guidelines for EventSource. |
| 365 | 369 | ||
| 366 | You can use either Developer Tools Console to see incoming messages or you can | 370 | You can use either Developer Tools Console to see incoming messages or you can |
| 367 | defer to Debugging with Google Chrome section above to see all EventStream | 371 | defer to Debugging with Google Chrome section above to see all EventStream |
| 368 | messages. | 372 | messages. |
| 369 | 373 | ||
| 370 | > Don't be alarmed if the subscriber gets disconnected from the server every | 374 | > Don't be alarmed if the subscriber gets disconnected from the server every so |
| 371 | > so often. The code we have here resets connection every 15s but it | 375 | > often. The code we have here resets connection every 15s but it automatically |
| 372 | > automatically get reconnected and fetches all messages up to last received | 376 | > get reconnected and fetches all messages up to last received message id. This |
| 373 | > message id. This setting can be adjusted in `server.js` file; search for | 377 | > setting can be adjusted in `server.js` file; search for the |
| 374 | > the `maxStreamDuration` variable. | 378 | > `maxStreamDuration` variable. |
| 375 | 379 | ||
| 376 | ```html | 380 | ```html |
| 377 | <!DOCTYPE html> | 381 | <!DOCTYPE html> |
