From 43b0708769eb61392050045b881f8e6ba39c5b66 Mon Sep 17 00:00:00 2001 From: Mitja Felicijan Date: Fri, 26 May 2023 00:40:40 +0200 Subject: Massive update to posts, archetypes Added a archetypes for creating notes and posts so it auto-populates fields. Fixed existing posts so they align with the rule of 80 columns now. --- .../2020-03-22-simple-sse-based-pubsub-server.md | 109 +++++++++++++++------ 1 file changed, 81 insertions(+), 28 deletions(-) (limited to 'content/posts/2020-03-22-simple-sse-based-pubsub-server.md') diff --git a/content/posts/2020-03-22-simple-sse-based-pubsub-server.md b/content/posts/2020-03-22-simple-sse-based-pubsub-server.md index 939a62d..77c42a9 100644 --- a/content/posts/2020-03-22-simple-sse-based-pubsub-server.md +++ b/content/posts/2020-03-22-simple-sse-based-pubsub-server.md @@ -1,25 +1,41 @@ --- title: Simple Server-Sent Events based PubSub Server url: simple-server-sent-events-based-pubsub-server.html -date: 2020-03-22 +date: 2020-03-22T12:00:00+02:00 draft: false --- ## Before we continue ... -Publisher Subscriber model is nothing new and there are many amazing solutions out there, so writing a new one would be a waste of time if other solutions wouldn't have quite complex install procedures and weren't so hard to maintain. But to be fair, comparing this simple server with something like [Kafka](https://kafka.apache.org/) or [RabbitMQ](https://www.rabbitmq.com/) is laughable at the least. Those solutions are enterprise grade and have many mechanisms there to ensure messages aren't lost and much more. Regardless of these drawbacks, this method has been tested on a large website and worked until now without any problems. So now, that we got that cleared up, let's continue. - -***Wiki definition:** Publish/subscribe messaging, or pub/sub messaging, is a form of asynchronous service-to-service communication used in serverless and microservices architectures. In a pub/sub model, any message published to a topic is immediately received by all the subscribers to the topic.* +Publisher Subscriber model is nothing new and there are many amazing solutions +out there, so writing a new one would be a waste of time if other solutions +wouldn't have quite complex install procedures and weren't so hard to maintain. +But to be fair, comparing this simple server with something like +[Kafka](https://kafka.apache.org/) or [RabbitMQ](https://www.rabbitmq.com/) is +laughable at the least. Those solutions are enterprise grade and have many +mechanisms there to ensure messages aren't lost and much more. Regardless of +these drawbacks, this method has been tested on a large website and worked +until now without any problems. So now, that we got that cleared up, let's +continue. + +***Wiki definition:** Publish/subscribe messaging, or pub/sub messaging, is a +form of asynchronous service-to-service communication used in serverless and +microservices architectures. In a pub/sub model, any message published to a +topic is immediately received by all the subscribers to the topic.* ## General goals - provide a simple server that relays messages to all the connected clients, - messages can be posted on specific topics, -- messages get sent via [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) to all the subscribers. +- messages get sent via [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) + to all the subscribers. ## How exactly does the pub/sub model work? -The easiest way to explain this is with diagram bellow. Basic function is simple. We have subscribers that receive messages, and we have publishers that create and post messages. Similar model is also well know pattern that works on a premise of consumers and producers, and they take similar roles. +The easiest way to explain this is with diagram bellow. Basic function is +simple. We have subscribers that receive messages, and we have publishers that +create and post messages. Similar model is also well know pattern that works +on a premise of consumers and producers, and they take similar roles. ![How PubSub works](/assets/simple-pubsub-server/pubsub-overview.png) @@ -29,16 +45,23 @@ The easiest way to explain this is with diagram bellow. Basic function is simple - consumer is receiving messages from subscribed topic, - servers is also known as Broker, - broker does not store messages or tracks success, -- broker uses [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) method for delivering messages, -- if consumer wants to receive messages from a topic, producer and consumer topics must match, +- broker uses [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) + method for delivering messages, +- if consumer wants to receive messages from a topic, producer and consumer topics + must match, - consumer can subscribe to multiple topics, - producer can publish to multiple topics, - each message has a messageId. **Known drawbacks:** -- messages will not be stored in a persistent queue or unreceived messages like [DeadLetterQueue](https://en.wikipedia.org/wiki/Dead_letter_queue) so old messages could be lost on server restart, -- [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) opens a long-running connection between the client and the server so make sure if your setup is load balanced that the load balancer in this case can have long opened connection, +- messages will not be stored in a persistent queue or unreceived messages like + [DeadLetterQueue](https://en.wikipedia.org/wiki/Dead_letter_queue) so old + messages could be lost on server restart, +- [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) + opens a long-running connection between the client and the server so make + sure if your setup is load balanced that the load balancer in this case can + have long opened connection, - no system moderation due to the dynamic nature of creating queues. ## Server-Sent Events @@ -49,13 +72,16 @@ Read more about it on [official specification page](https://html.spec.whatwg.org ![Browser support](/assets/simple-pubsub-server/caniuse.png) -Check [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) for latest information about browser support. +Check [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) +for latest information about browser support. ### Known issues - Firefox 52 and below do not support EventSource in web/shared workers -- In Firefox prior to version 36 server-sent events do not reconnect automatically in case of a connection interrupt (bug) -- Reportedly, CORS in EventSource is currently supported in Firefox 10+, Opera 12+, Chrome 26+, Safari 7.0+. +- In Firefox prior to version 36 server-sent events do not reconnect + automatically in case of a connection interrupt (bug) +- Reportedly, CORS in EventSource is currently supported in Firefox 10+, + Opera 12+, Chrome 26+, Safari 7.0+. - Antivirus software may block the event streaming data chunks. Source: [https://caniuse.com/#feat=eventsource](https://caniuse.com/#feat=eventsource) @@ -78,7 +104,8 @@ data: this is line two ``` -And you can specify your own event types (the above messages will all trigger the message event): +And you can specify your own event types (the above messages will all trigger +the message event): ```bash id: 36 @@ -89,7 +116,8 @@ data: 103.34 ### Server requirements -The important thing is how you send headers and which headers are sent by the server that triggers browser to threat response as a EventStream. +The important thing is how you send headers and which headers are sent by the +server that triggers browser to threat response as a EventStream. Headers responsible for this are: @@ -101,15 +129,23 @@ Connection: keep-alive ### Debugging with Google Chrome -Google Chrome provides build-in debugging and exploration tool for [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) which is quite nice and available from Developer Tools under Network tab. +Google Chrome provides build-in debugging and exploration tool for +[Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) +which is quite nice and available from Developer Tools under Network tab. -> You can debug only client side events that get received and not the server ones. For debugging server events add `console.log` to `server.js` code and print out events. +> You can debug only client side events that get received and not the server +> ones. For debugging server events add `console.log` to `server.js` code and +> print out events. ![Google Chrome Developer Tools EventStream](/assets/simple-pubsub-server/chrome-debugging.png) ## Server implementation -For the sake of this example we will use [Node.js](https://nodejs.org/en/) with [Express](https://expressjs.com) as our router since this is the easiest way to get started and we will use already written SSE library for node [sse-pubsub](https://www.npmjs.com/package/sse-pubsub) so we don't reinvent the wheel. +For the sake of this example we will use [Node.js](https://nodejs.org/en/) +with [Express](https://expressjs.com) as our router since this is the easiest +way to get started and we will use already written SSE library for node +[sse-pubsub](https://www.npmjs.com/package/sse-pubsub) so we don't reinvent +the wheel. ```bash npm init --yes @@ -216,11 +252,17 @@ app.listen(port, () => { ### Our custom message format -Each message posted on a server must be in a specific format that out server accepts. Having structure like this allows us to have multiple separated type of events on each topic. +Each message posted on a server must be in a specific format that out server +accepts. Having structure like this allows us to have multiple separated type +of events on each topic. -With this we can separate streams and only receive events that belong to the topic. +With this we can separate streams and only receive events that belong to the +topic. -One example would be, that we have index page and we want to receive messages about new upvotes or new subscribers but we don't want to follow events for other pages. This reduces clutter and overall network. And structure is much nicer and maintanable. +One example would be, that we have index page and we want to receive messages +about new upvotes or new subscribers but we don't want to follow events for +other pages. This reduces clutter and overall network. And structure is much +nicer and maintanable. ```json { @@ -236,11 +278,15 @@ One example would be, that we have index page and we want to receive messages ab -You can download [the code](../simple-pubsub-server/sse-pubsub-server.zip) and follow along. +You can download [the code](../simple-pubsub-server/sse-pubsub-server.zip) +and follow along. ### Publisher -As talked about above publisher is the one that send messages to the broker/server. Message inside the payload can be whatever you want (string, object, array). I would however personally avoid send large chunks of data like blobs and such. +As talked about above publisher is the one that send messages to the +broker/server. Message inside the payload can be whatever you want (string, +object, array). I would however personally avoid send large chunks of data +like blobs and such. ```html @@ -309,16 +355,23 @@ As talked about above publisher is the one that send messages to the broker/serv - ``` ### Subscriber -Subscriber is responsible for receiving new messages that come from server via publisher. The code bellow is very rudimentary but works and follows the implementation guidelines for EventSource. +Subscriber is responsible for receiving new messages that come from server via +publisher. The code bellow is very rudimentary but works and follows the +implementation guidelines for EventSource. -You can use either Developer Tools Console to see incoming messages or you can defer to Debugging with Google Chrome section above to see all EventStream messages. +You can use either Developer Tools Console to see incoming messages or you can +defer to Debugging with Google Chrome section above to see all EventStream +messages. -> Don't be alarmed if the subscriber gets disconnected from the server every so often. The code we have here resets connection every 15s but it automatically get reconnected and fetches all messages up to last received message id. This setting can be adjusted in `server.js` file; search for the `maxStreamDuration` variable. +> Don't be alarmed if the subscriber gets disconnected from the server every +> so often. The code we have here resets connection every 15s but it +> automatically get reconnected and fetches all messages up to last received +> message id. This setting can be adjusted in `server.js` file; search for +> the `maxStreamDuration` variable. ```html @@ -383,7 +436,6 @@ You can use either Developer Tools Console to see incoming messages or you can d - ``` ## Reading further @@ -394,3 +446,4 @@ You can use either Developer Tools Console to see incoming messages or you can d - [An HTTP/2 extension for bidirectional messaging communication](https://tools.ietf.org/id/draft-xie-bidirectional-messaging-01.html) - [Introduction to HTTP/2](https://developers.google.com/web/fundamentals/performance/http2) - [The WebSocket API (WebSockets)](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) + -- cgit v1.2.3