ARTICLE AD BOX
colleagues!
- I use the "nats" message broker in my Golang project.
- Currently, when using "NATS," large messages aren't compressed at all, causing the producer to crash with an error (the default maximum payload message size is 1 MB).
- Working with "nats" is done via Jetstream streams.
- What I've already tried:
- At first, I wanted to set the "--compression s2" option when creating a stream (but it only compresses messages AFTER the message is published to the nats server and compresses them at the last moment BEFORE saving to disk). Therefore, this option is NOT suitable for us.
- You can change the "max_payload = 10 MB" parameter in the nats-server.conf configuration file (or max_payload = 20, 30, 40 MB, etc., to set the desired maximum message size). This solved the problem LOCALLY.
- BUT, according to the nats documentation, the max_payload parameter can be set to no more than 64 MB, and in a production environment, no more than 8 MB is recommended.
- Therefore, we need a solution that WILL NOT WORK. Adjust the max_payload parameter!
- I see the following solutions:
Split the message (file) into chunks and merge them back into a single file upon receipt.
Use minio file storage and only send a link to the file in messages.
Compress files before sending (could you suggest a library for Golang?)
Please, colleagues, what are your suggestions?
- I'll say right away - not all files are large; most fit under 8MB, but I'd like to come up with a solution that covers all possible cases.
