Add fileSizeMax to sender #1285

[andreleblanc11]

The sender will now refuse to send files if fileSizeMax is set and if a message has a size greater then the fileSizeMax value.

'fileSizeMax': 32768,
...
2024-11-05 20:58:47,278 [INFO] sarracenia.flowcb.log after_accept accepted: (lag: 2.40 )  exchange: xpublic subtopic: 20241105.SSC-DATAINTERCHANGE.MSC-BULLETINS.20241105.SSC-DATAINTERCHANGE.MSC-BULLETINS.FP.KOKX.20 a file with baseUrl: http://ddsr-cmc-dev01.cmc.ec.gc.ca relPath: 20241105/SSC-DATAINTERCHANGE/MSC-BULLETINS/20241105/SSC-DATAINTERCHANGE/MSC-BULLETINS/FP/KOKX/20/FPUS51_KOKX_052058___234 sundew_extension: cvt_nws_bulletins-sr:KOKX:FP:3:Direct:20241105205843 id: OKcNKq5 size: 40834
2024-11-05 20:58:47,279 [WARNING] sarracenia setReport unknown report code supplied: 413:Payload Too Large http://ddsr-cmc-dev01.cmc.ec.gc.ca /20241105/SSC-DATAINTERCHANGE/MSC-BULLETINS/20241105/SSC-DATAINTERCHANGE/MSC-BULLETINS/FP/KOKX/20/FPUS51_KOKX_052058___234

I also accomodated the AM sender code to use fileSizeMax and attribute a default value if ever fileSizeMax is missing

[petersilva]

I'm fine with the sender logic... the am... is a bit more debatable.
I don't think we should raise an exception. Raising an exception will cause a retry.
You want to return None as the message, and have the thing check the return value from wrapmsg and return if it is None instead of trying to send...

It seems like any error in wrapmsg is going to result in a retry, but it isn't clear that anything that goes wrong there will go right at a subsequent attempt.

We should generate an error message, fine... but there is no reason to retry a failure that is permanent.

[petersilva]

I accepted the change... so... um usually I would just merge at that point...
but I whined about am .also... @andreleblanc11 do you want to get this merged now and work on AM some other time? or do you want to work on it, while you're already in the area ?

[github-actions]

Test Results

238 tests   236 ✅  1m 31s ⏱️
  1 suites    1 💤
  1 files      1 ❌

For more details on these failures, see this check.

Results for commit eea313b.

♻️ This comment has been updated with latest results.

[andreleblanc11]

That's something that I wanted to change, but I didn't really think about it while writing the PR. The PR was a bit of an end of the day rush.

I'll add it in tommorow.

[andreleblanc11]

The only way the AM sender will try sending files greater then fileSizeMax is if the option is unset in the configuration.

We are already emitting a logger warning when this happens in the AM sender.
So an analyst should theoretically see this, then add the option in the configuration (likely along with retry_refilter True to get all the current retries out of the loop).

I think this should be enough padding to avoid unnecessary retries.