Support verification of s3 delivered files

[johnbradley]

We need a method to verify that the files that have been delivered and stored in s3 have not been modified. This should include issues with a bad actor modifying the files in the bucket.
It would also be nice if the verification could take place even without access to the D4S2 database.

We have discussed adding a MD5 check sum to the metadata and/or adding a digital signature.
There is a chicken and egg problem with creating a digital signature based on the metadata, and storing that signature in the metadata.

[johnbradley]

Spoke with @dleehr about this.

Current Plan

Sender digitally signs all files, D4S2 creates a manifest containing all file's signatures, D4S2 signs the manifest. Recipient receives a bucket with all files and the manifest file. Each received file has a metadata signature.

Step by step:

• Sender uploads files
  • each file is digitally signed using sender's private key
  • this signature is added to the file's metadata when uploading
• D4S2 responds to the sender requesting to send the delivery
  • D4S2 creates a manifest and stores it in the database
  • D4S2 digitally signs the manifest using D4S2's private key
  • D4S2 stores the signature in the database
  • D4S2 emails Recipient
• Recipient accepts delivery via D4S2
  • D4S2 copies files into the recipient's bucket (retaining the metadata signature)
  • D4S2 copies the manifest into the recipient's bucket (adding the metadata signature)
• Verifying recipient's bucket without access to D4S2
  • verify the manifest file against the D4S2 public key
  • verify other files against the Sender's public key
  • verify the signatures of the files recorded in the manifest file match those of the files

[johnbradley]

One concern is to see how long it takes to sign all files being transferred.

[hlapp]

I like this a lot, very thoughtful.

[johnbradley]

Questions

• Do we need to keep a copy of the public keys of users delivering data?
• Would storing signatures in files be better? This would no not rely on s3 metadata.
• Do we need to store manifest as text in addition/instead of JSON?
• Do we need to store the manifest signature in the database?

Notes

Metadata size limitation

AWS S3 metadata size limitation:

user-defined metadata is limited to 2 KB in size
In practice this hasn't been a problem with attaching base64 encoded SHA256 digests as metadata.

Large file signing performance

Signing a 4G file took 12 seconds on a Bespin VM: openssl dgst -sha256 -sign ...