How to split the update in compressed wire format into less than 64K pieces?

[tonusoo]

dns.query.tcp() raises the dns.exception.TooBig exception if the update in compressed wire format is larger than 64 KiB. As a workaround for large DDNS updates, I check the size of the update and send it in around 32 KiB chunks. Basically something like this:

# Initialize a DNS Update object.
update = dns.update.Update(origin, keyring=keyring, keyalgorithm=key_algorithm)

for dns_update in dns_updates:
  update.add(...)

  if sys.getsizeof(update.to_wire()) > 32 * 1024:
    # Send the ~32KiB chunk and reinitialize a new DNS Update object.
    response = dns.query.tcp(update, ns, port=ns_port)
    update = dns.update.Update(origin, keyring=keyring, keyalgorithm=key_algorithm)

# Send the only or last chunk of updates to DDNS server.
response = dns.query.tcp(update, ns, port=ns_port)

This works, but maybe there is a more efficient way to handle large updates? Perhaps something like:

try:
  dns.query.tcp(update, ns, port=ns_port)
except dns.exception.TooBig:
  # split the update in compressed wire format into less than 64K chunks

I'm using dnspython version 1.16.0.

[rthalley]

If you knew a max size bound for your updates, the easiest thing would be to just assume every update added the max bound and then stop before you got to 64K, but if your updates are not easily bounded, that won't work.

If the message hitting 64K is rare, then splitting (perhaps repeatedly) your dns_update list until you got N right-size messages would work, but you don't want to be sending things multiple times, so you'd need to call message.to_wire() yourself to check the sizes, and then you could save the generated wire format on a list, and once you got a complete list, send them all with multiple calls to dns.query.send_tcp(), which can take a wire format message.

The most efficient way would be look at how the Message.to_wire() code uses the dns.renderer.Renderer, and then directly call into the renderer until you either get to the end of your list or get TooBig. When the renderer raises TooBig, it rolls back the partial write so the output is still good. If you also set the max size to a bit less than 64K to leave room for the TSIG, then you could render until you got to big, manually set max_size to 64K on the renderer object, and then render your TSIG. Then you'd send the renderer output with tcp_send() and then start over until you'd processed all your updates.