Need some help.

[jalotra]

I have been trying to do handwriting recognition on Doctor prescription images. Since doctors don't necessarily write in a single line. I am wondering if this Network definition can fit my dataset after training on IAM Database on segmented word images. Any insight will be helpful. I have implemented the architecture defined here: https://arxiv.org/pdf/1507.05717.pdf. But this overfits on the IAM DataBase. What should I do?