Could be the provider dropping calls, have you contacted them to ask if they see anything in their logs?
It could also be a bad timeout setting in the router, or poorly designed nat traversal/upnp are you using a STUN server?
Have you tried a different transport setting (ie. TCP or UDP)?