Senior members of David Ruffley's Suffolk Tory association say they see nothing wrong with having a man who assaulted his girlfriend remain as their MP.