“We’ve said we haven’t ruled anything off the table… but actually to have a second referendum would be to throw this Government a lifeline.”