Don't "sort of" have a contact. Try to either ride with a connection, or with a loose rein. That "in between" area, where there is sometimes a feel and sometimes not, is where horses learn to fear and/or evade the contact.