"On the bit or on the buckle."
Don't "sort of" have a contact. Try to either ride with a connection, or ride with a loose rein. That "in between" area where there is sometimes a feel and sometimes not, is where horses learn to fear and/or evade the contact.
