Affairs could well be ‘sexy’ however they are rarely purely regarding gender