Don't call setsockopt with an invalid fd