VMS UI_OpenSSL: if the TT device isn't a tty, flag instead of error