ssl/statem: Replace size_t with int and add the checks