make sure custom cipher flag doesn't use any mode bits