Parameter selection is important to the classification performance of support vector machine (SVM),which is essentially a search of optimum. This paper proposes a parameter selection method for SVM with the algorithm of improved simulated annealing particle swarm optimization (IM-SAPSO) to search the best parameters. The minimized K-fold cross-validation error is used as the object of IM-SAPSO. The optimized SVM is then used to classify the symbols 0 and 1 passing the impacting filter of an extended binary phase shift keying (EBPSK) communication system. Comparison is made for the detection performance of EBPSK detector between the proposed IM-SAPSO and other methods including those based on SVM, PSO-SVM and amplitude integral decision. Simulation results show that IM-SAPSO and SVM are significantly better than the other three methods.