Impacting filter can change phase jump of the extended binary phase shift keying (EBPSK) signal into magnitude impact so as to enlarge the difference between waveforms of 0 and 1. To take full advantage of the special filtering mechanism, we design an RS coded EBPSK-MODEM based on the support vector machine (SVM) discrimination. After briefly describing the principle of EBPSK modem, an EBPSK detector model based on SVM is proposed, which uses the classification ability of SVM. Performances of binary phase shift keying (BPSK) and EBPSK are compared, and an RS coded EBPSK modem based on SVM is designed. Simulation results show that, taking bit·s−1/Hz/SNR as an aggregate indicator, EBPSK system outperforms BPSK, and, compared with the RS coded EBPSK modem based on adaptive threshold decision, the proposed SVM-based method can obtain about 1 dB coding gain in AWGN channel at BER of 10−4.