Due to large number of training parameters when applying convolutional neural network (CNN) and its variant model to image classification, the training of them is too complicated to increase the cost and cause the problem of gradient disappearance or gradient explosion. A CNN with moving average and rule decision was proposed and applied to image classification. The feature mapping layer was combined with the multilayer perceptron(MLP) layer, the weight parameters between the network layers were adjusted using the moving average, and the prediction target was optimized using the confidence rule strategy to improve the generalization performance of the model. Experimental results show that the CNN model based on moving average and rule decision has better robustness and classification effect.