In the case of insufficient or unbalanced source data， deep learning method is diffi-cult to achieve satisfactory emotion recognition effect on small sample set.Therefore， thispaper constructs a three-layer random forest emotion recognition network， which separatesthe easily distinguishable emotion categories in each layer， through the importance scoringmethod.A feature set for identifying specific categories is constructed for each layer of thenetwork， each feature of the feature set is weighted according to the contribution degree toensure that the feature factor pairs that contribute more to the classification result， thegreater the impact.The multi-level emotion recognition network constructed in this paper improves the overall recognition rate of speech emotion recognition in small sample sets by5%and 7%respectively compared with single-layer random forest network and supportvector machine， and the network increased by 12%compared to popular deep learningmethod CNN.The experimental results and theoretical analysis show that the multi-levelrandom forest network based on importance score has higher recognition accuracy andspeech emotion recognition than other methods when the source data sample size is smalland partially unbalanced， so that it has the practical significance of the direction.