Non-grain production of cultivated land (NGPCL) threatened food security. Therefore, scholars have begun study this area in China and other countries, but most of the studies have focused on large scales, and few studies have focused on plot scale analysis. This study presents an analytical framework to shed light on the causes of NGPCL in the hilly mountainous regions of southern China. First, we categorized NGPCL into severe damage class and slight damage class according to the degree of damage of NGPCL to cultivated soils and the difficulty of restoring food production capacity. Then, we revealed the characteristics of spatial differentiation and causes of NGPCL in the southern hilly areas by using methods such as binary logistic regression model and spatial correlation analysis. Finally, the results in the study showed that: (1) the overall NGPCL areal in 2020 was 11288.46 hm2, accounts for 38.14%, of which the areas of NGPCL in the minor damage class and the serious damage class were 27.32% and 10.82%, respectively. (2) The spatial differentiation of NGPCL was obvious, which showed a clustered distribution pattern, with NGPCLs of the minor damage class clustered at high values in the topographically flat areas of the east-central zone, and NGPCLs of the severe damage class clustered at high values in the vicinity of the central urban area. (3) The levels of NGPCL for each type were significantly correlated with the three dimensions of natural, locational, and socio-economic factors, while topography, cultivated land infrastructure conditions and the degree of centralized and contiguous cultivated land were the important drivers of spatial differentiation of NGPCL. This paper reveals the distribution and influencing factors of NGPCL at the plot scale, that can provide theoretical reference and categorized governance suggestions for NGPCL governance in similar regions in China and even in the world.