Recognition of apple targets before fruit thinning by robot based on R-FCN deep convolution neural network

【Author】

WANG Dandan;HE Dongjian

【Institution】

College of Mechanical and Electronic Engineering, Northwest A&F University;Key Laboratory of Agricultural Internet of Things, Ministry of Agriculture and Rural Affairs;Shaanxi Key Laboratory of Agricultural Information Perception and Intelligent Service;College of Mechanical and Electronic Engineering, Northwest A&F University;Key Laboratory of Agricultural Internet of Things, Ministry of Agriculture and Rural Affairs;Shaanxi Key Laboratory of Agricultural Information Perception and Intelligent Service

【Abstract】

Before fruit thinning, the factors such as complex background, various illumination conditions, foliage occlusion, fruit clustering, especially the extreme similarities between apples and background, made the recognition of small apple targets very difficult. To solve these problems, we proposed a region-based fully convolutional network (R-FCN) recognition method. Firstly, the deep convolution neural network including ResNet-50 based on R-FCN and ResNet-101 based on R-FCN were studied and analyzed. After the framework of these two networks was analyzed, it was obvious that the difference between these two networks was the ‘conv4’ block. The ‘conv4’ block of ResNet-101 based on R-FCN was 51 more layers than that of ResNet-50 based on R-FCN, but the recognition accuracy of these two networks was almost the same. By comparing the framework and recognition result of ResNet-50 based on R-FCN and ResNet-101 based on R-FCN, we designed the R-FCN based on ResNet-44 to improve the recognition accuracy and simplify the network. The main operation to simplify the network was to simplify the ‘conv4’ block, and the ‘conv4’ block of ResNet-44 based on R-FCN was six layers fewer than that of ResNet-50 based on R-FCN. The Res Net-44 based on R-FCN consisted of ResNet-44 fully convolutional network (FCN), region proposal network (RPN) and region of interest (RoI) sub-network. Res Net-44 FCN, the backbone network of R-FCN, was used to extract the features of the image. The features were then used by RPN to generate RoIs. After that, the features extracted by ResNet-44 FCN and RoIs generated by RPN were used by RoI sub-network to recognize and locate small apple targets. A total of 3 165 images were captured in an experimental apple orchard in the College of Horticulture, Northwest A&F University, in Yangling City, China. After image resizing and manual annotation, 332 images, including 85 images captured under sunny direct sunlight condition, 88 images captured under sunny backlight condition, 86 images captured under cloudy direct sunlight condition, 74 images captured under cloudy backlight condition, were selected as test set, and the other 2 833 images were used to train and optimize the network. To enrich image training set, the data augment, including brightness enhancement and reduction, chroma enhancement and reduction, contrast enhancement and reduction, sharpness enhancement and reduction, and adding Gaussian noise, was performed, then a total of 28 330 images were obtained with 23 591 images randomly selected as the training sets, and the other 4 739 images as validation sets. After training, the simplified ResNet-44 based on R-FCN was tested on the test set, and the experimental results indicate that the method can effectively apply to the images captured under different illumination conditions. The method can recognize clustering apples, occluded apples, vague apples and apples with shadows, strong illumination and weak illumination on the surface. In addition, the apples divided into parts by branched or petiole cloud can also be recognized effectively. Overall, the recognition recall rate can achieve 85.7%. The recognition accuracy and false recognition rate are 95.2% and 4.9%, respectively. The average recognition time is 0.187 s per image. To further test the performance of the proposed method, we compared the other three methods, including Faster R-CNN, ResNet-50 based on R-FCN and ResNet-101 based on R-FCN. The F1 of the proposed method is increased by 16.4, 0.7 and 0.7 percentage points, respectively. The average running time of the proposed method improves by 0.010 s and 0.041 s compared with that of Res Net-50 based on R-FCN and Res Net-101 based on R-FCN, respectively. The proposed method can achieve the recognition of small apple targets before fruits thinning which cannot be realized by traditional methods. It can also be widely applied to the recognition of other small targets whose features are similar to the background.

【Keywords】

image processing;algorithms;image recognition;small apple;target recognition;deep learning;R-FCN

References

To explore the background and basis of the node document

Springer Journals Database

Total: 17 articles

  • [1] Zhao Kaixuan;He Dongjian;College of Mechanical and Electronic Engineering, Northwest A&F University;, Recognition of individual dairy cattle based on convolutional neural networks, Transactions of the Chinese Society of Agricultural Engineering,
  • [2] Sang Nong;Ni Zihan;School of Automation,Huazhong University of Science and Technology;Key Laboratory of Ministry of Education for Image Processing and Intelligent Control,Huazhong University of Science and Technology;, Gesture recognition based on R-FCN in complex scenes, Journal of Huazhong University of Science and Technology(Natural Science Edition),
  • [3] Zhou Yuncheng;Xu Tongyu;Zheng Wei;Deng Hanbing;College of Information and Electrical Engineering, Shenyang Agricultural University;, Classification and recognition approaches of tomato main organs based on DCNN, Transactions of the Chinese Society of Agricultural Engineering,
  • [4] XU Yizhi;YAO Xiaojing;LI Xiang;ZHOU Nan;HU Yuan;Institute of Remote Sensing and Digital Earth,CAS;University of Chinese Academy of Sciences;Imagesky Remote Sensing Technology Co.Ltd.;, Object Detection in High Resolution Remote Sensing Images Based on Fully Convolution Networks, Bulletin of Surveying and Mapping,

More>>