Improving the quality of image segmentation in Ultrasound images using Reinforcement Learning

Ultrasound imaging is one of the imaging techniques in medicine in which the images resulted from ultrasonic wave propagation are used for diagnostic applications. Digital processing of these images for reasons is a difficult process due to causes such as noise, same distribution of intensity values, and unknown border of organs from image tissue. In this study, image segmentation and improving image and improving quality of the image segmentation was done during three stages: preprocessing, processing and postprocessing. Multi-agent dimensional structure is selected which has the best result in the image segmentation. In the processing reinforcement learning factor, threshold operator values and opening operator dimensions obtained for image segmentation. In post-processing process, for improving the quality of image segmentation, the most appropriate dimensions are determined for morphology operators. The results of implementing the presented method on the sample of the prostate is displayed.


Introduction
In recent decades, reinforcement learning is employed as a method for image segmentation.In [17], the implemented reinforcement learning method is used as a feedback for object recognition.In [18], reinforcement learning is used for determining optimized threshold values in image segmentation.The image segmentation method using reinforcement learning with algorithm Q(λ) is implemented in [3].In [14] and [15], reinforcement learning is used for segmentation of medical images after breaking it to a set of subimages.In [16], a method is employed in which threshold values for image segmentation are calculated from total weight threshold value in different various methods and reinforcement learning agent proceed to determine weights values dedicated to these values.In [8] for segmentation medical images a set of reinforcement learning agents are employed.In [1] a multi-agent structure in which each agent has a specific task is presented for image segmentation.The extension of using new technologies in various fields such as medicine application, alongside providing appropriate conditions for increasing efficiency diagnosis methods and treatment of patients, has caused the collection of high volume of data.Processing this volume of data is very time consuming and costly.One of the widely used cases in processing medical data is processing medical images.The process of image segmentation has been recognized as a fundamental method in image processing and implementing other processes about image processing such as object recognition is done according to it.Segmentation of medical images for the reason of quality of these images is considered challenging and hard and the need for using efficient methods in facing with this problem is felt.Ultrasonic images have various applications in diagnostic fields.Using an intelligent method alongside a human user can significantly help improving quality and speeding up the process of the data of this kind of images.In the presented article at first preprocessing is defined.In this process image is divided to a set of sub images, in the previously done works in segmentation medical images the impact of the sizes of sub images as a effective parameter in result of segmentation has been paid less attention.In the used method a multi-agent structure is used for selecting the most appropriate sizes of sub images and its architecture and its functionality has been explained.More in the processing stage, segmentation image has been expressed using reinforcement learning.States, actions and rewards, has been introduced in this method for reinforcement learning agent.In post-processing stage, the most appropriate sizes for morphological operators which are used for improving the quality of segmented image will be studied using trial and error method.In the end practical results of image segmentation using reinforcement learning has been displayed for different values of states.

Preliminaries and notations
Definition 2.1.Segmentation: Segmentation subdivides an image into its constituent regions or objects.The level to which the subdivision is carried depends on the problem being solved.[10].

Segmentation of Ultra sound images using Reinforcement Learning method
For processing ultrasound images using reinforcement learning method, two main phases have been considered which are as: 1-training phase 2-testing phase.In the training phase, agent acts to learn appropriate parameters according to reinforcement learning method for hand segmented images [14].At the end of training phase, a strategy table is obtained in which for each state, appropriate action is resulted, also the most appropriate sizes of sub-images and values of morphological operators for improving segmented image quality is determined.Then in testing phase according to results from training phase images will be segmented.In figure 1

Processing Stage
In this stage by using image thresholding, image is segmented then the resulted image is processed using opening morphological operator.Reinforcement learning agent results threshold values and morphological operator radius for each sub image using learning algorithm Q-learning, this algorithm is shown in figure 3.

Figure3: Q-Learning Learning Algorithm
In figure 4, the block diagram of processing stage is indicated.

 
1 1 1 2 ( , ), ( , ), ..., ( , For displaying states according to equation (3.5) vector S is used.The components of this vector include area, compression, position of sub image and the number of objects in image segmentation which according to [14] are respectively displayed by 1 3 2 4 χ ,χ ,χ and χ . in phrase (3.5) the state vector is shown by symbol .

Post-Processing Stage
The main purpose of post processing is to improve quality of the segmented image in processing stage by using opening and closing morphological operators and determining the most appropriate sizes for these operators.For determining the most appropriate sizes for morphological operators trial and error method is used.After segmenting image for various values   ,  = 1,2, … .,  which are considered as morphological operator radius with disk shape, the value of λ* corresponds with the minimum dissimilarity factor   is considered as the radius of these operators.equation (3.7) shows the manner of determining  * .* ** , arg min

The results
Obtained results are created using Matlab software.For rendering training phase ultrasound image of prostate mature person is used.Intensity matrix have size 445 × 290.Table 1 shows the values of two categories of states which for them the program was run.In each row of figure 6 the images are segmented using a discrete reinforcement learning agent with sizes according to table 2 for sub images in preprocessing stage.
Table 2: the sizes of sub images at preprocessing stage for three reinforcement learning agents In figure 7 the result of image segmentation for the minimum value of dissimilarity factor is shown which is related to agent that receives sub images with size 5 and 15.In the presented method it could be shown that by selecting sub images which have more pixel in their own column relative to row, more acceptable results in segmented image is resulted.Also by selecting the sizes of the sub images so that more sub images are produced, better quality in final segmented image is obtained.Using opening and closing morphological operators successively on the segmented image has a significant impact on the increasing quality of the final result and it gives more precise volume and the position of organs in segmented image.By decreasing size of morphological operators the real borders of the image will diminish gradually and the quality of the segmented image will decrease dramatically, using morphological operators with disk shape with small radius has a significant effect for getting result for segmented image.

Figure 1 :
Figure 1: The block diagram of employed stages in training phase

Figure 2 :
Figure 2: Architecture of multi-agent structure 2) Equation (3.3) shows mathematical expression for dissimilarity factor values   .equation (3.3),I is the image binary value resulted from segmentation by reinforcement learning agent,   indicates binary values of hand segmented image and values , ,   ,   are the numbers of available pixels in rows and columns of image and rows and columns of the image respectively.

Figure 4 :
Figure 4: The block diagram of processing

1 2 3 4 [
, , , ] S      (3.5)The reward values are displayed by , for getting reward values two successive dissimilarity values are compared to each other according to relationship (3.6), and according to the result of this comparison one of values R1 or R2 is attributed as reward to reinforcement learning agent.http://www.ispacs.com/journals/cacsa/2017/cacsa-

Table1:
Values of two categories of states (from left to right first category and second category respectively) In figure 5 ultra sound of prostate gland and manually image segmentation which are used in training phase are shown.

Figure5: prostate image
Figure5: prostate image and manually segmented image

Figure 6 :2
Figure 6: the results of prostate ultrasound image for two categories of states

Figure 7 :
Figure 7: (the right figure) segmented image (the left figure) ultrasound figure

Figure 8 :
Figure 8: the diagram of reward average values for two sub images
, the block diagram of available stages in training phase is given.http://www.ispacs.com/journals/cacsa/2017/cacsa-00072/ International Scientific Publications and Consulting Services