Inpainting is an art of restoring an image which could be damaged, torn or distorted. It has been performed from centuries for restoring paintings performed by professional curators (usually using colours). The modern inpainting is performed on digital images and pictures which makes it different from the traditional practice. With applications such as object removal or text concealment or restoring a missing part of an image or many other such applications, this field is boosting. The algorithms used are deep learning based- GAN and CNN and non-deep learning based- Diffusion based, Exemplar and search based, Hybrid inpainting and many more. The CNN and GAN are used for large scale data.
Abstract
Inpainting is an art of restoring an image which could be damaged, torn or distorted. It has been performed from centuries for restoring paintings performed by professional curators (usually using colours). The modern inpainting is performed on digital images and pictures which makes it different from the traditional practice. With applications such as object removal or text concealment or restoring a missing part of an image or many other such applications, this field is boosting. The algorithms used are deep learning based- GAN and CNN and non-deep learning based- Diffusion based, Exemplar and search based, Hybrid inpainting and many more. The CNN and GAN are used for large scale data. This paper emphasises on non-deep learning-based algorithms.
Introduction
Image restoration has been practiced through several decades. Initially, image restoration was carried out by mainly curators or professionals on hand drawn paintings or physical arts. This is referred as inpainting. Inpainting is the art of re-constructing the damaged areas or missing parts or the one which is deteriorated. Inpainting has been practiced from medieval age to the day now. Back then, the damaged areas were filled with some colors. In this era of digital world, image inpainting is performed not only on physical art, but also on digital arts or images. There are many theories and algorithms used for inpainting. The diffusion based inpainting algorithms works well with PDEs (Partial Differential Equations). This is totally based without any user involvement (once the region to be inpainted is selected). Thus, the task to specify the texture (which to put where) is resolved [1]. The texture synthesis based inpainting is based on textures that are replicated or synthesized via a sampled texture [2]. Given two types of textures: 1. Regular 2. Stochastic. The regular textures contain repeated texels (texture elements) whereas the stochastic textures are contained without explicit texels [3-4]. Criminisi et al. proposed the exemplar based inpainting algorithm in which the order of filling region is priority based using onion peel filling process [5]. The amalgamation of texture and PDE based algorithms results in hybrid inpainting method where image is decomposed on set of images [6].
Fundamentals
The region to be inpainted is denoted by Ω and its contour is denoted by ∂Ω. being the source region.
Figure not included in this sample
Fig.1 an image with some defect that is to be inpainted
Diffusion Processes
Basically, three diffusion methods are used: Isotropic diffusion, Non-linear diffusion and Anisotropic diffusion. The isotropic diffusion uses linear heat equation, suppressing high frequencies in an image, and acts as a low-pass filter resulting blur close to edge and contours. Perona et al. [7] introduced a diffusion coefficient or conductivity coefficient which in turn reduces the diffusion and smoothing around edges and boundaries respectively. While the anisotropic regularization smoothens in directions which are due spatial [8].
Texture synthesis-based image inpainting
The missing regions are filled up with comparable neighbours of the damaged pixels utilising methods based on texture synthesis. It is helpful for regions that are larger using texture information. A three phase inpainting approach by [21] is as follows:
i. Matching and landmark extraction.
ii. Directional information is interpolated.
iii. Copy data from one picture to another.
Local inpainting techniques can be used in conjunction with interpolation to enhance the outcome if the inpainting reveals certain abnormalities in the intensity levels in the updated image. From initial seed, synthesis a texture algorithm into a pixel following the preservation of the local image [3] based on Markov Random Field (MRF), basically sampling and copying in the neighbouring pixels. Sampling and copying are performed to fill the holes. The differences in various texture algorithms are how they maintain the continuity from holes of pixel and the image pixel. As per [3], the following steps were presented:
i. Parametric statistical based –Produced stochastic textures successfully, failed in producing regular textures.
ii. Non-parametric patch based –performed on patches (patch-by-patch sampled textures) resulting in faster and good regular textures.
iii. Non-parametric pixel based –Improved quality of image than statistical based but fails in structured texture. Performs pixel-by-pixel sampled texture.
These methods often attempt to characterize the process by extracting some statistics using compact parametric statistical models from an input texture. These methods then start with an output image that contains only noise in order to create a new texture, and then keep perturbing that image until the estimated statistics of the input texture are met. For the case of image sequences, parametric statistical models have also been suggested in addition to the synthesis of still images. The fundamental disadvantage of all parametric statistical model-based approaches is that, as was already indicated, they can only be used to solve the problem of texture creation and not the more general issue of image completion. However, they can only synthesis in the limited instance of texture synthesis. Highly stochastic textures typically fail to do so for textures that also contain structures. However, on occasion, where applicable, parametric models provide you more versatility when it comes to changing the texture's properties. These techniques can also be very helpful for the procedure that is the opposite of texture generation, namely the analysis of textures [22-23].
PDE based inpainting
PDE based algorithm is more preferred to fill smaller regions and on still images using diffusion process stating that the PDE based inpainting are capable and quite handy with diffusion processes. Algorithms used for smaller regions uses several frames to restore an image. As per [6], the isophotes (grey level lines) arrives at the boundary ∂Ω, the direction of its arrival is estimated, as a result, the original image undergoes through an anisotropic diffusion (generally using tensor fields) to minimize the noise. The direction of the highest spatial change is determined by computing a discretized gradient vector at each pixel along the inpainting contour, then rotating the resultant vector by 90 degrees, following an LUV color model.
The primary issue with [6] was the replication of vast textured regions. Chan and Shen [10], based on [6] proposed a model based on total variational inpainting which follows the restoration model [12], following the Euler-Lagrange’s equation. The drawback using TV model is that is cannot inpaint a single object which is distanced from the inpainting region, due which restoration is not possible.
Figure not included in this sample
Fig.2 RGB color model. [6]
Later Chan and Shen in [11] introduced CCDs – Curvature Driven Diffusions based on non-texture images and anisotropic diffusion as used in TV model and Bertalmio [6]. The CCD model can inpaint larger areas compared to TV model. In [13], the authors proposed an algorithm which inpaints rapidly in expense of the quality of the image, depending upon convolution operator using an average kernel, weighted zero at the center. It is applicable for small areas where artifact (blurry) images are not a high concern.
Figure not included in this sample
Fig.3 The TV model is against the connectivity principle. [10]
Gopinath et al. in [14] introduced inpainting using median diffusion. Here, the Laplacian distribution's median is the location's greatest probability estimate. In order to inpaint the inner region, the suggested technique diffuses pixels with the median value from the outside area into that area. The edge is preserved by the median filter.
Exemplar based inpainting
This technique, developed by Criminisi et al. [5], uses isophote-driven inpainting and texture creation to great effect with capability to breed in both linear and 2-D structure. The priority-based method in this algorithm is utilised to choose the order in which regions are filled. They introduced a new filling process rather than the conventional filling process viz. onion peel.
Any exemplar-based technique follows following steps:
i. Set up the target region.
ii. Determine the target region’s boundaries.
iii. Choose a patch from the area that needs to be inpainted.
iv. Find the patch in the image that most closely resembles the one you’ve chosen. –Mean Square Error (MSE) used to determine the best match patch.
v. Update the image’s metadata.
The priority function at Criminisi’s algorithm is defined as a product of two term:
Formula not included in this sample
Where, C(p) and D(p) represents confidence term and data term of patch respectively.
Formulas not included in this sample
, α is a normalization factor
, |ψp| is the area of p
, np is a unit vector orthogonal to the front δΩ in the point p and ⊥ denotes the orthogonal operator.
Set up the target region. Without implicit and explicit segmentation, the algorithm is successful in filling the target region, without no edge blur and over-shooting artifacts. Speed and accuracy are also key points to note in this algorithm. The limitations are that curved structures are not handled correctly by this approach, and biassing results from improper patch selection.
In [15], the authors approached a percentile priority based concentric filling (PPCF), based on K-nearest neighbour approach using hierarchical decomposition. In the direction of isophote direction, the image structures are well preserved but less than [5]. The template matching is based on SSD – sum of squared matching, leading to greater visual gain. In [16], the Robust Exemplar Based Algorithm -REBA (a modified version of Classical Exemplar based algorithm) using an additive form of computation of priority function.
Formulas not included in this sample
α and β are the components weight. The regularising factor ω regulates the smoothness of the curve. The above algorithms produced visually poor results as the mean square error of 2 or more similar patches are same.
Figure not included in this sample
Fig.4 Selection of incorrect patch. [19]
In [19], the authors tried to resolve the above issue using the calculation of variance of MSE i.e., the variance of the patch’s pixel.
In [17], the approach for automatically removing local flaws such blotches and impulse noise from ancient motion picture films is presented in this study. Fuzzy prefiltering, motion-compensated blotch detection, and spatiotemporal inpainting are all phases in the fully automated procedure. Based on the examination of color distribution, the authors in [18] proposed algorithms for priority matching. Using non-local means, the author in [20], uses multiple samples from the image giving more quantitative and qualitative image with the help of weighted aggregated equation.
Formula not included in this sample
Where h is coefficient of decay.
Hybrid inpainting
In [24], the authors proposed a hybrid inpainting model using TV-L1equation (to decompose textural part of an image) and a bi-directional PDE (based on inviscid Helmholtz vorticity equation, to decompose structural part of an image). The TV-L 1 have a wider dynamic range. TV model is as follows:
Formula not included in this sample
TV-L 1 model is as follows:
Formula not included in this sample
Where, I0is an original image, Iis the resultant image, the texture part is represented as the difference between Iand I0.The exemplar model can preserve texture information while PDE can’t inpaint textured region well. So, both are well used in this model. In the exemplar model, the confidence term equation in texture part is:
Formula not included in this sample
Where, the entire image is Ω.
Based on Bezier curves (to model smooth curves), in [25], later using exemplar and edge-based image restoration algorithm (based on [26]), the author proposed a new hybrid model using Otsu’s thresholding for restoring image structures.
The modified Lowry method is regarded as one of the accepted techniques for calculating the protein content during the quality control of latex gloves. However, this conventional approach necessitates a number of tiresome chemical procedures, which leads to time-consuming. To overcome such, a process named Computerised Colorimetric Protein Estimation Method (CCPE) was implemented in [27], with determination of protein concentration efficiently.
Figure not included in this sample
Fig.5 A comparison between two different techniques (a) a synthetic image (b) masked image (c) Bezier curves based inpainted image (d) Exemplar based inpainted image. [25]
A portion of the CCPE approach will include scanning glove samples with a scanner to create digital images of the samples. After that, computational techniques will be used to the picture of the digitalized glove sample to determine the protein content. Wrinkles on the image will, however, reduce the accuracy of the estimation findings throughout the scanning procedure. To eliminate wrinkle of the image the authors in [27] used inpainting method based on Bezier-Exemplar Hybrid based image inpainting with 96% accuracy of protein concentration and 99% restored inpainted wrinkled image. [1-32]
Study of different methods
Table not included in this sample
Conclusion
This paper gives a glimpse on image inpainting along with its categories and different techniques used to perform it. Various algorithms are proposed by different researchers. Here, a few of them are mentioned. Currently, there are no particular methods that complete the task to do so. All techniques have their flaws and benefits with various usages.
Future Work
In future, it would be delighted to work on the current algorithms and to implement it on various sources. It would be done so to help the humankind in restoring old photographs as they carry immense memories for one.
References
1. Bertalmio, Marcelo, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. "Image inpainting." In Proceedings of the 27th annual conference on Computer graphics and interactive techniques , (2000) pp. 417-424.
2. Suthar, Rajul, and Mr Krunal R. Patel. "A survey on various image inpainting techniques to restore image." Int. Journal of Engineering Research and Applications 4, no. 2 (2014): 85-88.
3. Efros, Alexei A., and Thomas K. Leung. "Texture synthesis by non-parametric sampling." In Proceedings of the seventh IEEE international conference on computer vision , vol. 2,(1999) pp. 1033-1038. IEEE.
4. Patel, Pritika, Ankit Prajapati, and Shailendra Mishra. "Review of different inpainting algorithms." International Journal of Computer Applications 59, no. 18 (2012).
5. Criminisi, Antonio, Patrick Pérez, and Kentaro Toyama. "Region filling and object removal by exemplar-based image inpainting." IEEE Transactions on image processing 13, no. 9 (2004): 1200-1212.
6. Bertalmio, Marcelo, Luminita Vese, Guillermo Sapiro, and Stanley Osher. "Simultaneous structure and texture image inpainting." IEEE transactions on image processing 12, no. 8 (2003): 882-889.
7. Perona, Pietro, and Jitendra Malik. "Scale-space and edge detection using anisotropic diffusion." IEEE Transactions on pattern analysis and machine intelligence 12, no. 7 (1990): 629-639.
8. Guillemot, Christine, and Olivier Le Meur. "Image inpainting: Overview and recent advances." IEEE signal processing magazine 31, no. 1 (2013): 127-144.
9. Rathish Kumar, B. V., and Abdul Halim. "A linear fourth-order PDE-based gray-scale image inpainting model." Computational and Applied Mathematics 38, no. 1 (2019): 1-21.
10. Chan, T., Shen, J. Mathematical Models for Local Deterministic Inpaintings . UCLA CAM TR (2000) March 00-11.
11. Chan, Tony F., and Jianhong Shen. "Nontexture inpainting by curvature-driven diffusions." Journal of visual communication and image representation 12, no. 4 (2001): 436-449.
12. Rudin, Leonid I., and Stanley Osher. "Total variation-based image restoration with free local constraints." In Proceedings of 1st international conference on image processing , vol. 1, (1994) pp. 31-35. IEEE.
13. Richard, M. M. O. B. B., and M. Y. S. Chang. "Fast digital image inpainting." In Appeared in the Proceedings of the International Conference on Visualization, Imaging and Image Processing (VIIP 2001), Marbella, Spain , pp. 106-107.
14. Gopinath, Nithin, K. Arjun, J. Adithya Shankar, and Jyothisha J. Nair. "Complex difffusion based image inpainting." In 2015 1st International Conference on Next Generation Computing Technologies (NGCT) , (2015) pp. 976-980. IEEE.
15. Le Meur, Olivier, Josselin Gautier, and Christine Guillemot. "Examplar-based inpainting based on local geometry." In 2011 18th IEEE international conference on image processing , (2011) pp. 3401-3404. IEEE.
16. Cheng, Wen-Huang, Chun-Wei Hsieh, Sheng-Kai Lin, Chia-Wei Wang, and Ja-Ling Wu. "Robust algorithm for exemplar-based image inpainting." In Proceedings of International Conference on Computer Graphics, Imaging and Visualization , (2005) pp. 64-69.
17. Gangal, Ali, and Bekir Dizdaroglu. "Automatic restoration of old motion picture films using spatiotemporal exemplar-based inpainting." In International Conference on Advanced Concepts for Intelligent Vision Systems , (2006) pp. 55-66. Springer, Berlin, Heidelberg.
18. Zhang, Qing, and Jianjun Lin. "Exemplar-based image inpainting using color distribution analysis." Journal of information science and engineering 28, no. 4 (2012): 641-654.
19. Goyal, Pulkit, and Sapan Diwakar. "Fast and enhanced algorithm for exemplar based image inpainting." In 2010 Fourth Pacific-Rim Symposium on Image and Video Technology , (2010) pp. 325-330. IEEE.
20. Wong, Alexander, and Jeff Orchard. "A nonlocal-means approach to exemplar-based inpainting." In 2008 15th IEEE International Conference on Image Processing , (2008) pp. 2600-2603. IEEE.
21. Sung Ha Kang, Chan, T. Chan, S. Soatto, "Inpainting from Multiple Views," First International Symposium on 3D Data Processing Visualization and Transmission , (2002), pp. 622-625.
22. Pandya, Nirali, and Bhailal Limbasiya. "A survey on image inpainting techniques." International Journal of Current Engineering and Technology 3, no. 5 (2013): 1828-1831.
23. Liu, Yaojie, and Chang Shu. "A comparison of image inpainting techniques." In Sixth International Conference on Graphic and Image Processing (ICGIP 2014) , vol. 9443, (2015) pp. 347-357. SPIE.
24. Wu, Jiying, and Qiuqi Ruan. "A novel hybrid image inpainting model." In 2008 International Conference on Audio, Language and Image Processing , (2008) pp. 138-142. IEEE.
25. Wu, Jiunn-Lin, and Yi-Ying Chou. "An effective content-aware image inpainting method." Journal of Information Science and Engineering 28, no. 4 (2012): 755-770.
26. Rares, Andrei, Marcel JT Reinders, and Jan Biemond. "Edge-based image restoration." IEEE Transactions on Image Processing 14, no. 10 (2005): 1454-1468.
27. Efros, Alexei A., and William T. Freeman. "Image quilting for texture synthesis and transfer." In Proceedings of the 28th annual conference on Computer graphics and interactive techniques , (2001) pp. 341-346.
28. Fang, Chih-Wei, and Jenn-Jier James Lien. "Rapid image completion system using multiresolution patch-based directional and nondirectional approaches." IEEE Transactions on Image Processing 18, no. 12 (2009): 2769-2779.
29. Modi, H., Sahasrabuddhe, M., Investigation and Implementation of Square Type Patch Based Algorithm on Texture Synthesis of Image Inpainting Technique for Medical Images, (2013) International Review on Computers and Software (IRECOS) , 8 (1), pp. 363-369.
30. Elharrouss, Omar, Noor Almaadeed, Somaya Al-Maadeed, and Younes Akbari. "Image inpainting: A review." Neural Processing Letters 51 (2020): 2007-2028.
31. Jam, Jireh, Connah Kendrick, Kevin Walker, Vincent Drouard, Jison Gee-Sern Hsu, and Moi Hoon Yap. "A comprehensive review of past and present image inpainting methods." Computer vision and image understanding 203 (2021): 103147.
32. Gupta, Megha, and R. Rama Kishore. "Different techniques of image inpainting." In Computational Methods and Data Engineering: Proceedings of ICMDE 2020, Volume 1, pp. 93-104. Springer Singapore, 2021.
- Citation du texte
- Hardik Modi (Auteur), Dhruva Patel (Auteur), Sagar Patel (Auteur), Fundamentals of Different Image Inpainting Techniques, Munich, GRIN Verlag, https://www.grin.com/document/1328014
-
-
-
-
Téléchargez vos propres textes! Gagnez de l'argent et un iPhone X. -
-
Téléchargez vos propres textes! Gagnez de l'argent et un iPhone X. -
-
Téléchargez vos propres textes! Gagnez de l'argent et un iPhone X. -
-
Téléchargez vos propres textes! Gagnez de l'argent et un iPhone X. -
-
Téléchargez vos propres textes! Gagnez de l'argent et un iPhone X. -
-
Téléchargez vos propres textes! Gagnez de l'argent et un iPhone X. -
-
Téléchargez vos propres textes! Gagnez de l'argent et un iPhone X. -
-
Téléchargez vos propres textes! Gagnez de l'argent et un iPhone X. -
-
Téléchargez vos propres textes! Gagnez de l'argent et un iPhone X. -
-
Téléchargez vos propres textes! Gagnez de l'argent et un iPhone X. -
-
Téléchargez vos propres textes! Gagnez de l'argent et un iPhone X.