Abstract
Replacing overexposed or dull skies in outdoor photographs is a desirable photo manipulation. It is often necessary to color correct the foreground after replacement to make it consistent with the new sky. Methods have been proposed to automate the process of sky replacement and color correction. However, many times a color correction is unwanted by the artist or may produce unrealistic results. We propose a data-driven approach to sky-replacement that avoids color correction by finding a diverse set of skies that are consistent in color and natural illumination with the query image foreground. Our database consists of \(\sim \)1200 natural images spanning many outdoor categories. Given a query image, we retrieve the most consistent images from the database according to \(L_2\) similarity in feature space and produce candidate composites. The candidates are re-ranked based on realism and diversity. We used pre-trained CNN features and a rich set of hand-crafted features that encode color statistics, structural layout, and natural illumination statistics, but observed color statistics to be the most effective for this task. We share our findings on feature selection and show qualitative results and a user-study based evaluation to show the effectiveness of the proposed method.
S. Rawat and S. Gairola— Both authors have contributed equally to this work.
Project page: https://cvit.iiit.ac.in/research/projects/cvit-projects/findmeasky.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Burt, P.J., Adelson, E.H.: A multiresolution spline with application to image mosaics. ACM Trans. Graph. 2(4) (1983)
Gollapudi, S., Sharma, A.: An axiomatic approach for result diversification. In: Proceedings of ACM WWW (2009)
Lalonde, J.-F., Efros, A.A.: Using color compatibility for assessing image realism. In: Proceedings of IEEE ICCV (2007)
Lalonde, J.-F., Efros, A.A., Narasimhan, S.G.: Estimating natural illumination from a single outdoor image. In: Proceedings of IEEE ICCV (2009)
Oliva, A., Torralba, A.: Modeling the shape of the scene: a holistic representation of the spatial envelope. Int. J. Comput. Vis. 42(3) (2001)
Rother, C., Kolmogorov, V., Blake, A.: “Grabcut”: interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23(3) (2004)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556 (2014)
Sivic, J., Zisserman, A.: Video Google: a text retrieval approach to object matching in videos. In: Proceedings of IEEE ICCV (2003)
Tao, L., Yuan, L., Sun, J.: Skyfinder: attribute-based sky image search. ACM Trans. Graph. 28(3) (2009)
Tsai, Y.-H., Shen, X., Lin, Z., Sunkavalli, K., Yang, M.-H.: Sky is not the limit: semantic-aware sky replacement. ACM Trans. Graph. 35(4) (2016)
Vedaldi, A., Fulkerson, B.: VLFeat - an open and portable library of computer vision algorithms. In: ACM MM (2010)
Wågberg, J.: OptProp: Matlab Toolbox for Calculation of Color Related Optical Properties: Version 2.1. FSCN-rapport (2007)
Wang, D., Jia, W., Li, G., Xiong, Y.: Natural image composition with inhomogeneous boundaries. In: Ho, Y.-S. (ed.) PSIVT 2011. LNCS, vol. 7088, pp. 92–103. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25346-1_9
Wang, J., Cohen, M.F.: Image and video matting: a survey. Found. Trends. Comput. Graph. Vis. 3(2) (2007)
Wong, B.Y., Shih, K.T., Liang, C.K., Chen, H.H.: Single image realism assessment and recoloring by color compatibility. IEEE Trans. Multimedia 14 (2012)
Xue, S., Agarwala, A., Dorsey, J., Rushmeier, H.: Understanding and improving the realism of image composites. ACM Trans. Graph. 31(4) (2012)
Zhou, B., Khosla, A., Lapedriza, À., Torralba, A., Oliva, A.: Places: an image database for deep scene understanding. CoRR, abs/1610.02055 (2016)
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: Proceedings of IEEE CVPR (2017)
Zhu, J.-Y., Krähenbühl, P., Shechtman, E., Efros, A.A.: Learning a discriminative model for the perception of realism in composite images. In: Proceedings of IEEE ICCV (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Rawat, S., Gairola, S., Shah, R., Narayanan, P.J. (2018). Find Me a Sky: A Data-Driven Method for Color-Consistent Sky Search and Replacement. In: Schoeffmann, K., et al. MultiMedia Modeling. MMM 2018. Lecture Notes in Computer Science(), vol 10704. Springer, Cham. https://doi.org/10.1007/978-3-319-73603-7_18
Download citation
DOI: https://doi.org/10.1007/978-3-319-73603-7_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-73602-0
Online ISBN: 978-3-319-73603-7
eBook Packages: Computer ScienceComputer Science (R0)