Interactive Photo Editing on Smartphones via Intrinsic Decomposition
- 1. 1
- 2. 2
- 3. Recoloring Stylization Retexturing Appearance Editing Meka et al., SIGGRAPH 2016 Bonneel et al., Eurographics STAR, 2017 Beigpour et al., JPI, 2018 3
- 4. 4 Design Objectives Feasibility • Accessibility – Possible on readily available smartphones. • Speed – All post-capture processing takes at most few seconds. • Interaction – Editing is possible at interactive rates. • Quality – Rendered output looks plausible. • Built-In Depth Sensors • High-End GPU • Graphics API
- 5. 5 Optimization-based Methods Learning-based Methods -- Mostly are not real-time or interactive -- Resource intensive -- Limited to low image resolutions -- Both class of methods assume only diffuse reflection Bell et al., SIGGRAPH 2014 Input Albedo Lettry et al., Pacific Graphics 2018 Albedo Input
- 6. 6
- 7. Diffuse + Depth Albedo Shading Output Depth RGB-D Input Specularity Removal Intrinsic Decomposition Image Editing Specular 7
- 8. 8 Works quite well for images in a controlled setting. Input Fu et al., 2019 Akashi et al., 2016 Shen et al., 2009 Ground Truth
- 9. 9 However, does not generalize well for real-world images. Input Fu et al., 2019 Akashi et al., 2016 Shen et al., 2009
- 10. Intensity (In) Intensity (Out) Chroma (In) Chroma (Out) Input Diffuse Reducing positive coefficients of subbands in the masked region Specular Mask Inpainting missing information by iterative bilateral hole-filling High Freq. Low Freq. Sub-bands 10
- 11. 11 Diffuse + Depth Albedo Shading Output Depth RGB-D Input Specularity Removal Intrinsic Decomposition Image Editing Specular 11
- 12. 𝐸 𝑎, 𝑠 = 1 2 𝜆𝑑𝐸𝑑 + 𝜆𝑟𝑎𝐸𝑟𝑎 + 𝜆𝑟𝑠𝐸𝑟𝑠 + 𝜆𝑠𝑝| 𝑎 |1 𝑎𝑟𝑔𝑚𝑖𝑛 𝑎, 𝑠 Data term Retinex-Albedo term Retinex-Shading term 𝐿1- Regularization of Albedo Intrinsic Decomposition 12
- 13. Intrinsic Decomposition Diffuse + Depth Shading Albedo 13
- 14. 14
- 15. 15 Input Ours Fu et al., 2019 Akashi et al., 2016 Shen et al., 2009
- 16. 16 Input Only Low Freq. Reduced Only High Freq. Reduced Both Freq. Reduced Both Freq. Red. + Chroma Inpaint.
- 17. 17 Input Only Low Freq. Spec. Removal Only High Freq. Spec. Removal Specularity Enhanced
- 18. Input 18 Ours (RGB-D) Bell et al., 2014 (RGB) Jeon et al., 2014 (RGB-D) Lettry et al., 2018 (RGB) Our results are comparable to existing techniques.
- 19. 19 Input Ours (RGB-D) Bell et al., 2014 (RGB) Jeon et al., 2014 (RGB-D) Lettry et al., 2018 (RGB) Existing methods do not perform comparatively well for outside scenes
- 20. 20 𝐸 𝑎, 𝑠 = 1 2 𝜆𝑟𝑎𝐸𝑟𝑎 + 𝜆𝑑𝐸𝑑 + 𝜆𝑟𝑠𝐸𝑟𝑠 + 𝜆𝑠𝑝| 𝑎 |1 𝑎𝑟𝑔𝑚𝑖𝑛 𝑎, 𝑠 Data term Retinex-Albedo term Retinex-Shading term 𝐿1- Regularization of Albedo 𝐸1 𝐸2 𝐸3 Input Albedo – Only 𝐸1 Albedo – 𝐸1+ 𝐸2 Albedo – 𝐸1+ 𝐸2 + 𝐸3
- 21. 21
- 22. Input Diffuse Specular Albedo Shading Diffuse Input Silk Mystique Glass God Rays Toon 22
- 24. Input Low-density Fog High-density Fog 24
- 25. Input God Rays God Rays + Fog 25
- 26. Input Comic Quickshift Halftone 26
- 27. 27 Input Ground Truth Ours Albedo Edited Albedo initialization with chromaticity leads to color shifts in the edited results. Input Diffuse Our specular pixel identification step is not robust, thus can potentially make the diffuse output darker.
- 28. 28 • Relax some of the existing assumptions such as -- single-color illuminant and only direct illumination. • Perform super-resolution of depth map thereby enhancing the result quality. • Make the specular mask generation more robust. • A novel specularity removal method, that works for real-world images. • A system for intrinsic decomposition of images on smartphones. • Our GPU-based implementation allows for interactive image editing. • We showcase a variety of applications. Future Work