Researchers Develop AI That Can Copy, Adjust, and Perfect Lighting in Any Image

Table of Links

• Abstract and 1. Introduction
• 2. Related Work
• 3. SwitchLight and 3.1. Preliminaries
• 3.2. Problem Formulation
• 3.3. Architecture
• 3.4. Objectives
• 4. Multi-Masked Autoencoder Pre-training
• 5. Data
• 6. Experiments
• 7. Conclusion

Appendix

• A. Implementation Details
• B. User Study Interface
• C. Video Demonstration
• D. Additional Qualitative Results & References

C. Video Demonstration

We present a detailed video demonstration of our SwitchLight framework. Initially, we use real-world videos from Pexels [1] to showcase its robust generalizability and practicality. Then, for state-of-the-art comparisons, we utilize the FFHQ dataset [25] to demonstrate its advanced relighting capabilities over previous methods. The presentation includes several key components:

1. De-rendering: This stage demonstrates the extraction of normal, albedo, roughness, and reflectivity attributes from any given image, a process known as inverse rendering.

2. Neural Relighting: Leveraging these intrinsic properties, our system adeptly relights images to align with a new, specified target lighting environment.

3. Real-time Physically Based Rendering (PBR): Utilizing the Three.js framework and integrating derived intrinsic properties with the Cook-Torrance reflectance model, we facilitate real-time rendering. This enables achieving 30 fps on a MacBook Pro with an Apple M1 chip (8-core CPU and 8-core GPU) and 16 GB of RAM.

4. Copy Light: Leveraging SwitchLight’s ability to predict lighting conditions of a given input image, we explore an intriguing application. This process involves two images, a source and a reference. We first extract their intrinsic surface attributes and lighting conditions. Then, by combining the source intrinsic attributes with the reference lighting condition, we generate a new, relit image. In this image, the source foreground remains unchanged, but its lighting is altered to match that of the reference image.

5. State-of-the-Art Comparisons: We benchmark our framework against leading methods, specifically Total Relight [34] and Lumos [52], to highlight substantial performance improvements over these approaches.