In this very intriguing experiment, an ARRI ALEXA Classic 2K footage were undergone AI ‘treatment’ to output 8K footage by using machine learning technologies. How much Artificial Intelligence methodologies can assist us in resolution upscaling, instead of perusing after 8K sensors?
In a video titled “This Is What an 8K ARRI ALEXA Would Look Like…”, a very neat demonstration of technology called AI upscaling was presented. The concept is very simple. Take the very high-quality 2K ProRes footage from the old but mighty ARRI ALEXA Classic, and by utilizing AI (Artificial Intelligence) upscaling machine learning technology, turn it into 8K footage. As you can figure out, the process demands some time (for a video from HD to 8K, it typically takes 2-3 seconds for each frame). This technology has been established by a company called Topaz. Its related product is called Video Enhance AI, which can enhance, and improve many aspects of videos, from implementing slow motion, to resolution upscaling.
As stated by Topaz: “Using the power of machine-learning AI technology, Video Enhance AI is trained on thousands of videos and combines information from multiple input video frames. It can enlarge HD video to 8K resolution with correct details and motion consistency. For example, you can upscale SD video to HD or even 4K resolution”. The technique being presented here is known as “AI upscaling.” In essence, you feed an algorithm a low-resolution image, and, based on training data it’s seen, it spits out a version that looks the same but has more pixels in it. “We support up to 8K video output. Any size larger than 8K will cause a failure. But you can set the output format as a sequence of images (png/tif/tiff) which can support nearly any size” Topaz emphasizes.
Upscaling, as a general technique, has been around for a long time, but the use of AI has drastically improved the speed and quality of results. In the case of banal upscaling algorithms, these rules are often pretty simple. If you want to upscale a 50 x 50-pixel image to double its size, for example, a traditional algorithm just inserts new pixels between the existing ones, selecting the new pixels’ color based on an average of its neighbors. However, Video Enhance AI utilizes Artificial Intelligence algorithms created by machine learning by ‘looking’ at whole sections of images at a time.
“We originally built Video Enhance AI to improve the most common quality issues affecting videos, including de-interlacing footage, intelligent upscaling up to 8K resolution, and seamless frame rate conversion,” says Topaz, and adds: “Because it treats video as a series of constant rate images, it led to problems when the app’s AI models had to process variable frame rate footage, frame repeats, and scene changes. It also complicated our ability to develop new models to address stabilization or scene detection because they require multiple passes over the video”. Hence, to develop these new features and to continue innovating, Topaz has built this next generation of the app at the top with an entirely new code base and a brand new user interface with numerous functionality improvements, such as stacking multiple AI models, executing parallel tasks, and project queue filtering.
The price of the Topaz Video AI app is $300. You can try it for free as well. The system’s minimum requirements are:
- computer: A DirectX12 compatible GPU (NVidia or AMD). An Intel CPU from 2015 onwards or an AMD CPU from 2016 onwards are our minimum requirements for the CPU. On a PC with Nvidia GTX 1080 for a video from HD to 8K, each frame typically takes 2-3 seconds. If you upscale a video from SD to HD, it will take about 0.4 seconds for each frame.
- mac: 8GB or more RAM, 2GB of VRAM. MacOS 10.14 (Mojave) to run on CPU only. MacOS 10.15 (Catalina) or newer to run on GPU. This gives the best performance.
Back to the ALEXA Classic experiment. It’s not a secret that this camera owns a very solid big-screen history. Its 2K resolution footage was used in IMAX theaters (equivalent to 11K resolution). Hence, it’s not a big deal that its files would look good at 8K. However, we can discuss the 8K resolution race that many camera manufacturers are obsessed with. AI-upscaling solutions are here to stay. If so, a 4K sensor would be good enough for 8K enhancement without the need to develop an 8K sensor with impaired dynamic range capabilities. Thus, a 4K sensor plus machine learning software can do a better job than a native 8K sensor for 8K imagery. What are your takes on this?