Completely ordinary photos are being transformed into clean, high-resolution 3D worlds thanks to algorithms from the Harvest4D consortium.

This is a guest blog post written by the project coordinator, Michael Wimmer, TU Wien

As our Harvest4D EU project is drawing to an end, it is time to reflect on what we have achieved in the past 3 years. Generally, it is no longer unusual for computers to display our world in three dimensions. 3D scanners can scan faces, buildings or entire landscapes, and the data can be used to generate 3D models. In most cases though, this process is very complex. In the 'Harvest4D' research project, we have developed algorithms that can be used to generate these three-dimensional worlds much more easily, using existing image data not necessarily collected for this purpose. Even the development of these worlds over time can be studied on the computer in this way. This type of software solution can be applied to a range of different fields, from archaeology to flood research.

We are very proud to lead this EU funded consortium of excellent researchers (TU Wien, University of Bonn, ParisTech Telecom, TU Darmstadt, ISTI CNR Pisa, TU Delft) from the various fields of computer graphics and computer vision..

Creating 3D worlds from 2D photos

Up until now 3D images of reality have mainly been created using extremely complicated methods such as laser scanners. Experts have to plan a scan campaign carefully, take high-resolution images and painstakingly edit the data at the end. That is the only way to generate beautiful, smooth surfaces and clear shapes from a three-dimensional point cloud.

We are envisioning a radical paradigm shift in this area. A range of data is available to us that is more comprehensive than ever before, and the right algorithms enable us to use this image data. Devices that can take high-resolution images are becoming more and more commonplace. Almost everyone has a camera phone with them these days. A few years ago, quadcopters with webcams were still expensive, high-tech equipment; today they are an affordable gadget.

Appropriate calculation methods can be used to generate three-dimensional worlds from large collections of data. The images are not just pieced together on the computer; we use them to compute a complete 3D model. You can freely move around an object on the screen and view it from any angle, including those that were not originally present in the images. Excavations can be studied and analysed virtually; they can even be made accessible in high quality to users at home.

In order to achieve an “error-tolerant image analysis”, it is necessary to develop programs that tolerate errors, as the data is never perfect. Images are sometimes out of focus, they show objects in varying light conditions, and scans can include artefacts that interfere with the reconstruction. Despite this, the calculation methods developed in our research project make it possible to process this type of image data automatically. Archaeological excavations have, for example, been visualised in 3D in this way, where the raw data consisted of completely ordinary photos. In this case, the rough structure was also measured with a laser scan. The level of detail of the visualisation does not always have to be the same: for ordinary stonemasonry a more basic representation suffices, whereas you may want to look at valuable frescos on the wall in more detail. High-resolution images of frescos can therefore be additionally incorporated into 3D models.

If you do not need the data for scientific purposes, you do not necessarily need to use the maximum level of detail. If you are satisfied with a bit less precision, even end devices such as laptops and smartphones can present smooth 3D models. TU Wien's Institute of Computer Graphics designed the potree viewer especially for this purpose. It uses specialist techniques to display large point clouds that do not fit into the memories of low-power end devices.

Changes in 4D

If you know the time the images were taken, you can add a temporal dimension to the three spatial dimensions, and investigate how the observed 3D world changes. This is also a complex task for computers: have the depicted objects actually changed, or do the images just look different because they were taken in different light conditions, from different angles or with different devices? Human beings usually find it pretty easy to answer such questions, but it is a big challenge to teach this skill to a computer.

EURONEWS TV documentary on Harvest4D on air 6th June 2016

During our last project meeting in May here in Vienna, we made an experience with the Euronews film crew observing a Harvest4D capturing campaign in action. We could present how all of our project components work together. It was a fascinating experience to explain our methods in a generally understandable way. Even more so, to see our methods pass another practical test scenario! You can watch the video on Euronews Futuris programme online.