Fast scanning with video and photogrammetry

(Mesheritage) #1

Hi everyone,

Following a first experiment I wanted to share with the community to seek any interest/feedback in order to continue/enhance the development. Might spend more time on it if people are interested!

To give the context, MESH is working on picture gathered on internet to produce 3D models with photogrammetry. We need to deal with what we have, which is often not that great! Difference in exposition, light, contrast, time of day, quality, shadows etc..We are more focusing on processing with terrible databse rather than optimizing the scanning (Which would be way easier but we can't!).

I had a try with a fast scanning processing, which consist on taking a video around an object without any intention to make a good video. Basically, I just start the recording on my camera (Olympus Stylus) while turning it toward the object. I didn't looked at the screen nor checking the focus. I did a pretty bad job as a cameraman but it was the point!

Above are two frames from the resulting video where you can spot some obvious issues. I used Matlab to extract the frames in a basic fashion but it can be improve in many ways. If you take one frame every 20 for example, you might end up on the blurry one or with something blocking the view (like bellow). I am planning to add some selective process but one step at a time! The other way would be to extract in a fanatic way, like every frame, and sorting out by hand but I'm a lazy physicist..
Also depending on the speed and movement, it might be able to average the frame in order to increase the image quality, which I didn't do in that case.

Once the frame extracted for both object, I did no further processing. We use enhancement while taking picture from internet and got sometime better result with it, but it was nice for once to work with pics that have the same format (:wink:). So directly into the photogrammetry softwares that were in that case VisualSFM and photoscan. Photoscan worked better, which is not always true for us (reader should not forget that most of our work is based on collected images!), VisualSFM still managed to get most of the object.

Here is the model of the stone:

And here is the boat:

Note that the boat was a bit more difficult since you have the canal, the tree and pillar around while the stone occupy the entire field of view. Also due to the canal, impossible to take a shoot in front of the boat as I didn't fell like swimming during winter!

As you can see, both models could be improve with post-processing but as an experimental process the focus was on the video, if anyone want to join our adventure to refine the models we are open collaboration.

The stone model was quite straight forward, the first results were impressive.The object is a bit difficult as it contains many holes (which is the point of these stones by the way). We could think about extracting more frame around the holes, but as you can see on the first pics, the light was not the best to see inside features. The boat was more problematic, at first only half was modeled. I tried to select the "best pictures" (without anything in front) which resulted in a worst model. Turn out the best result was achieve with some bad ones (including even out of focus..).

I tried to use some markers from photoscan which worked but didn't really improve the model. Also, these markers had to be manually corrected on each images which greatly increase the time to invest. Mask didn't worked well neither but I did a quick job for that.

At the end, I was quite happy with the stone (minus the holes not modeled making holes into the model..tricky) and it appears that fastscan can be interesting to exploit even though it would require further work. If it is of interest to anybody, I would be happy to provide the code of frame selection once I spend time on it.

(Bart) #2

Super interesting approach! With regards to discarding blurry images - I wonder if you could do a 2D Fourier transform on them and compare the frequency distribution. Ones with obvious different distributions to their neighbours, or with more low than high frequencies might be the blurry ones?

It might not be perfect, but it should help you optimise the image quality.

Could you tell us a bit more about the MESH project? Are there any similarities with Project Mosul, who are crowd sourcing images from lost artefacts for 3D reconstruction?

(Mauricesvay) #3

Detecting blurry images is quite easy with OpenCV:

When shooting video, I found out that using the slow-motion mode of my iPhone gave pretty good results: it reduces motion-blur a bit because the shutter speed has to be increased to get enough frames per second. The drawback is that you need enough light.

(Vibrantnebula) #4

I am mostly processing frames from videos lately and have learned that physical stabilization of the camera is the primary bottleneck for this kind of stuff since most sensors are rolling rather than global. I suspect that the DJI Osmo platform would be amazing for this kind of capture.

(Mesheritage) #5

Thanks for your interest! Indeed, there are plenty of tool to detect blur, over exposition or to see which frame actually contain the object. Matlab contains a lot of package for image processing as well. For now, I used some Laplacian filters, which seem to be similar to the tool from OpenCV proposed by @mauricesvay.

My goal at that point was to use bad quality pictures so I didn't apply pre-selection. Now I will try to do a more serious job, but the app at the end could be use as stand alone. It's matlab package but it can be installed without the licence.

One problem is that the frame are not like "out of focus", often a small part of the object is a bit blurry with very nice focus a well detailed background, which will mess up a Laplacian filter. So there is a bit of work to do to identify the object of interest and focus the analysis on it.

MESH is volunteer based research group, where we look at digital tools to enhance the conservation, documentation and teaching of cultural heritage as well as the education. It's broad enought for us to work on different interesting project. For example we do 3D model of lost heritage, but also drone imaging of archeological site, 3D scan of heritage material and data analysis (from social networks for example).
The project started by two current PhD students at UCL (UK) in material science for cultural heritage, so we are not directly related to 3D or digital heritage but we kind of sneak in!

I guess we do have some similarity with ProjectMosul as well as other similar group when we consider the lost heritage side. But we are not doing only crow-sourced based photogrammetry nor only lost heritage. We tend to work on focused topic instead of asking people to send thing, for example on Abbeville church project, we managed to get in touch with a local photographer who supplied pictures. Another similarity is that one of MESH's founder is @ruimx who supplied some model to ProjectMosul like the Nimrud entrance (don't be surprise if you see this model in our work then :wink:).

@mauricesvay I heard nice thing about OpenCV, I'm working with Matlab which I'm aware is not common outside academia but it's quite similar. I worked also with Laplacian filter but it's awlays nice to have these article in order to optmise the code :wink: Also nice for image stacking by the way!
Thanks for the suggestion about slow-mo, I will try!

@vibrantnebula I did some model from video from a DJI that went very slowly to take the video, the result was direct (bellow), I used few image and it can be improve, but I agree on the stability. However the two models I presented in the first part, the physical stability was really horrible as the camera was just "hanging", I barely hold it, just enough to get the correct orientation. I believe we could apply some correction or doing some averaging, I will try to put that in the app, let see how it goes!