Flickering effects and some distortion in depth data output of D435
Hello,
Now I am testing for elderly monitoring project at the hospital with D435 camera and the raw depth data is exported as CSV files. Then I converted CSV files into image files and the result I got is as following GIFs. I attached the GIFs recorded in both AM and PM. The image resolution is 320x180 and taken with 4FPS.
Image data in AM: https://www.dropbox.com/s/qm4lxhxjl4j19zx/AM.gif?dl=0
Image data in PM: https://www.dropbox.com/s/et5umh0hxkkzpov/PM.gif?dl=0
As you can see, there are some flickering effects and noise (especially in night time) between successive frames and thus the depth data is unstable. Is it because of mis-calibration or mis-tuning? What can cause this effects and how can I reduce this? And the final one is that can I get stable depth data for every frame with D435 camera?
Thank you for your time.
-
Hi Yehtet4utycc If the location that the camera is being used in has fluorescent lights such as ceiling strip-lights then they can flicker at frequencies that are hard to see with the human eye and affect the camera image. You may be able to reduce this effect by using the camera at an FPS frequency close to the operating frequency of the lights. For some lights it may be 30 FPS and for others it may be 60 FPS.
If you are seeking a constant FPS then you can either use a certain range of manual exposure value or - if you are using auto-exposure - you can keep auto-exposure enabled and disable a separate function called Auto-Exposure Priority to enforce a constant FPS.
https://github.com/IntelRealSense/librealsense/issues/1957#issuecomment-400715579
Another way to improve results may be to use an industrial-grade USB cable, for situations where dropping frames is not desirable (such as with a medical scanner). A proven supplier of these cables for RealSense is Newnex, and they are also officially cited as a source for obtaining USB cables in the data sheet from the May 2020 edition onward.
https://www.newnex.com/realsense-3d-camera-connectivity.php
For the D435, the optimal depth resolution to use to help to ensure accuracy of measurements is 848x480. Using lower resolutions may decrease accuracy of measurements. If data size is a concern then you may be able to compress the depth data by using colorization with it, as described in Intel's white-paper document on the subject.
Intel has an excellent camera tuning guide that can provide further advice about improving results.
-
Hello MartyG,
Thank you for your prompt response.
So the flickering effects caused in my data is known as fluorescent lights because of low frame rate and lower resolution, right?
Now, I am going to perform tuning again by the tuning guide (also webinar about this) and your other suggestions step by step. Thanks again.
-
The flickering is likely more to do with the frame rate than the resolution, though as mentioned, when there are fluorescent lights you should try to match the FPS to the lights as best as possible.
The excessive noise on the PM image looks as though it may be a noise phenomenon called "laser speckle" that is caused by the IR dot-pattern projection from the camera (which is much more visible in the dark).
The dot projection may be needed in a dark scene where there is no ambient light, as the camera can use the dots to process the depth of objects in the absence of a light-source (the camera can use ambient light for depth analysis during the daytime). If an external LED dot-pattern projector is used instead of the camera's laser-based built in projector then laser speckle should be reduced. The tuning guide has a section on external projectors, and Intel also have a white-paper document on the subject of external projectors.
https://dev.intelrealsense.com/docs/projectors
You could alternatively provide a light-source for the camera at night by using an infrared LED illuminator lamp near the camera.
https://github.com/IntelRealSense/librealsense/issues/2000
Or you could try applying a post-processing filter called a Hole-Filling Filter to the image to close up the holes.
https://dev.intelrealsense.com/docs/post-processing-filters#section-holes-filling-filter
There is a web version of the camera tuning guide in the link below that will make it easier for you to bookmark sections of the guide that you are interested in.
https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance
-
Hello MartyG,
I tried to reduce the flickering effects by changing the frame rate to both 30FPS and 60FPS as in the following GIFs.
30FPS: https://www.dropbox.com/s/q2veucgaiwvo4rw/30fps.gif?dl=0
60FPS: https://www.dropbox.com/s/8bm3d0ibbyjg643/60fps.gif?dl=0
But the flickering effects stiil exist in the image sequenes.
I tested with the following settings using pyrealsense2 -
- Depth frame resolution - 848x480
- FPS - 30FPS/ 60FPS
- Auto-exposure - Enable
- Downsample - 2
What problem would that be? Also the USB cable I used is original cable and have not tried Newnex cable yet.
The python code is <https://github.com/ngaye1511/D435_testing.git>
-
I re-read the case from the start. I note that you are obtaining the GIFs by converting the camera data to a CSV file and then converting the CSV to image files. If some frames were dropped during the original recording process and are missing from the recording (as can happen), those missing frames would have an effect on the subsequent conversions to other formats. And so you could end up with a flickering effect similar to a picture flip-book.
In the new 2.35.2 SDK release, Intel introduced a new feature for global-shutter cameras such as D435 called Genlock that gives far more control over frequency, including very slow ones such as 4 FPS.
https://www.intelrealsense.com/depth-camera-external-sync-trigger/
It is also worth bearing in mind that writing to csv is considered by the RealSense developers to be inefficient, and that bag file recording is the most efficient method. Normally this can generate very large files, especially during long-running. Considering that your application uses low resolution and only 4 FPS though, the generated file sizes may be much smaller.
Once you have a bag file recorded, you can either convert it to a csv file or extract PNGs directly. You may prefer to stay with the csv format though if you must use the gif format as part of your project.
-
Thank you for your advice MartyG!
Actually, I don't use GIFs in my project, I just use now to demonstrate and I also want to use only depth frames.
As for the flickering effects, I got the better results after factory reset and re-calibrating again with some additional post-processing.
I'm wondering what is the best way to save as bag file with pyrealsense2 using python for further processing. Can you recommend me some references?
Thanks for your time.
-
Unless your project absolutely depends on Python, it may be easier to use the RealSense Viewer program and use its in-built bag record function to record a depth stream into a bag at 6 FPS and a resolution of 424x240 (as close to your original resolution of 320x180 as the Viewer can get without using a lower resolution).
The Viewer also has in-built post-processing filters for improving image quality that can be turned on and off, and would have to be programmed manually into a Python project.
If use of Python is a requirement then there were not many script references I could find for recording a bag. The link below was the best that came out of my research.
https://github.com/IntelRealSense/librealsense/issues/2172
-
Hi MartyG,
Thank you for your advices along these days.
Today I think I solved the problem of flickering effects and laser speckle noises in the night time by tuning, calibrating and doing some post-processing. Firstly I made the tuning with the ReslSense Viewer and then I implemented this settings with Python using pyrealsense2. I made a comaprison as in the following links. If you see something unreliable in this case, I would be glad to know it.
Before Calibration: https://www.dropbox.com/s/5d7d51ww1flhmrf/30FPS_Before.gif?dl=0
After calibration without filters (with ambient light in the daytime): https://www.dropbox.com/s/3fn9p26xnv587k5/30FPS_No_Filters_AM.gif?dl=0
After calibration with filters (with ambient light in the daytime): https://www.dropbox.com/s/9rmpvcx9fyvufpa/30FPS_Filters_AM.gif?dl=0
After calibration without filters (with IR projector in the night time): https://www.dropbox.com/s/jqymlcdkwchyagh/30FPS_No_Filters_PM.gif?dl=0
After calibration with filters (with IR projector in the night time): https://www.dropbox.com/s/t1bnfx1rttur0ih/30FPS_Filters_PM.gif?dl=0
The settings used in the tuning are -
- Resolution: 848x480
- Frame Rate: 30
- Depth Format: Z16
- Auto Exposure: Enable
- Post-processing
- Decimation Filter: 2
- Threshold Filter: (0.3 – 8)m
- Spatial Filter: Magnitude(2), Alpha(0.5), Delta(20)
- Temporal Filter: Alpha(0.4), Delta(20)
- Hole Filling Filter: Fill from left
The only target I still needed to do is to record the depth frames as CSV files in every 3 minutes with Python. Is there any references you know for this?
Thank you for your time again! Have a nice day.
-
The filtered images look good, bearing in mind the resolution being used. Thanks for sharing your results in detail so that others can benefit from the knowledge.
Regarding a Python timer for saving a csv file: I do not have a pre-made script available, but the link below may provide some insights about how one might be created.
Please sign in to leave a comment.
Comments
11 comments