Cannot get stable framerate on RealSense D455
Hi all,
I recently acquired Intel RealSense D455 for my research project, but cannot get it working properly.
Whatever combination of FPS and Resolution I do, I get random frame drops when using both Depth and RGB channels. Camera does not stabilize on some fixed FPS. So far I tried changing the cable, installing latest drivers, nothing really helped. My machine is rather powerful one (below is all configurations), so I think it's unlikely that it is machine problems.
Another problem is that it looks like RealSense Viewer has memory leaks. The longer RealSense Viewer works, the more memory is being consumed.
Here's the configuration:
Computer: Dell Optiplex 7080 Small Form Factor, Intel i7 processor, 16Gb RAM, Intel Integrated Graphics (UHD Graphics 630).
Camera: Intel Realsense D455. Firmware version: 05.12.13.50
Cable: Newnex USB 3.1 (recommended to use with Realsense in some other posts here).
SDK version: 2.45.
Camera Driver version (as seen in Windows Device Manager): 6.1.160.22
I'd greatly appreciate the help.
Thanks.
-
Hi Dmitry Isaev The RealSense UWP driver (6.1.160.22) is not required to use RealSense cameras with Windows. Its purpose is to provide RealSense compatibility with interfaces that use Microsoft's Universal Windows Platform (UWP), such as the Microsoft Hololens headset.
The main RealSense 400 Series drivers are a pair of Depth and RGB drivers that can be found under the Cameras category of the Device Manager.
The frame drops that you are experiencing are a known effect of using depth and RGB at the same time. If you have auto-exposure enabled then you can enforce a constant FPS in the RealSense Viewer by disabling an option in the RGB section of the Viewer's options side-panel called Auto-Exposure Priority, under the section's Controls sub-section. If it is currently On (indicated by a blue box beside it) then it can be set to Off by left-clicking on the blue box to change its color to black.
If you have auto-exposure disabled and are using manual exposure then the FPS speed can be enforced by defining an exposure value that does not exceed an 'upper bound', as described in the link below.
https://github.com/IntelRealSense/librealsense/issues/1957#issuecomment-400715579
Could you test the memory consumption again once a constant FPS is enforced and let me know whether you still observe the suspected memory leak please?
-
Thanks so much for prompt reply!
First, I tried to run with just depth at 1280x720 and 15fps, and still I cannot get a stable framerate, even without RGB.
I also get a weird black spot on the left of the screen, see below. However, when somebody is approaching the camera from that side, the black spot disappears, and object is visible.
Next, here's my configuration when I just use the depth stream:
I will share in the next post screenshots of how I tried to combine RGB and Depth.
-
I note that in the image of the RGB section above, you have RGB Auto-Exposure disabled (black box beside it). Auto-Exposure is not the same as Auto-Exposure Priority (which does need to be set to black to be disabled). If Auto-Exposure is disabled then the FPS can still vary.
Could you left-click on the box beside the color Auto-Exposure to turn it blue (on) please and let me know if a constant FPS is then enforced when Depth and RGB are used together with Auto-Exposure Priority disabled?
In regard to the black spot, this may be an area of the scene that the camera has difficulty analyzing for depth (perhaps due to its surface color) but the approaching person is easier for the camera to read depth detail from. Would it be possible to provide an RGB image of the scene when the black area is present so that I can see the object that is in that area of the scene?
-
MartyG, here's the configuration I just tried, but with no luck:
-
I ran depth + RGB tests in version 2.45.0 of the SDK (the one that you are using) and found it far more difficult to generate a frame drop by doing so than in previous SDK versions, whether Auto-Exposure Priority was enabled or not. So this may be why disabling Auto-Exposure Priority does not make a noticable difference now.
However, using the Frame Drops per Second meter may not be the best way of measuring FPS stability. A more reliable metric is to enable the stream information display for each individual stream. This can be done by left-clicking on the 'i' option at the top of each stream panel.
Information about how the Frame Drops per Second display is calculated is provided in the link below.
https://github.com/IntelRealSense/librealsense/issues/7488#issuecomment-704124850
It states "Frame drops are more strict with regards to latency. If frames do not arrive at regular intervals, it will be reflected as drops but not impact FPS".
-
MartyG thanks for replying!
I just did an experiment - I set the resolution to 640x480 and 15 FPS in both streams, and clicked the 'i' button.
It shows me FPS of 14.99 - 15.03 almost all the time, BUT: on the screen I see evident lags. I tried to check whether it's just visualization lags and tried to record through RealSense Viewer, but then when I play the recordings - the lags persist.
so in my experience - when the Frame Drops Per second visualization shows '0' - the video flow is smooth, when it's randomly lagging - I see the lags on the video too.
-
MartyG In the thread you mentioned there's your comment:
"I would therefore recommend using FPS and the visual feedback of the stream on-screen as a primary benchmark of the performance stability of active streams."...
so indeed there's real lag, not just visualization glitch...
-
Next, could you try disabling the GLSL options in the Viewer's settings using the instructions in the link below to see whether it makes a positive difference to performance.
https://github.com/IntelRealSense/librealsense/issues/8110#issuecomment-754705023
-
I Disabled both "GLSL for Rendering" and "GLSL for Processing" (before that it was only "Processing" on - I tried already playing with it). It did not help with framerate, but it helped with the memory issues.
Now RealSense Viewer does not eat up memory increasingly.
However, sometimes started getting the following messages:
'types.h:1044 USB SCP Overflow'
'notifications.cpp: 514 USB SCP Overflow'
and now just got a bunch of:
'synthetic-stream.cpp:48 Exception was thrown during user processing callback: Out of frame resources!'
-
MartyG still laggy.
I recorded just depth, which has lags, and shared it here to demonstrate what I'm talking about:
https://duke.box.com/s/5y1v74zwugxxak4lnd38y8odijtgpsan
(look at the hand closer to the end of the video).
thanks!
-
MartyG - I have to switch to another project now, and will reply to next message tomorrow. Thank you so much for your help, I will continue replying to your messages and trying recommendations on-line tomorrow. Hope we will be able to solve it.
Thanks,
Dmitry
-
Hi MartyG, I'm back online, ready to resume troubleshooting. Thanks!
-
Thank you Dmitry Isaev So to recep, the main remaining issue is the lag, please?
-
I recall that you are not experiencing variations in the FPS and that it is holding steady at 15, but the image is lagging. And the lag is apparent on playback of the recording.
It may be useful to confirm whether the lag is only occurring in the RealSense Viewer. Could you run the rs-capture tool please and wave your arm in front of the camera? rs-capture should show both depth and color as separate panels like the 2D mode of the Viewer does, so although you cannot record a bag file with this tool, you should be able to gauge whether there is apparent hiccups in the real-time stream.
You can find rs-capture by right-clicking on the RealSense Viewer launch icon on your Windows desktop and selecting the Open file location menu option, which will take you to the SDK's tools folder where the Viewer and pre-built executable example programs are located. Plug in the camera and double-leftclick on rs-capture.exe to launch it.
-
MartyG I tried rs-capture. In general it seems that it's a bit better than Realsense Viewer, but the lags persist. You can notice it in the video I attach below.
Also, the framerate is varying as can be seen from the rs-capture output.
-
CPU and memory usage in the Task Manager looks low in your video, so it may not be an issue related to the computer hardware that you are using.
I notice that there are often unusual values in the Exposure setting in your Viewer images, with values such as '400' and '1560' set on the slider. Whilst this may not have an impact if Auto-Exposure is enabled, I wonder if you are aiming to set the Setpoint value for the Auto-Exposure ROI instead (the Setpoint being the average intensity at which the ROI tries to maintain pixels). If so, that value should be defined in the AE Controls sub-category of the Advanced Controls.
If you are recording with the Graphics Command Center though, that in itself could contribute to overall lag on the computer, as external video capture tools can impose an additional burden on top of the burden of the RealSense program itself.
-
MartyG , no these strange numbers were from time when you suggested to remove auto-exposure priority and explicitly set the exposure time for RGB to ensure the framerate is achieved.
now I use auto-exposure all the time.
The lags persist even when I exit the Graphics Command Center.
it's interesting that rs-capture requires so little amount of resources, yet still gives the lags...
-
Could you unplug the USB cable from the side of the camera at the micro-end of the cable, turn the connector upside down and re-insert it into the side of the camera please (USB-C cables are two-way insertion at the micro end), and test if there is any difference.
It may also be worth launching the Camera app of Windows that is designed for ordinary cameras and see whether its RGB image has lag. That could add weight to the possibility of a USB-related issue if there is lag and the recording app or RealSense are not running at the time. You can find the Camera app by typing camera into the text box at the bottom of your Windows screen, next to the button in the corner where you shut down the PC.
-
so I tried the Camera app in both positions of USB. RGB image in this app has no lags.
(as well as it does not have any lags if I select just RGB in Realsense Viewer).
when I changed the cable - just qualitatively it seemed that in rs-capture the lags became worse, but I'm not sure how big we should believe that - after a while it got back to 30 and 30 fps, then dropped again, then back...
-
MartyG I have another hypothesis... it seems to me that when the FPS performance of the camera gets lower in rs-capture, simultaneously other processes in background sometimes become more active (from what I see in task manager). Particularly "Desktop Windows Manager", "Client Server Runtime Process" and "windows driver foundation - user mode driver framework"...
also, for some reason, rs-capture does not give stable CPU and GPU resources consumption (which I would assume if the camera wants to provide a stable picture in regular intervals)...
-
MartyG after about 8 mins of rs-capture it is just a bit warm. For sure it's not "hot".
-
MartyG could you please point me to which graphic drivers I should update? I assumed I installed the latest drivers from Dell/Intel websites...
is there some special OpenGL drivers?
I'd appreciate some guidance on where to look for updated drivers.
Please sign in to leave a comment.
Comments
48 comments