Stability issues with mulitple camera setup
|
We are using a setup with 12 D415 cameras connected to a single PC (CPU i7, RAM 16GB) with the provided usb cables connecting directly to USB 3.0 ports on the system(no. of ports expanded via cards). We had an existing setup with different depth cameras which was working fine. With the current setup (using OpenNI2 wrappers for minimal code changes) I am facing multiple issues: (1) While enumerating devices, sometimes it fails to list all devices during a re-run of the application. There was only one instance where I was able to get the output as before this setup. So it is not a case of code correctness, rather whether is it optimal for this setup or issues related to the cameras and this multi cam setup itself. |
-
Problems with all cameras in a multiple setup being detected by enumeration have been reported by other users, so it is probably not an issue specific to OpenNI2. In those other cases, it typically occurs after a soft reboot rather than a program shutdown and restart.
Since you only use 4 cameras at a time and have an i7, your PC hardware should be able to cope fine with that if it is a full PC and not a single-board computing device like a Raspberry Pi.
A solution a couple of users tried was to incorporate a wake alarm to reset USB ports if cameras are not detected on enumeration. Though this approach would only work for a Linux-based setup.
https://github.com/IntelRealSense/librealsense/issues/1615#issuecomment-501050619
-
Okay. Good to know that the OpenNI2 wrapper isn't creating any issues. In our case, we are observing the enumeration issue mostly with program shutdown and restart and less frequently with a system reboot. We are using a Windows 10 system so the fix in that link wouldn't help in our case.
Also about the streaming issue, yes, we are using a full PC. I have tried capturing the depth frames from 4 cameras at a time using OpenMP parallel threads as well as sequentially. The code gets stuck in both cases and only rarely gets through. I tried reducing the bandwidth by changing the frame resolution and fps, but that didn't seem to help. Could it be a cable issue? Even though we are using all the original cables for the 12 cameras? Is there a better way to debug/get more insight into this(data bandwidth/power/cable)?
Thanks again for the help.
-
You are very welcome. :)
Although it is not impossible for USB-C cables to have quality issues, in general using the supplied cables should minimize the chances of problems being related to the cabling.
If you try the cameras in the RealSense Viewer software then that is a good way to find a problem during streaming, as error messages may appear in the log window that can give very useful information for Intel support team members to reach a diagnosis.
The RealSense Viewer is also an easy way to determine if the cameras are being mis-recognized as running in the much more limited USB 2 mode instead of USB 3. If the camera is being mistakenly detected as being on a USB 2 connection then the camera name in the Viewer;s side panel will say "Intel RealSense USB2" instead of referring to the camera as a D415.
May I also confirm that you are not using the cameras outdoors, please? I ask this because there is a known issue where the camera's depth can freeze if the camera is pointed towards the sky, due to the IR sensor becoming saturated (though it is typically related to USB2 connections).
Also, are you using sync cables to link the cameras together please, or are they unsyched. The reason for this question is that if a long sync cable (longer than 3 ft) is used then the camera's counters can reset if an electrostatic discharge occurs unless ESD protection components are built into the sync cable. On the D415, an ESD event has the added effect that it may cause the RGB stream to freeze.
-
We are using the supplied cables as of now, but we plan to use longer cables with a max length of 16ft(~5m) going forward. I wanted to make sure that there are no issues with this present setup first. Also while choosing longer cables do you think these https://www.ntcdistributing.com/usb-3-1-type-c/a-to-c/usb-3-1-a-male-to-c-male-cable/ would work?
I have tried viewing the auto detected usb connection type in Realsense Viewer. They all show up as USB 3.1.
We are not using the cameras outdoors.
Hardware syncing is planned but we aren't using that presently.
A little more info on the errors:
While our program is running, I see this following error constantly
Exception thrown at 0x76B345A2 in ourprog.exe: Microsoft C++ exception: librealsense::windows_backend_exception at memory location 0x19E8E394.
and this sometimes
Exception thrown at 0x76B345A2 (KernelBase.dll) in ourprog.exe: WinRT originate error - 0xC00D36B3 : 'The stream number provided was invalid.'.
It doesn't break at these two exceptions but is it something to be concerned about? This is with the OpenNI2 devices 'created' and regardless whether they are streaming or not.
-
I could not find anything definitively useful on the memory location error, though there was the suggestion that it might occur if OpenNI2 is trying to call an image when the camera is not active (which may be consistent with your report of the error continually generating even when the stream is not started - though I note it also occurs when the stream is running).
In regard to the WinRT error, a few other people have encountered this and offered their code fixes - see the comment linked to below and the comments below it.
https://github.com/IntelRealSense/librealsense/issues/1837#issuecomment-417662311
If you plan to use a 5m cable, it should ideally be a high quality one. High-grade (industrial grade) cables that I have seen of this length tend to be around $45 USD. Newnex cables like the one that you linked to are industrial-grade and proven to work with the 400 Series cameras, so they are usually my primary recommendation for cables based on that proof.
https://youtube.com/watch?v=GLQgR1jT04M
Here is an alternative example of a high-grade cable:
https://www.amazon.co.uk/Tether-Tools-USB-USB-C-orange/dp/B0794B1SDR
Alternatively, range can be extended by constructing an Ethernet network for the cameras such as the EtherSense project.
https://github.com/krejov100/EtherSense
And the new FRAMOS D435e industrial RealSense camera uses ethernet instead of USB as its primary means of data communication.
-
I tried upgrading to the Newnex cables. With this new setup I'm observing frequent connect/disconnect with the cameras. In the Realsense viewer app, the connection shows up as USB 3.2, so I believe the cable quality can be ruled out. Is that correct?
I also want to point out that we're using 3 of these PCI USB cards https://www.amazon.com/High-Point-RocketU-1144A-PCI-Express/dp/B005EWTJPG
with an ASUS x99-Deluxe motherboard with a 1200W supply. We're using 3 ports on each of the cards so as to not exhaust it and three from the motherboard. I believe these should be okay, right?
I have uploaded two logs from the realsense viewer (the second (larger) and more useful being post-reboot). I generated the log using the RealSense Viewer being run directly after boot, so that our app wouldn't create any issues with the stability/log process. https://gofile.io/?c=dZVqUg
So what could be causing these issues?
-
It does sound as though the USB cable is fine. A Newnex cable should certainly minimize the risk of quality-related cable problems occurring.
Using a high-wattage PSU does not guarantee against disconnections, unfortunately. A disconnection may still occur if a power instability occurs at the USB port. When using internal USB expansion cards, it would still be a "passive" USB connection (one that draws its power from the PSU). Greater stability can be achieved with an "active" USB connection (using USB hubs powered externally by mains electricity instead of the PC). The USB standard allows up to 5 USB hubs to be chained on one PC (known as '5 deep').
Technically it is possible in the USB standard to link 127 USB devices together in a 'daisy chain' on the same computer. In practice though, the resources available on the computer to run the devices is a limitation. As you are only activating 4 cameras at a time from the batch of 12 cameras, it should be well within your i7 PC's capabilities though.
I see in your log file that the error 'try_fetch_usb_device(...) failed is occurring extremely frequently. Another RealSense user who encountered this error attributed it to the USB bus not resetting properly when closing the camera stream. That user was advised to combine a hardware_reset instruction with the rs2::device_hub instruction to get a synchronous flow.
https://github.com/IntelRealSense/librealsense/issues/1086#issuecomment-361300887
-
I note that with the above hardware_reset method, it may not be ideal if using more than one camera. The link below has a script for iterating through all connected cameras when checking which one should be reset.
-
I agree that high-wattage PSU solely won't fix the power instability issue at the USB port, but I wanted to point out that in addition to that, the PCI cards we are using are of high quality and that we had this setup for other depth cameras before. Do you believe these cards with the motherboard in combination are lacking something? or that power sensitivity is an issue with the Intel cameras especially with this mulit-cam setup?
I am considering two other options when it comes to an "active" setup.
1. Powered PCI USB card like https://www.amazon.com/dp/B071DFQ6TW/ref=psdc_229185_t3_B00J8Z3YYW. Do you think it would make a difference?
2. Hubs for e.g. like https://www.transcend-info.com/Products/No-402 ? I am worried if these could have bandwidth issues and we would have to go for an industrial grade one, which given our setup would be quite expensive.
Also, in that log, I hadn't started streaming from any camera. It was just with opening the viewer app and then closing it after a while after seeing muliple warning/error notifications. So, the fix there is useful and I'll incorporate that in the future, but it doesn't seem to help now.
-
I cannot speak for other brands of depth camera, but RealSense cameras are sensitive to the state of the USB port due to the large amount of data that the cameras transmit along the USB cable.
The 400 Series cameras are extremely flexible with the hardware that they will work with. They can operate with any Intel or ARM processor. So it is unlikely to be an issue with the PC motherboard.
I re-read through your original message. I wonder if the OpenNI2 wrapper may be a weak link in your setup, as I have heard before of issues related to it. The discussion below may be interesting to you:
-
Okay, that makes sense. So getting the port quality up to par would be extremely important in that case. Could you comment on the two other "active" options I'm considering?
I have been considering using the Realsense API directly, but in this case the connect/disconnect issues are occuring without our app being run, which the logs show. I would be using the API eventually as it allows for finer access, but now I want to fix these issues first.
Thanks again for the quick replies!
-
You're very welcome. :)
Of the two "active" options you referenced, the first one looks like it is still a 'passive' system, since like the cards you have now, it is drawing its power from the PC PSU instead of mains electricity,
Regarding the second option, a powered hub, expensive is not necessarily better where RealSense and hubs are concerned. I have seen a small number of reports of problems with modern USB 3.2 hubs, including industrial grade ones, whereas USB 3.1 hubs worked fine with RealSense on the same computer. In Intel's multiple camera paper, they tested with an AmazonBasics powered 4-port hub and provided a link to the model that they used.
-
We are trying now with the newnex cables and using powered usb3 hubs(4 Transcend hubs with 3 cams connected and leaving one port free on each). The system is a lot better now, as in very less connect/disconnect issues. I was wondering if you have a suggestion for a cleaner/minimal setup.
Now when it comes to our application, I am seeing intermittent failure with the driver(realsense and rs2driver dlls for OpenNI2) with what seems like a heap corruption. I'm using the release driver built from librealsense2 SDK 2.22(which is part of the last know BKM config). Any idea what could be causing this?
-
You could reduce the complexity by using a powered hub with 12 ports instead of 4. You can see what is available by searching stores such as Amazon for 'powered USB 3.0 hub 12 ports'. Given the proven reliability of an AmazonBasics powered hub in Intel tests in the multiple camera white paper, their 10-port model may be of interest.
https://www.amazon.co.uk/AmazonBasics-USB-10-Port-Power-Adapter/dp/B076YFNTRS
You could also reduce the size of the setup by using an Intel NUC mini PC, which are very small and available in a range of different configurations from budget processor to i7. Last year Intel released a very powerful model for around $1000 called NUC 8 VR. There are typically additional costs with NUC kits for supplying your own OS, memory and storage.
In regard to ntdll.dll errors, I have never encountered them before. The link below provides some advice.
https://www.lifewire.com/how-to-fix-ntdll-dll-errors-2624474
-
Okay. I'll look into those options. I'm aware of the 10-port hub, but I'm concerned whether the single hub will be able to handle the bandwidth. Also power stability of port is what I think was the culprit(after using the better cables of course).
Regarding the crash, I've been seeing that on 2 separate systems. So many of the fixes which are listed there wouldn't apply in our case. I believe it is some kind of heap corruption which occurs intermittently when the driver calls to free up memory. Is there more debug info which could help? This is an urgent issue which we are trying to resolve, so maybe you could involve the driver developer folks as well?
-
If there were a bug in the RealSense DLLs, how long it would take to release a fix would depend on the complexity of the problem. It would take longer if the fix had to be incorporated into the firmware driver.
Debug tools can be found here:
https://github.com/IntelRealSense/librealsense/tree/master/tools#debug-tools
With the firmware logger tool, the log file generated has to be submitted to Intel support staff for analysis (for example, by attaching it to a forum comment) as the end-user does not have the tool to read its output.
I do not have the ability to refer cases to the developers. I recommend opening a case on the GitHub forum to do that.
-
If I'm logging firmware, it has to be alongside our app, so that we can see the sequence of events. Also since OpenNi2 wrapper is being used, it would be necessary to check if everything is alright there. Is there a way to do that?
Also I've been trying to get the realsense logger to work. But I'm not sure what I'm doing wrong. I have set up the variable LRS_LOG_LEVEL= DEBUG but I don't see any logs?
-
I have filed the driver crash issue on github. https://github.com/IntelRealSense/librealsense/issues/5076
I still haven't got a response. The issue is critical to us. Is there a way to escalate it further? Can you help us get in touch with someone who could take this forward? Thanks again.
Please sign in to leave a comment.
Comments
24 comments