External Triggering on D415 with External HW
Hi everyone,
I am using a setup that currently runs with 5 D415 cameras in streaming. There is no need for streaming, and I could do just fine with getting 1 frame from a camera when I send a command.
I read this post and this document. I don't need any synchronization between cameras, each one will be triggered separately at different times. This will enable me to add as many cameras as possible to the setup without bandwidth issues.
So, I have several questions:
1. Besides setting all the cameras to salve mode and connecting to the 9pin connector. Is there a configuration I need to change to disable streaming and get only frames when I want?
2. If I don't care about syncing just to "press the shutter", do I still need to find the real frame rate of cameras?
3. If yes, will I have to do that for every camera? or if they have the same configuration, doing it to one f them will be enough.
4. HW wise, any recommendation on how to send this 1.8v trigger command to the cameras directly from a pc? or will I need some kind of I/O module to do that?
-
Hi Eli Dejou If you do not need to sync the cameras then you could not use hardware synchronization at all and instead use the D415 script at the link below that takes a single capture from all attached cameras simultanoeusly.
https://github.com/IntelRealSense/librealsense/issues/2219#issuecomment-412899468
In regard to your questions:
1. If you use the genlock form of hardware sync then if you set the slave cameras to Inter Cam Sync Mode '4' instead of '2', the cameras will wait indefinitely to receive a trigger signal and only capture when a trigger is received.
Setting up a genlock slave is more complicated then a mode-2 slave and the method is no longer recommended by Intel. If you need to use it though then a PDF guide to genlock sync can be downloaded here:
https://github.com/IntelRealSense/librealsense/issues/10926#issuecomment-1351603500
2. If using slave mode '2' then you need to configure the externally generated trigger pulse to very closely match the actual FPS of the camera (such as '30.02'), as described in the section of Intel's multiple camera guide linked to below,
https://dev.intelrealsense.com/docs/multiple-depth-cameras-configuration#e-external-trigger
The above guide is Intel's recommended method of hardware sync, as it is mature and validated by Intel. Unlike slave mode '4' though, mode '2' does not wait indefinitely for a trigger but instead only listens for a trigger for a certain period of time on each frame and then automatically initiates an unsynched capture. So you would be receiving many more frames than you desire.
3. A single externally generated trigger signal should be applicable to all slave cameras.
4. The genlock PDF paper describes using a component called a level shifter component (also known as a voltage shifter) to configure the voltage in conjunction with the signal generator. RealSense users have used level shifter similar to the one at the link below.
-
Hi MartyG,
Thank you so much for the detailed answer!
If you do not need to sync the cameras, then you could not use hardware synchronization at all and instead use the D415 script at the link below that takes a single capture from all attached cameras simultaneously.
The thing is I need to trigger every camera separately in different times. Triggering them at the same time will again limit the amount of cameras I can use due to bandwidth.
If you use the genlock form of hardware sync then if you set the slave cameras to Inter Cam Sync Mode '4' instead of '2', the cameras will wait indefinitely to receive a trigger signal and only capture when a trigger is received.
Do you know what was the reason for intel not recommending this? In addition. I understand that in mode 4 I dont need to find the exact frame rate. right?
-
Yes, with mode 4 it is much easier to match up an externally generated trigger to the slave camera FPS.
The genlock sync system was an experimental proof of concept that did not always work as described in the guide document. Mode 2, which is part of the original sync system that came before the genlock system, is considered by Intel to be the official method of hardware sync that is lab-validated and recommended. The genlock sync guide was therefore withdrawn from online availability, though the archived PDF copy preserves the contents of that guide for those who need it.
-
If you are aiming to only capture 1 frame per camera at occasional time periods and not be capturing continuously then 5 simultaneous cameras is not likely to place a significant processing burden on the computer.
A typical USB hub has a maximum bandwidth of 5000 Mbps / 5 Gbps that is shared out between all devices attached to the hub. So if there is a possibility of exceeding that bandwidth cap with the 5 cameras, you could consider putting some of the cameras on a second hub, as the USB standard allows a chain of up to 5 hubs.
-
Currently, I have 5 Cameras but in future architecture, I will have 10 or even 12. This change is the reason I am trying to switch from streaming mode to triggering mode. Triggering all of the 12 cameras at once will stumble upon bandwidth issues, like streaming. So, my goal is to be able to work with 12 cameras with each camera triggered separately at different times enabling me to use the same bandwidth.
-
The L515 lidar depth model's multiple camera system staggers the capture of multiple cameras. The L515 is now a retired model though, so may not be available for purchase in future as there are limited stocks remaining.
https://dev.intelrealsense.com/docs/lidar-camera-l515-multi-camera-setup
Alternatively, using 6 FPS mode on D415 instead of 30 FPS will reduce the amount of data produced by each camera during capture and so make it easier to scale up.
It is possible that you may need more than one USB hub anyway, as it is recommendable to budget 2 watts power per camera. A typical mains electricity powered USB hub can provide 12 W for all attached cameras, meaning that 2 hubs (6 cameras per hub at 2 W per camera) may be needed to meet the maximum power needs of 12 cameras on the same computer.
Another possibility may be to define a separate pipeline and configuration for each camera, starting a pipeline, capturing, stopping it and then starting the next pipeline until you loop back to the first pipeline and start the cycle again. This approach would be inflexible though as you would have to pre-program the serial number of each individual camera into a script, manually adding further pipelines / configurations as the number of cameras in your project expanded. A Python example of such a script for 2 cameras is at the link below.
https://github.com/IntelRealSense/librealsense/issues/1735#issuecomment-390840337
-
Thanks for the answer.
I am already using 6fps in my cameras, and I want to remain with the D415. The system is industrial and has an industrial USB hub, so no problem on that front.
From what I read and understand I can just have an I/O Module that outputs 1.8v signals. Each output will be connected to a different camera. And I will trigger each camera separately. I am ok with preprogramming the serial numbers. I just don't understand why I would need to open and close pipelines every time. Can't I just trigger the camera to take a frame and that's it?
-
By starting and stopping the unique pipeline of each individual camera, you are effectively toggling the cameras on and off in sequence instead of having all cameras capturing simultaneously.
If you have an I/O module that can switch between outputs and send a trigger to individual cameras then that could work so long as you are using genlock mode 4 (genlock slave) so that each camera does not capture a frameset until it receives a trigger.
-
Thanks MartyG, that's what I would do.
-
MartyG, A few additional questions:
Unlike slave mode '4' though, mode '2' does not wait indefinitely for a trigger but instead only listens for a trigger for a certain period of time on each frame and then automatically initiates an unsynched capture. So you would be receiving many more frames than you desire
1. How long is this certain period of time? is it in seconds or in ms? if it's in 6fps minimum than it would be the same as streaming at 6 fps. No? (Bandwidth wise)
2. Reading this conversation you had. In genlock mode, is the RGB stream continuously transmitting or can it be blocked until triggered? I intend to trigger the camera once every 6 seconds, and I want to ensure that both the RGB and depth camera streams are not transmitting until the trigger is received. I don't require RGB and depth synchronization, I am ok with having a short offset, as the target is static.
3. In the genlock white paper they compare the frame capture delay:
Mode 4: Optical capture starts with Trigger.
Mode 2: Readout (not capture) starts with Trigger.
Can you elaborate on this a bit more?
I saw this figure for genlock, I dont understand how the mode 2 figure looks like
-
Mode 2 listens on every frame, so it would be the same period of time no matter which FPS rate is set.
AS far as I am aware, the precise wait period on each frame has not been defined. The original source of the advice about unsynced capture after a certain period is at the link below.
https://github.com/IntelRealSense/librealsense/issues/2171#issuecomment-409559217
"When you set a camera to slave it will still acquire frames at the right frame rate even if there is no sync input signal. It tries to sync but if there is no signal that comes in at the right time it gives up and triggers itself".
-
Ok I think I understand. If I want to really get a snapshot every few seconds. Genlock is the only way to go. As in mode 2, the slave camera will still stream in the configured fps (synced or not synced). Please correct me if I am wrong
I have edited my previous post to include additional questions before your answer, can you talk a look?. Sorry for the misunderstanding. And thank you so much for the assistance and knowledge sharing.
-
You are very welcome!
You are correct about Mode 2, it will still stream in the configured FPS whether sync is active or not.
In regard to your further questions:
2. RGB is not synced in genlock mode 4+, which only sync depth. Depth and RGB are only synced in Mode 3 (Full Slave). In practice though, it was only supported for the D415 camera model, which has the RGB sensor integrated onto its circuit board. On models where the RGB sensor is attached as a separate component via a cable, such as D435i and D455, sync of RGB in Mode 3 will not work.
3. My understanding of the diagram is that in Mode 4, there is a period of exposure after the trigger and readout does not occur until after exposure has completed, taking about 1 frame. Whilst with Mode 2, there is not this exposure-related delay and readout is immediate after the trigger.
-
Regarding the RGB question. I think we are talking in different terms. You refer to pin 5 as sync and I refer to it as the trigger. As I don't need to sync it to any external camera, I just want to take a snapshot. So I will try to rephrase my question. In genlock mode, will RGB stream continuously or send frame only when triggered (for example once a minute)?
Thanks for answering question 3.
-
In Mode 3 (Full Slave) it will behave like Mode 2. So it will stream continuously instead of waiting for a trigger. It is Genlock mode 4 and above that waits for a trigger, and those modes only sync depth, not RGB.
So you can have depth and RGB enabled at the same time, but if Mode 4 or above is used then only the depth will be synced and waiting for a trigger before capture. RGB will be unsynced and streaming continuously.
-
Thank you again Marty!. Some additional questions:
Hi,
As I said before, we are using D415 cameras in their default mode (mode 1) with a 6 fps frame rate. We have encountered a few issues. Firstly, we are not getting time stamps for the sample frames, and secondly, the left and right depth cameras are not synchronized, leading to distorted images. Same with RGB not timestamps.
We are considering switching to genlock mode, as this post suggests. Thus, I would like to know if this mode (4) can also address the issues we are experiencing. Specifically, I would like to know:
- If we can obtain timestamps for the cameras in genlock mode?
- If the left and right depth cameras will be synchronized in genlock mode?
- I know that the RGB is not synced in genlock but is it possible to get a timestamp?
-
The depth image is not generated from the left and right depth streams 'Infrared' and 'Infrared 2'. Instead, a depth frame is constructed from the raw left and right infrared frames in the camera hardware at the point of capture at the camera lenses, and then the assembled depth frame is sent along the USB cable to the computer.
If the Infrared and Infrared 2 streams are not enabled then the depth stream still functions because its frames are not constructed from these two streams.
You should be able to receive timestamps in Mode 1 or 2 whether hardware sync is enabled or not. If hardware sync is not working then you should still receive the timestamps but they will be unsyched. You can either receive the timestamps by creating scripting to do so, like the example C++ code snippet in the multi-cam sync paper at the link below, or overlay timestamps over the image in the RealSense Viewer by left-clicking on the Show frame metadata button in the top corner of the stream panel.
https://dev.intelrealsense.com/docs/multiple-depth-cameras-configuration#f-system-vs-hw-time
The genlock white-paper guide does not provide guidance in regard to timestamps in genlock mode (4+) and I could find no other references about it. My expectation (not certainty) would be that a timestamp would be generated only at each time that frame capture is initiated when a trigger is received.
3. You should be able to obtain an unsynced RGB timestamp. Again, the Show frame metadata option at the top of the RGB stream's panel in the RealSense Viewer overlays the RGB timestamps.
-
Thank you, Marty, that is very helpful.
The genlock guide states that it is compatible with SDK version 2.34.0 or newer. In addition, your camera firmware driver version should be 5.12.4.0 or newer.
I seem to have firmware driver version 5.14.00 and SDK v2.53.1, but still can't see the genlock mode. Any suggestions?
-
You said earlier in this discussion that "I am trying to change the Inter Cam Sync Mode to 4, but only able to see up to 2". What interface are you attempting to change the value in (such as the RealSense Viewer or with a program script). In the Viewer's Inter Cam Sync Mode setting for example, its slider allows the setting of any value up to a maximum of 260.
-
I checked the genlock sync white-paper document. It states "The Genlock feature only works on Intel RealSense cameras that are based on Global Shutter stereo imagers. This means that it currently works on the D435 and D435i, but not on D415". This is likely the reason why your D415 is not supporting the ability to select an Inter Cam Sync Mode higher than '2', as the D415 is not equipped with Global shutters. It has slower Rolling shutters. I do apologize.
Please sign in to leave a comment.
Comments
26 comments