SR300 depth detection method and night performance
What is the method of depth calculation in sr300? by using emit and receive the infrared light or by using the two camera lenses as stereo (since it has two camera lenses)?
Is it possible to get infrared video output instead of normal video output and depth map output? does sr300 work in darkness and night?
Actually we need to generate infrared data which will be collected at night (darkness).
-
Hi Lghasemz The F200 and SR300 use a pattern projection method called coded light, also known as structured light, and are not stereo like the 400 Series cameras are. The technology is similar to the original version of Microsoft Kinect.
Yes, the F200 and SR300 have an infrared stream from a single IR sensor (unlike the 400 Series, which has a pair of left and right IR sensors, hence the term Stereo).
The SR300 and SR305 can operate in zero 'lux' illumination (darkness). This is because infrared light is projected onto the scene from the camera. This is confirmed in the store listing for the SR305 (which is essentially an SR300 in an updated casing that makes the camera easier to mount).
https://store.intelrealsense.com/buy-intel-realsense-depth-camera-sr305.html
A data sheet document was not created for the F200 (the original RealSense model) but as the technology of the F200 and SR300 are closely related, with SR300 being F200's direct successor and having back-compatibility, I would assume that F200 has the same zero-lux operating capability,
Please bear in mind that the F200, SR300 and SR305 have a depth sensing range of around 1.5 meters compared to the 10 meter range of the 400 Series stereo cameras, so this may affect whether an F200 or SR300 / SR305 is suitable for your project.
It should also be emphasised that the F200 and SR300 no longer receive updates from Intel, and the F200 is not supported in the modern RealSense SDK 2.0 software (the SR300 / SR305 is supported).
-
Hello,
Thank you for the answer but it is not completely what I wanted to know.
I don't need depth video, I don't need general digital video stream output. I need to record an infrared video in darkness/night.
Assume that I am in a dark room without any light, I want to use camera and see what objects are in the room, I want to see things and people in darkness as an infrared video like what security infrared cameras do.
Can I have such these video stream output from realsense SR300 or F200? If yes, how can I activate it or how can I achieve this option of camera?
I will appreciate if you help me through this issue.
-
The situation that you describe sounds straightforward to implement. Yes, the SR300 can display a live infrared stream. In regard to recording, you have a couple of options.
- Use the RealSense Viewer software to generate the infrared stream and record it to a type of file called a 'bag' that is like a video containing camera data. The Viewer has an in-built recorder for doing this. So in theory, you already have everything that you need provided in a pre-made package. However, bag files can take up a lot of gigabytes of storage space, so you would need a computer with a large-capacity hard drive if you are recording the stream.
- Alternatively, generate the infrared stream with the RealSense Viewer and run a video capture program in the background that captures the screen contents and records it to a conventional video file such as an AVI, MPEG, etc.
Below is a test image that I just created of the SR300 displaying its infrared stream in 640x480 resolution (the best that is available on this camera model). If you need a higher resolution such as 840x480 or 1280x720 then you should use a 400 Series camera. The larger the resolution, the larger the file size of the recordings will be. So the 640x480 of the SR300 could work well if you are doing multi-hour long recordings.
-
Thank you very much, This is the exactly thing that I am looking for :)
Using the realsense viewer I can see just two streams, depth and general streams. How can I activate the infrared stream?
I have attached a photo down ,but of course this is in daylight, I will try it in darkness when I realize how to open the Infrared stream window.
-
Expand open the Coded Light Depth Sensor options and left-click on the boxes beside the Depth and the Infrared 1 options. You can toggle the boxes between blue color (on) and black color (off). Turn the Depth option's box to black and the Infrared 1 option's box to blue, and then activate the stream. Only the infrared stream will be displayed.
-
Hello,
I have a question regarding possibility of recording the both infrared and RGB streams at the same time. Is it possible?
I run both streams at the same time as shown in screenshot below, and press the record bottom. but at the end there is just one '.bag' file created.
Since I was not able to open the file I did not understand if the single .bag file contains both recorded streams or not. does it contain both recorded videos?
and at the end, to be able to open the .bag file is it necessary to install ROS? or there is another way to see the recorded and saved video?
Thank you
-
Yes, any stream that is activated in the RealSense Viewer when its record button is clicked will be included in the bag file. You can play back a recorded bag file in the Viewer simply by drag and dropping the file into the center panel of the Viewer (or by clicking the Add Source button and selecting the 'Load Recorded Sequence' option). So you do not need ROS to view the bag.
Below is an image from a test bag that I just recorded with an SR300, showing RGB and infrared together.
-
I prefer Python.
Let me explain what I want to do exactly.
I am implementing these codes:
https://github.com/vardanagarwal/Proctoring-AI
I specially concentrate on the eye tracking part:
https://towardsdatascience.com/real-time-eye-tracking-using-opencv-and-dlib-b504ca724ac6
I tried these codes with both laptop webcam (by inserting '0' in line 168) and RGB camera of SR300 (by inserting '2' in line 168), they work well. but the thing is that I want to apply the method/code on infrared video streams not RGB. I wasn't successful to use infrared stream of SR300 within the code. I mean applying eye tracking code on infrared stream of SR300 in REAL TIME. But I think there should be a solution to this issue, please let me know if there is any.
Since I didn't find solution, I decided to record and save the infrared output of SR300 and then use it with the code later (not real time).
-
If you are seeking information about reading data from a bag and applying post-processing to it, Intel provide a Python tutorial here:
https://github.com/IntelRealSense/librealsense/blob/jupyter/notebooks/depth_filters.ipynb
I do not think there are any further resources that I can add to the research paper about SR300 and gaze tracking that I linked to in your earlier case.
https://support.intelrealsense.com/hc/en-us/community/posts/360049101974/comments/360012965013
-
You can reduce the bag file size by using a lower resolution. For example, I recently did a test where I recorded a 10 second bag at 1280x720 and 848x480. The 1280x720 one was about 287 mb in size and the 848x480 one was 133 mb. So for a 4-hour recording you should use a very large capacity hard drive to write it to (e.g 1 terabyte capacity). Such drives are very low-priced now though. Google for 1 tb hard drive for examples.
You could alternatively record the bag in ROS instead of in the RealSense SDK. This would provide the ability to use a split command where you can specify a maximum file size or duration for a bag. When the user-defined maximum is reached, the current recording closes and a new bag file is automatically started. So you can split a recording into multiple sequential bags instead of one long giant-sized one.
https://github.com/IntelRealSense/librealsense/issues/2424#issuecomment-423284173
-
Hello again,
Using drives isn't a suitable solution for companies, since it is forbidden, also I don't want reduce the the quality of steams because our model would be affected by low resolution.
Thank you for the useful info.
Another question:
We have tested the infrared stream of SR300 with different sunglasses, but the result is a bit confusing.
We had several polarized and non-polarized sunglasses, during the tests we figured out that IR can pass through some of the non-polarized as well as some polarized ones not all of polarized or non-polarized. We were not able to figure out the general rule or exact classification.
I have attached some samples down.
non-polarized:
Polarized:
Do have any idea?
-
I would recommend treating it simply as a situation where glare is generated by the reflectivity of the lenses. A linear polarizer filter applied over the camera lens may significantly reduce the negative effects of glare.
The link below provides a range of useful information about use of physical filters with RealSense:
https://github.com/IntelRealSense/librealsense/issues/7384#issuecomment-696234935
-
I think I didn’t explained my question well 😊
We want the camera to show us the eyes even when the person wearing sunglasses, making sunglasses like transparent to be able to detect eyes. Why the infrared treats some of those sunglasses not all of them? The glare appears on sunglasses isn’t our concern right now.just look at the first two photos (first two rows) I attached, at the left there are the RGB stream versions of the person wearing sunglasses and at the right there are IR version of the same person wearing the same sunglasses shown at the left. In the first row in both RGB and IR streams we can't see eyes. but ın the second row, for the RGB we can't see eyes but for the that glass in IR we can see the eyes.
Is infrared capable of making ALL types of sunglasses transparent. If yes, why it wasn’t able to do it for some of the sunglasses I sent. If not why it made some sunglasses transparent? -
I considered your question very carefully but it involves specialist IR knowledge that agrunnet the Chief Technology Officer for the RealSense Group at Intel is better able to answer. I recommend reposting your question at the RealSense GitHub forum and including the nametag @agrunnet in your message.
https://github.com/IntelRealSense/librealsense/issues
-
Hi Lghasemz The SR300 / SR305 has a single infrared stream, whilst the 400 Series cameras have a pair of left and right streams (which is why the 400 Series is called a Stereo depth camera).
When an SR300 / SR305 is connected to the RealSense Viewer program, its single infrared stream has Y8 and Y16 modes available. Y8 is the most commonly used. It can display 640x480 resolution
-
Hello Marty,
There are 4 open issues that we have not find certain and valid solution for them.
1) Eliminating the glare points on glass/sunglasses:
As your last suggestion we can use an external iluminator/IR source or physical filters over the IR/lazer emitter.
In case of using external illuminator/IR source we are not sure that we can get a desired output for our algorithm, we do not know how the stream will be. Is there any demo or resource that we can refer to it? and we are not sure that putting the hard-filter over the emitter would resolve the glare completely!
2) To be able to transparent all the sunglasses:
You suggest to use wavelength of 940nm, but you do not have any product with that requirement, as far as I know. and is it guaranty that 940nm will work for all sunglasses?
3) low quality IR stream:
we have not discuss about this issue before.
As you know we have collected a dataset of both IR and RGB at the same time, the quality of RGB is acceptable but the IR is not enough at all (we set the resolution of 640*480 for both!!). our algorithm can not detect eyes even faces in IR stream. It seems distance is far or the quality of IR is not enough. we want clear and high quality IR stream. regarding this issue I have sent yo some screenshot/photos via email.
4) IR different performance at daylight and different weather conditions (cloudy, sunny ...):
We have not discuss this issue before as well.
We decided to work on IR rather than RGB because IR does not affected by light condition/direction. We also tested it before data collection and the result was positive, but during the data collection we faced several false reactions. Why this happened? How we can resolve it? as far as I know IR should not get affected by light. regarding this issue I have sent you some photos via email.
Thank you very much
-
I tested all of the points above extensively and found that the simplest solution seemed to be to raise the gain value. Doing so causes a RealSense IR image to brighten. Doing so can also reduce depth detail, but that is not a problem in this case as depth is not being used.
My tests found that although skin was significantly brightened, the eye pupils remained dark on the image. This contrast may make it easier for your algorithm to track the pupil, no matter the resolution or distance.
I tested the method with ordinary spectacles and sunglasses that I managed to find. Gain increase worked fine for ordinary glasses but sunglasses still reflected.
-
Gain is an electronic amplification of a signal, such as the signal from a camera sensor.
I should first emphasize that I tested the gain method on a RealSense D415, as it has a gain control in the RealSense Viewer and so is the easiest way to test. I performed further tests and checks of the SDK code, and it appears that changing infrared gain is not an option that is available on the SR300 model.
My sunglasses image above supports the opinion by the RealSense CTO (agrunnet) that the problem seems to be with how sunglasses manufacturers handle light and transparency, rather than a problem with the RealSense cameras.
https://github.com/IntelRealSense/librealsense/issues/7421#issuecomment-700038238
https://github.com/IntelRealSense/librealsense/issues/7421#issuecomment-704326862
So even if an external filter can eliminate glare from the sunglasses lenses, it may not guarantee that the eyes behind the glasses are shown, depending on how the lenses have been designed.
As I said in another message, is there any possibility that you could track the orientation of the visible nose-tip instead, such as whether it is looking to the side (indicating inattentiveness during driving) or dropped down from straight-ahead (indicating fatigue)
-
Thanks
I think I found sth similar to gain in SR300, that is 'Motion Range'. I will send some photos regarding this issue via email for your information.
What is the range of IR camera of D455, D435I/D435 and D415?
There is a range parameter in the link you have sent us before regarding 'Camera Compare', but I think this is not the thing we are looking for.
reminding that we need clear,close and obvious IR stream.
Please sign in to leave a comment.
Comments
56 comments