View my account

Post-processing filters in C: leak and segmentation fault

Comments

35 comments

  • MartyG

    Hi Fabrizio Dini  I should emphasise from the start that my pure C (not C++ or C#) knowledge is limited.  I will do my best to provide some useful insights though.

    Yes, releasing resources is important, otherwise you will likely have a memory leak that leads to performance reduction and then a program crash over a period of time.  The period of time before a leak leads to failure may depend on factors such as the FPS and resolution being used (higher FPS / resolution = faster leakage).  

    As a starting point in investigating your leak, the SDK's C example program for depth may provide useful guidelines for your own program by studding the resource release code block at the base of the script.

    https://github.com/IntelRealSense/librealsense/blob/master/examples/C/depth/rs-depth.c#L235 

     

    Also note the use of rs2_release_frame at several points throughout the script.

    https://github.com/IntelRealSense/librealsense/blob/master/examples/C/depth/rs-depth.c#L162 

    0
    Comment actions Permalink
  • Fabrizio Dini

    HI MartyG, thank you for your response.

     

    I know the rs-depth.c example, it's there that I first looked for info. Unfortunately, it is a too basic example. To be 100% sure, I made a detailed reviewe of my code, and I can say every frame, and every object that gets allocated with libRealSense function has a corresponding "release". So this is not a trivial memory leak, it must have something to do with the internals of the library. Also, let me say that bypassing the librealsense code (i.e. using a simple webcam to get the 8bpp image I need) results in a code that passes a vagrind check (with --leak-check=full) with 0 errors and 0 warnings. So I am fairly sure this problems are somehow related to the librealsense code.

    As I said, I am using a number of post-precessing filters (in order: decimation, depth-to-disparity, spatial, temporal, disparity-to-depth, hole filling, threshold, and colorizer), each pushing processed frames into a dedicated queue. I strongly suspect my issue has something to do with how I use the filters. Valgrind report is filled with warnings and errors I cannot investigate deeply, so I started commenting out the filters and test the software. 

    Reducing the filters chain to: decimate -> colorizer seems to work...

    I'll start adding the filter one by one and see what happens...

     

    0
    Comment actions Permalink
  • MartyG

    Post-processing filters are processed on the computer and not in the camera hardware, so having most or all of them enabled would have an impact on performance.  For example, Dorodnic the RealSense SDK Manager has suggested that the spatial filter could be dropped by RealSense users due to having the largest processing time for a modest improvement.

    https://github.com/IntelRealSense/librealsense/issues/4468#issuecomment-513485662 

    Also, are you using the point cloud processing block in your project, please?

    0
    Comment actions Permalink
  • Fabrizio Dini

    Ah this is very interesting! I couldn't decide by myself what filters may be needed, so I followed a guide on the LibRealSense documentation site that was suggesting this specific order of the filters.

    And no, I am not doing any point-cloud processing in my code. However, on my laptop the code takes around 50% of CPU time. Disabling the filters doesn't seem to make a big difference, but I'll keep Dorodnic advice into account since I am going to port this on a RPI4.

    As I am typing, I am running the code with decimation -> hole filling -> threshold -> colorizer and it has been running for 10 minutes now...

    0
    Comment actions Permalink
  • MartyG

    decimation -> hole filling -> threshold -> colorizer is a good mix of filters.  The threshold filter sets a minimum and maximum for the distances from the camera that depth is rendered at, and excludes depth data from the image that is outside of that range.  So unless you need to restrict how much of the background is shown (for example, to focus on an area of the image near to the camera) then the threshold filter could be left out.  

    0
    Comment actions Permalink
  • Fabrizio Dini

    The threshold filter si very important, I cannot cut it out. Also, I think that will use temporal filter to smooth the depth frame.

    What about depth-to-disparity and disparity-to-depth? do they make a big difference? In the guide that I followed, they were making spatial and temporal filtering between depth-to-disparity and disparity-to-depth... is there any real reason to this order of the filters?

     

    Thanks so much MartyG for your comments!

    0
    Comment actions Permalink
  • MartyG

    I believe that depth_to_disparity converts depth data into a disparity map, and disparity_to_depth converts a disparity map to depth data.  

    https://support.intelrealsense.com/hc/en-us/community/posts/360051896473/comments/360013274934 

    0
    Comment actions Permalink
  • Fabrizio Dini

    Yes, sure, but is this something that has any effect on the data or is it just a different representation?

    I just added the temporal filtering to the chain, it seems to work pretty fine by now. So the current filter chain (for the records) is:

    decimation -> temporal -> hole filling -> threshold -> colorizer

     

    I might remove the hole filling filter by the way, since I am not going to sample any particular pixel on the frame...

     

    edit: very interesting link Marty, I am thinking that probably there's no need to pass through the depth-to-disparity and disparity-to-depth filters...

    Also another question came to my mind: I set all queues for the filters to have size 1. Can this have something to do with my issue? Depending on how the code is written, there could be a time when a queue as 2 or 0 frames in it...

    0
    Comment actions Permalink
  • MartyG

    Dorodnic has also recommended not changing the frame queue from its default of '1' due to the risk of accidentally breaking streams with an incorrect value.  '2' could be used if streaming depth and color at the same time, whilst '1' is fine for a single stream. So if you have set the frame queue to '1' then this would not be a program-breaking value, but you would not need to set it either as 1 is the default.

    0
    Comment actions Permalink
  • Fabrizio Dini

    OK, so let's stick to 1-size queues.

     

    Here something strange is happening. After I added the temporal filter to my chain of filters, I forgot to add back the release of the corresponding frame. Surprisingly, the code run for about 10-15 minutes without any clue of memory leak. When I realized the error, I passed the code through a valigring mem-check: it reports a lot of lost memory (practically any instantiated object is seen as a mem-leak, despite my efforts...) but none of them is related to the frame that comes out from my temporal filter (and is never released).

     

    Now, this question comes to my mind: what happens to frames that are given as input to filters? Do filters get ownership of the frames they are given? Is it correct NOT to release the frame that comes out from the temporal filter? (it is the input of the hole filling filter...)

     

     

    0
    Comment actions Permalink
  • MartyG

    When post-processing alters a frame, the original frame is preserved and a new copy of the frame is created, so you have two versions of the same frame in the frame queue.  As new frames enter the frame queue, old frames should get pushed out of the queue.

    0
    Comment actions Permalink
  • Fabrizio Dini

    And, when they get pushed out of the queue, the memory gets released?

    0
    Comment actions Permalink
  • MartyG

    That is my understanding (increasing the frame queue value increases latency, so frame-loss becomes less likely but memory consumption increases).

    https://dev.intelrealsense.com/docs/frame-management#section-frame-drops-vs-latency 

    0
    Comment actions Permalink
  • Fabrizio Dini

    Marty, sorry for all these questions (and thank you for your answers!), but I have a last one:

     

    Frames returned from a queue by rs2_wait_for_frames are phisically popped from the queue? it should be so, otherwise you could be left with a pointer to memory that could be released by someone else at a future time. So, obviously the processed frames that comes out from a filter of course must be released...

     

    Or, there should be somewhere in the API a method to physically pop frames out of a queue, but I cannot see one.

    0
    Comment actions Permalink
  • MartyG

    wait_for_frames() is a mechanism for blocking until a complete frame arrives.  poll_for_frames() delivers the frames immediately, though the timing of CPU sleep periods has to be managed or the CPU usage percentage can max out.

    https://github.com/IntelRealSense/librealsense/issues/2422#issuecomment-423254709 

    The SDK has a function called Keep() that keeps the frames stored in memory, though it is only suited for recording periods of up to 30 seconds as it progressively consumes the computer's memory capacity as the memory fills up with stored frames.  When the pipeline is closed, batch-operations such as post-processing and align can be performed on the set of frames and then they can be written to file in a single action.

    In C++, releasing frames to free up memory is relatively straightforward and not something that has to be thought about much.  When using C and C# though, making sure that frames get released properly becomes more complicated

    0
    Comment actions Permalink
  • Fabrizio Dini

    Ok, I am still struggling to get this to work. Let's see if I can summarize the general situation.

     

    I have turned off almost all filters. I only use the colorizer filter now, to see if I can ease things by using less components.

     

    On my laptop, I managed to run my code for a few hours without crashes (actually, I had a couple of filters on, but I think that's irrelevant), and with a constant memory footprint.

     

    The same code, ported to a Rasberry Pi 4, crashes in a couple of minutes.

     

    Note, however, that running my code in valgrind on my laptop, shows a huge number of warning and errors (I am using valgrind --leak-check=full). I have many (seven) "Conditional jump or move depends on uninitialised value(s)" and one "Use of uninitialised value of size 8" that have no reference to my code, but are instead related to a thread that I suppose is started in the librealsense library.

     

    I also have two invalid reads that are quite strange. My comprehension of the valgrind output is that rs2_pipeline_stop() causes the deletion of memory that is inside a block later freed by rs2_delete_pipeline().

     

    Then I have a long list of memory leaks. From the valgrind report it seems that every object created from librealsense (context, device list, device, pipeline, config and, of course, post processing filters) allocates memory that is not (properly) freed.

     

    Now, I cannot believe this is (only) due to librealsense bugs, there must be some error in the way I use the librealsense API. Still, I followed the examples, double checked every create and delete operations... and now I am using valgrind. It is know that valgrind is not 100% accurate, but if these messages are false positives, I would suggest Intel to publish a list of "valgrind suppression rules" for developers, like me, that uses valgrind to debug their code.

     

    I have looked for simila issues in this forum and found that I am not the first developer to encounter them, but there has been many bug fixes and version released since then so I expected that to be solved. Is it really so? MartyG do you have any insight about these problem?

     

    Thank you so much for your help!

     

    0
    Comment actions Permalink
  • MartyG

    If you want to check whether there is an issue with your Pi 4 setup, you could install the Pi 4 SD card image of a librealsense installation provided by Intel in the link below.  This can be helpful as a reference build when comparing to a setup that you have done yourself.

    It does not have wrappers such as Python and ROS in the build but may be okay as it is if you are working in C.

    https://dev.intelrealsense.com/docs/open-source-ethernet-networking-for-intel-realsense-depth-cameras#section-2-3-preparing-the-sd-card 

    It is also worth mentioning that if you have problems with pipeline.Stop(), an alternative is to stop the sensor instead.  So you start pipeline and stop sensor, instead of doing both start and stop on the pipeline.

    https://github.com/IntelRealSense/librealsense/issues/7252#issuecomment-695194103 

    0
    Comment actions Permalink
  • Fabrizio Dini

    Thank you MartyG! I will surely try both solutions!

    What about the valgrind suppression list? that would be of great help for developers, removing all valgrind false positives and letting them focus on real problems.

     

    Again thank you for your support!

    0
    Comment actions Permalink
  • MartyG

    You are very welcome   :)

    I cannot fnd a mention in this case of the librealsense version that you are using.  However, there was a bug fix for valgrind in SDK 2.35.2 that is titled "Valgrind: Conditional jump or move depends on uninitialized variable".

    https://github.com/IntelRealSense/librealsense/issues/4332 

    0
    Comment actions Permalink
  • Fabrizio Dini

    Yes, I saw that issue too. I am currently on version 2.40.

    0
    Comment actions Permalink
  • MartyG

    I do not have any further librealsense information about valgrind.  You can define a custom suppression list in valgrind though, according to its documentation.

    https://www.valgrind.org/docs/manual/manual-core.html#manual-core.suppress 

    0
    Comment actions Permalink
  • Fabrizio Dini

    Yes I know, but doing this may be a huge amount of work without knowing the library internals. It would be a nice-to-have. Plus, doing it on my own is possible but not safe, since I cannot know if the errors I am suppressing are actual false positives or not. That is the main reason why librealsense mainterners should do that.

    0
    Comment actions Permalink
  • MartyG

    The best way to proceed may be to create a new post on this forum, asking for a Feature Request about setting up the valgrind feature that you have described above.   It is not something that I myself am able to arrange on this forum.  

    0
    Comment actions Permalink
  • Fabrizio Dini

    Yes, I was thinking something similar MartyG...  that will take a moment, hold on a second...

     

    Here it is: https://support.intelrealsense.com/hc/en-us/community/posts/360052556133-Feature-request-Valgrind-suppression-list-for-librealsense 

    0
    Comment actions Permalink
  • MartyG

    Thanks very much.   I will not be handling that case.  One of the Intel Support team members on this forum should pick it up.  Good luck!

    0
    Comment actions Permalink
  • Fabrizio Dini

    Hello there! A little update...

     

    Since on my laptop I somehow managed to run the application for hours with no apparent leak, I wanted to investigate further my problems on the Raspberry Pi 4, where the application crashes in very few minutes. I have a raspian buster there, with librealsense 2.40 built and installed correctly (samples run just fine or so it seems). To see if I have the same problems on the RPI4 I built the examples in Debug mode and run valgrind on rs-depth example. Strangely, valgrind does not manage to run the example. It is common to have severe slow-down of applications run in valgrind, but it seems the application is stuck. However, killing the app I still can see the  (partial) valgrind report, which shows the following:

     

    At the beginning:

     

    --00:00:00:00.005 3129-- WARNING: Serious error when reading debug info
    --00:00:00:00.005 3129-- When reading debug info from /lib/arm-linux-gnueabihf/ld-2.28.so:
    --00:00:00:00.006 3129-- Ignoring non-Dwarf2/3/4 block in .debug_info
    --00:00:00:00.006 3129-- WARNING: Serious error when reading debug info
    --00:00:00:00.006 3129-- When reading debug info from /lib/arm-linux-gnueabihf/ld-2.28.so:
    --00:00:00:00.006 3129-- Last block truncated in .debug_info; ignoring

     

    and then lots of "Conditional jump or move depends on uninitialised value(s)" and "Use of unintialised value of size 4".

     

    I will test the ready-made image MartyG has linked in the above comments and see if there is any difference

     

     

     

    0
    Comment actions Permalink
  • MartyG

    Thanks very much for your update!

    The Conditional jump or move depends on uninitialised value(s) Valgrind error has been a Known Issue on the SDK's Release Notes since March 2019 and continues to be listed on Known Issues for the most recent SDK version at the time of writing this (2.40.0).  So the RealSense developers are aware of that particular issue but there is not a fix yet.

    https://github.com/IntelRealSense/librealsense/wiki/Release-Notes#known-issues-18

    0
    Comment actions Permalink
  • Fabrizio Dini

    Thank you too MartyG! It's good to know it's a known issue. Hopefully that is not something that could crash an application... I think! Otherwise I should probably experience very early crashes on both architectures. Of course this is something it would be better to fix, but I think my issues might not be related to that valgrind error. 

     

    Also, I found that other errors more related to memory leaks (invalid reads, bytes lost and so on) are only reported when the application closes, so they should not be a real issue during the usual application execution. That makes them less critical. Unfortunately, this can only be verified on my laptop, where I can shut down the application gracefully. As I said before, on the RPI4 even the simple rs2 examples (rs-depth, rs-distance) get stuck when run in valgrind, and I am forced to kill the process. The resulting valgrind report is never complete.

     

    I will keep on testing on the RPI4 and keep this topic up to date.

     

     

    0
    Comment actions Permalink
  • MartyG

    Thanks very much for your ongoing detailed feedback.   Good luck with your tests!

    0
    Comment actions Permalink
  • Fabrizio Dini

    Update: I have tried by using the Raspbian image linked at this page:

     

    https://dev.intelrealsense.com/docs/open-source-ethernet-networking-for-intel-realsense-depth-cameras?_ga=2.261147470.210502128.1607078596-1050659194.1604762783#section-2-3-preparing-the-sd-card 

     

    It seems to be just a usual Raspian 10 (Buster) with librealsense v.2.34 loaded and built in the user home folder (no system-wide installation). To make my application work I had to install some dependencies (that required an "apt update" + "apt upgrade") and rebuild and install (with sudo make install) the existing version of librealsense2.

     

    There is no difference: the application still crashes in 1-2 minutes. I am about to give up.

     

     

    0
    Comment actions Permalink

Please sign in to leave a comment.