RealSense D456 & TouchDesigner
Hi, I want to use the D456 in TouchDesigner to do Pose Tracking.
I have a brand new RealSense D456 connected to a MiniMac (Intel) with MacOS Mojave 10.14, with the librealsense installed and the RealSense Viewer detecting the device and displaying the 3D image alright.
My problem is that in TouchDesigner the RealSense TOP does not detect the RealSense D456. I tried putting a videodevin1 TOP and it does not see the D456 under Devices either.
I assumed the librealsense should be listed under Libraries but it’s not.
**Possible problems I see**
1. I installed the RealSense SDK 2.54.1 because it has support for the D456, but I read that the TouchDesigner top for RealSense was developed using the SDK version v2.50.0.
2. I also see that the TouchDesigner top only has option till D455 in the dropdown model menu.
3. I'm using TouchDesigner non-commercial version bundle 2021.16410 because that's the last one to work with my outdated Mojave 10.14 that I would rather not update.
I read the documentation before I bought the camera and noticed I might have problems, but I really needed a water resilient camera that could run on MacOS and I thought this is my option and that I could work around it somehow.
Do you think it is any likely that I can use the D455 option of the TouchDesigner TOP to work with my D456?
Does it make sense that I try downgrading to SDK v2.50.0 or will it not be supporting the D456?
Is there any chance to work around these limitations or should I give up on using the D456 camera with TouchDesigner?
In that case do you have a suggestion of how to send the image data directly to Max/MSP?
I'm rather new to all this and any help would be really REALLY appreciated.
Thank you!
-
Hi Dafni Xanthopoulou When new RealSense camera models are introduced and they are detected as having a new model number that has not been used in the past, support for those models is not backdated to older RealSense SDK versions, so the SDK version that the support was introduced in or a newer version must be used with those cameras.
If the problem is that the camera needs to be detected as D455 in order to work with TouchDesigner TOP then obtaining a D457 (which also has the IP65 water-resistant casing) and putting it in USB mode so that it is detected as a D455 might be a practical solution, though one that comes with a financial cost.
At the top of this case you say that you want to do pose tracking. As you are interested in using TouchDesigner, do you mean gesture tracking, please (initiating actions based on certain finger poses). Pose tracking means something different (finding the position and rotation of a camera or an object that the camera is observing).
Can you also clarify please which programs you are referring to when you say Max / MSP. Thanks!
-
Thank you so much or your answer MartyG. I want to find the position and rotation of a camera or an object that the camera is observing. When I say Max/Msp I mean specifically Max 8 by Cycling '74. In any case, my main goal is to get the RealSense camera working in TouchDesigner. So if I understand right, is there no workarounds you could suggest other than buying a D457? In that case I'm also not sure why would that work in TouchDesigner, since the TouchDesigner TOP looks like it's meant to recognise up to D455 models?
-
D457 supports two types of cable connection, GMSL / FAKRA (a high speed connection used in the automotive industry) or USB. By default the D457 operates in GMSL / FAKRA mode and is detected as a D457. However, under a removable cap on top of the camera's casing there is a physical switch that can be pushed sidewards so that the camera operates on a USB connection and is detected as a D455, with the same performance and features as a D455.
If your goal is to pose-track a camera or object then TouchDesigner will not be able to help with that, as its purpose is to detect gestures made with the hand / fingers, such as pinching finger and thumb together.
It appears that the RealSense SDK has a line in its code (at the link below) that changes the camera name from D455 to D456 if it is detected that the camera has an IP65 protective casing and the name D455. It's possible that you could edit that check out of the SDK's source code and then build the SDK from the modified source code and see if the D456 detects as a D455.
If pose tracking (finding the position and rotation of a camera or an object that the camera is observing) is your goal then TouchDesigner will not provide that function. The D456 (or any other camera in the RealSense 400 Series range) could also not provide the position of the camera. It is possible though to detect the pose (position and rotation) of an object with a 400 Series model, or to find the rotation of the camera (but not its position in the world). This could be done with open-source program code, which should not care if the camera is a D456.
-
Thank you for this insight, I will try to build from source changing the line in the code and let you know how it went.
What I really need to do is skeletal tracking, so If that doesn't work I was wondering, will it be possible to build from scratch a skeleton tracking app in macOS Mojave? In this section of the website it doesn't mention macOS compatibility, I wonder if it means that it's not supported or if the information has not been updated: https://dev.intelrealsense.com/docs/skeleton-tracking-sdk-installation-guide
-
Intel have a YouTube video guide at the link below for building RealSense body joint tracking from scratch.
-
MartyG after building from source the latest SDK editing out the line of code that changes the name from D455 to D456 as suggested, Touch Designer detects the camera as D455 -which is great to begin with, but still I encountered this error in the realsense TOP.
Realsense Firmware does not match SDK version.
If experiencing problems, change firmware to version 05.12.12.100. To change firmware, use RealSense Viewer from SDK.v2.44.0.
So then I built from source SDK v2.44.0, and then tried to downgrade the firmware from version: 05.15.01.00 to version: 05.12.12.100. from command line but I get this error:
ERROR [0x1078825c0] (d400-device.cpp:125) Firmware version isn't compatible 5.12.12.100
This firmware version is not compatible with Intel RealSense D456
Apparently in the Device Name detected by the fw-update the camera shows up as Realsense D456 although TouchDesigner shows it under D455 and the RealSense Viewer also detects it as D455.
-
@MartyG
Thank you. I have tried a few different approaches, but still struggling. My goal now would be to use the infrared image of the realsense (you see I need to use it in a dark environment with blinking lights) with this mediapipe plugin for TD: https://github.com/torinmb/mediapipe-touchdesigner
When I built the realsense SDK editing out that line in the code as you suggested on a Mac mini (2018) (Intel) running on Mojave, the real-sense viewer and the Realsense TouchDesigner TOP would detect the camera as D455 and so I could extract the IR image in TD using the TouchDesigner TOP (although the skeleton tracking is discontinued, the TOP has some basic functions). But still I was not able to use this IR image as source for the Mediapipe plugin, as it seems to receive input only from webcam and realsense was not being detected as a webcam. Is this because of the two kinds of cable connections that you mentioned, GMSL / FAKRA and USB?
Since that was not going anywhere, I decided to try and use Python and pyrealsense2, and I had to upgrade to macOS Ventura because mediapipe was not supporting in Mojave.
Now with MacOS Ventura 13.6.5 the camera is detected as webcam in the computer and in TouchDesigner it connects as ‘video device in’ in the Mediapipe Plugin but the TouchDesigner TOP does not detect it so I can’t use it to extract the infrared image. The Real-Sense Viewer is also not detecting the device. I also tried with Pyrealsense2 to test a pose-detector git in Python but the script returns RuntimeError: No device connected. This was not happening in Mojave actually..
I also tried to do all this in my Macbook Air that runs on Monterey 12.4 but it also doesn’t detect the camera and the real-sense viewer crushes after a second and shows some USB error, I think it’s because of Apple Silicon and usb-c. I followed the LightBuzz instructions with no luck.
I’m not sure what else to try, if you have any suggestions I'd be happy to hear them!
-
Infrared should be available on both a USB and a GMSL / FAKRA cable connection.
A plugin will only be able to display a particular stream type if it is designed to do so. For example, a plugin might only be designed to access the depth stream. The mediapipe plugin that you quoted appears to only be making use of an ordinary webcam RGB image to perform its various types of analysis rather than being able to make use of RealSense-specific data streams such as depth and infrared.
In regard to Ventura's non-detection, recent MacOS versions such as Ventura and Sonoma have been having difficulty detecting RealSense cameras. Monterey also had these problems for a time, though eventually the Lightbuzz guide that you mentioned made it possible for the cameras to work with Monterey.
The Lightbuzz guide is designed to work with both Intel and Apple Silicon Macs, though use of a USB C to C cable on a Mac does make it more likely that the camera will not be detected.
-
MartyG hi, I realised that there's a lot of macOS Ventura and Monterey problems with the error I'm getting "failed to set power state" or "failed to claim usb interface" and I see some unreliable suggestions of how to override this that will not work for me as I need to configure realsense for a permanent installation and I cannot risk any instability that would cause serious issues in its mantainance.
Now I consider downgrading to another version of MacOS and I'm trying to decide which one it should be.
The computer I'm working on is Mac Mini Intel (2018).
I want to try this as a last resort before switching to a Windows computer, but I see there is so many cases with any version of macOS that I'm not sure if I should just skip this step and already switch to Windows.
I know now that High Sierra macOS is the only one that had been officially supported, but I doubt I can downgrade to that one and if I did I assume I would have terrible problems with all the libraries and programs I need to use to develop.
What would you recommend?
Please sign in to leave a comment.
Comments
11 comments