intel Realsense for unity - how to control a game with the given wrapper
how can I control a game using realsense instead of using a keyboard , still confused about how to do that , I see in the scenes which are inside the sdk for unity , that you can turn the images that the camera sees into depth and color textures , but how can that be used to control a game instead of using arrows in the keyboard for example , knowing that you don't have built in support for hand tracking and gestures (that what I understand correct me if I'm wrong) , please give ideas for using the sdk without using any external packages unless they are completly free to use
-
Hi Saadisaadi1 The old 2016 version of the RealSense compatibility plugin for Unity at the link below was best suited for integration of RealSense into games using face, hand and gesture control. It works with the now retired RealSense SR300 camera model but is not compatible with modern RealSense cameras such as the 400 Series, unfortunately and was not designed for recent Unity versions.
https://github.com/IntelRealSense/librealsense/issues/10114#issuecomment-1002456887
The current version of the Unity compatibility wrapper of the RealSense SDK does not have similar control-input ability.
In regard to external free options, a product called Nuitrack 3D that is compatible with modern RealSense cameras can track movement of body parts and has Unity integration. It has an unlimited free trial which is limited to a 3 minute session.
https://github.com/3DiVi/nuitrack-sdk/blob/master/doc/Unity_Basic.md
-
thank you MartyG , but what can I do with the unity wrapper right now , like how can I use it to control a game, what the built in functionality that you have right now?
-
Ok MartyG that would prevent me from integrating it and using it with my already made games.
So I want to ask about the sdk that work with opencv , what can I do with that?, I heared that I can use opencv with mediapipe to track hands and know the current gestures , can I use the realsense also to know the distance of the hand from the camera? how can I use it with opencv , and what the functionalities that I can get so I can use this to control a game?
This is so important to know because that would dictate if I'm going to learn the API of both opencv mediapipe and your sdk for python and opencv
-
The Unity Asset Store has an OpenCV interface, though it is not inexpensive.
https://assetstore.unity.com/packages/tools/integration/opencv-for-unity-21088
Alternatively, as Unity uses C# code (and therefore so does the RealSense Unity wrapper) it is possible to interface with the C# wrapper of the RealSense SDK to access its functions from Unity.
-
MartyG In my last comment I was not talking about unity , I was talking about opencv and mediapipe in python , I would make games instead using pygame , can you please give me what functionality can I get ? I mean can I get the distance from the camera of each pixel in the image , like using this line :
imageRGB = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
this line gives an image for the rgb colors , does your sdk have functionality , that returns similar thing but distance texture , so I can for example from the code on the top after processing it with mediapipe get the x and y of a finger , and from your image get the z using x and y , so I know where is the finger in 3d and that way I can do some controling based on the depth of the hand and distance from camera
-
The link below has a script that uses the SDK's Python language (pyrealsense2) and mediapipe to get the distance using X and Y.
-
thank you MartyG I will receive the camera in couple weeks , I will test the code and go back to you if there is something else to ask , for now thank you very much
Please sign in to leave a comment.
Comments
9 comments