Vector based implementation of "rs2_deproject_pixel_to_point" function
I have a code to get the real-world coordinates of a pixel which loops through each pixel and then calculates the real-world coordinate. I want a vector based implementation of the same function so that is becomes GPU optimised. Here is what I am doing currently:
def filter_data(depth_image,color_image):
#This Function is used to filter out the pixels which are in the vertical range of 1m of the camera
h,w = np.shape(depth_image)
depth_scale = depth_sensor.get_depth_scale()
#Getting the intrinsic parameters of the camera for the calculation of the xyz values
depth_intr = depth_frame_1.profile.as_video_stream_profile().get_intrinsics()
for y in range(h):
for x in range(w):
# The Function rs2_deproject_pixel_to_point() returns the realtime xyz values corresponding to the pixel
coordinate = rs.rs2_deproject_pixel_to_point(depth_intr,[y,x],depth_image[y,x]*depth_scale)
if abs(coordinate[0]) > 0.5 :
depth_image[y,x] = 65000.0
color_image[y,x] = 0
There is a lag of almost 2 seconds when I run this on my PC. I want to run this code on I.MX 8 mini processor on my drone. I am sure that in that processor, I will see latency of even more seconds so I want to use GPU for this function:
This is what I tried:
@jit(target_backend='cuda')
def filter_data(depth_image):
#This Function is used to filter out the pixels which are in the vertical range of 1m of the camera
h, w = depth_image.shape
# Create arrays of pixel coordinates
y_coords, x_coords = np.meshgrid(np.arange(h), np.arange(w), indexing='ij')
depth_scale = depth_sensor.get_depth_scale()
#Getting the intrinsic parameters of the camera for the calculation of the xyz values
depth_intr = depth_frame_1.profile.as_video_stream_profile().get_intrinsics()
# Calculate real-world coordinates for all pixels in one go
pixel_coords = np.stack((y_coords, x_coords), axis=-1) # we are getting coordinates in (y, x, z) because we have given the input image_pixel_coordinates in the from [y, x]
depth_values = depth_image * depth_scale
real_coords = rs.rs2_deproject_pixel_to_point(depth_intr, pixel_coords, depth_values)
condition = np.any(np.abs(real_coords) > 0.5, axis=-1)
depth_image[condition] = 65000.0
But I got an error:
TypeError: rs2_deproject_pixel_to_point(): incompatible function arguments. The following argument types are supported: 1. (intrin: pyrealsense2.pyrealsense2.intrinsics, pixel: List[float[2]], depth: float) -> List[float[3]]
which means that function expects the pixel argument to be a list of floats, but I to use NumPy arrays.
How to optimise code for GPU or make the code work faster on CPU
-
Hi Dhruvdarda2001 Instead of looping through each pixel to get an XYZ coordinate, it will use less processing power if you target one specific pixel with the rs2_project_color_pixel_to_depth_pixel instruction which converts a 2D color pixel to a 3D depth pixel. Using this method, it is not necessary to process the entire image. The link below demonstrates use of this instruction in Python.
https://github.com/IntelRealSense/librealsense/issues/5603#issuecomment-574019008
Official documentation about this instruction can be found here:
-
Thanks MartyG, that is insightful!
So, is there a function that can do exactly opposite of what "rs2_project_color_pixel_to_depth_pixel" is doing, as in, is there any pre-built function that can give me the (pixel_x, pixel_y) from an approximate (real_world_x, real_world_y) that I can get from other sources?
-
Thanks MartyG! This is helpful.
Please sign in to leave a comment.
Comments
4 comments