D455 extrinsic calibration

I have multiple D455 cameras in a robot cell and would like to calibrate them with respect to the world origin using an aruco marker.
I know exactly where the aruco marker is placed in the world frame (T world-aruco ). Using the RGB camera, i detect the aruco marker and get the aruco to camera transform (T aruco-RGB ) and through the realsense SDK I get the extrinsics from RGB to depth (T RGB-depth ) so in theory the final extrinsics will be: T world-depth= (T world-aruco ) *(T aruco-RGB ) *(T RGB-depth ) 

 Current flow of data acquisition is:

config.enable_stream(RS2_STREAM_DEPTH, 848, 480, RS2_FORMAT_Z16, 30);
config.enable_stream(RS2_STREAM_COLOR, 848, 480, RS2_FORMAT_BGR8, 30);
... 

rs2::depth_sensordepth_sensor=profile.get_device().first<rs2::depth_sensor>();
depth_sensor.set_option(RS2_OPTION_VISUAL_PRESET, RS2_RS400_VISUAL_PRESET_HIGH_ACCURACY);
rs2::video_stream_profiledepth_profile= profile.get_stream(rs2_stream::RS2_STREAM_DEPTH).as<rs2::video_stream_profile>();
rs2::video_stream_profilecolor_profile=
profile.get_stream(rs2_stream::RS2_STREAM_COLOR).as<rs2::video_stream_profile>();

rs2_extrinsics T-RGB-depth= depth_profile.get_extrinsics_to(color_profile);

....
auto depth_frame = frameset.get_depth_frame();
auto color_frame = frameset.get_color_frame();
auto depth_intrin = pipe.get_active_profile().get_stream(RS2_STREAM_DEPTH).as<rs2::video_stream_profile>().get_intrinsics();
auto color_intrin = pipe.get_active_profile().get_stream(RS2_STREAM_COLOR).as<rs2::video_stream_profile>().get_intrinsics();

Current issue: 
To detect the aruco marker, i need to use the depth intrinsics or else my extrinsic calibration has a tilt in it when i project the pointcloud which does not make sense. What am i doing wrong? I am not using the align block as i use the extrinsics to transform the RGB to depth frame.