Rotation tracking without IMU?

Hello, we just bought a Realsense device to play around with as we have a project we think it could be a good fit for, but I'm curious what people with experience with it think.

We are interested in using the camera to track the rotation of a handheld device. Trick is, we cant use the gyroscope, because the device might be used inside a room where the room itself is moving (think plane, truck, boat etc). We need to know the rotation of the device relative to the room, and not the earth. If the device is statically positioned in the room, but the room was moving, the gyroscope/accelerometer would be sensing rotation/position which we do not want. 

The thought is, with the realsense since it can build point clouds and do depth tracking, we could use that to calculate the rotation without using the gyro or accelerometer. We do not care about position tracking, just rotation.

Take for example, the camera placed inside a box, secured facing a perpendicular one of the sides. Even if you rotate the box, the camera is still facing the side and has not rotated relative to the box. Thats the rotation we are trying to track.