low image fps while using simGetImage python API

[cdbharath]

I am trying to apply image processing techniques on image stream received from simGetImage API. I want an image stream of 30 fps. But the image stream I get from simGetImage gives an fps of 10-12. The simulation fps (50 fps) is good when simGetImage API is not used.

My code (ROS wrapper):

self.image_publisher_rate = 0.01
rospy.Timer(rospy.Duration(self.image_publisher_rate), self.image_publisher_cb)

def image_publisher_cb(self, timer):
    image_data = self.get_camera_data()
    self.image_pub.publish(self.bridge.cv2_to_imgmsg(image_data, encoding='bgr8'))

def get_camera_data(self):
    rawImage = self.image_client.simGetImage(self.camera_name, airsim.ImageType.Scene)

    if rawImage is not None:
        frame = cv2.imdecode(airsim.string_to_uint8_array(rawImage), cv2.IMREAD_UNCHANGED)
        return frame

OS : Ubuntu 18.04
Game Engine : Unity 2020.1.4f1

Other answers I saw were mostly related to projects depending on Unreal Engine (as opposed to mine)

I want my camera fps to be equal to or more than 30 fps. How do I achieve this?

[m1baldwin]

@cdbharath There are a couple "tricks" you can do to get slightly higher frame rates.

1. change the "ViewMode" setting in the settings.json file to "NoDisplay". You won't be able to watch the simulation in the view port, but I think this allows you to get slightly more out of the FPS.
2. I have seen that I get higher FPS when I have the UE Editor in the foreground. If I minimize it, I see it drop. Maybe this is something you can play with?
3. Decrease the resolution of your camera a bit, if you can afford it.
4. Try getting the image as an uncompressed RGB image, and then convert to a numpy array and do what you need with it. Getting the uncompressed image and then converting might be more efficient in the image retrieval process than requesting the already compressed images.

[m1baldwin]

Also, the FPS rate of Airsim is something under pretty active development by many working on this project. It has already gotten much better over the past couple months but still needs some improving.

[cdbharath]

@m1baldwin I tried all your suggestions

1. Changing "ViewMode" didn't have any effect in the Unity simulation (It didn't stop the viewport. The simulation is still visible). I dont think the "ViewMode" parameter in the settings.json file is parsed by Unity. (It worked with prebuilt binaries though)
2. I use Unity Editor. This option didn't provide mnuch difference in the FPS
3. Yeah decreasing the camera resolution provided better results (But I wanna keep it as my last resort)
   1024*768 - 5-6 fps
   800*600 - 8-10 fps
   640*480 - 10-12 fps
   400*300 - 25 fps
4. Getting the image as uncompressed image also didn't provide much improvement in terms of FPS

Are there any other ways? The recent FPS rate development are already pushed into the repository?

[m1baldwin]

@cdbharath Thanks for the follow-up. Some of the improvements are in the main repo, but some have also been removed due to causing Urneal engine to fail. I think predominantly they were for the Unreal engine use cases so not sure how much they will help you. I'm not familiar with using Unity in this case.

[nightduck]

Duplicate (sort of) of #2026

[ICYPOLE]

@cdbharath There are a couple "tricks" you can do to get slightly higher frame rates.

1. change the "ViewMode" setting in the settings.json file to "NoDisplay". You won't be able to watch the simulation in the view port, but I think this allows you to get slightly more out of the FPS.
2. I have seen that I get higher FPS when I have the UE Editor in the foreground. If I minimize it, I see it drop. Maybe this is something you can play with?
3. Decrease the resolution of your camera a bit, if you can afford it.
4. Try getting the image as an uncompressed RGB image, and then convert to a numpy array and do what you need with it. Getting the uncompressed image and then converting might be more efficient in the image retrieval process than requesting the already compressed images.

@m1baldwin Thanks for your advice but for 4, how to get the uncompressed RGB image? Simply by modifying as 'Recording: Cameras: Compress: false' in settings.json?

[m1baldwin]

@ICYPOLE point 4 is if you are requesting the images via the python API. There are flags in the image request api you can set to change whether the image returned is compressed or uncompressed

[m1baldwin]

Haven't tested the setting in the settings.json file but maybe that has a similar effect

[m1baldwin]

There are some programming examples on this page to get compressed or uncompressed images: https://microsoft.github.io/AirSim/image_apis/#python_1

[zimmy87]

After running some tests with the image_benchmarker.py script, there doesn't appear to be any significant difference in performance after applying #4147, so I am reopening this issue.