Question about accessing images from multiple vehicles/cameras

[saihv]

This is not an issue per se, but I was wondering if I could get some advice on something I'm working on with AirSim. My aim is to spawn and access images from multiple vehicles. I was able to spawn two drones that can be controlled independently: This was by replicating the Vehicle Pawn spawn blueprints (so I have two BP_FlyingPawns) and instantiating two AFlyingPawn objects.. and now I am trying to access their camera images.

As per my Unreal blueprints, right now I have only one Camera Director. In the single vehicle case, accessing the CameraDirector->TargetPawn provides access to the images. But because the camera director has only one 'Target Pawn', in order to access images from two vehicles, I am using the vehicle_pawn_ variable in SimModeMultiRotor.cpp, and invoking the method vehicle_pawn_->getFpvCamera() separately for both vehicles, and then logging two screenshots per iteration. Ideally, both vehicles have their own individual PIPCamera objects, so I was hoping to see different images: but I can see images from only the first vehicle's perspective.

Debugging using Visual Studio, I can see that this getFpvCamera() method is being called separately for both vehicles I have, and both vehicles have their own PIPCamera objects, so I guess there's something else that has to be done to ensure the scene is rendered from the two cameras separately, especially at the APIPCamera::getScreenshot() method: because although it's being called from separate instances, it's doing the same thing in both cases.

USceneCaptureComponent2D* capture = getCaptureComponent(camera_type, true);; FTextureRenderTargetResource* resource = capture->TextureTarget->GameThread_GetRenderTargetResource();

I was wondering if anyone can shed some light on what's a good way to achieve this. Thanks in advance!

[saihv]

Update, in case anyone's interested in this:

The way to solve this is that the second vehicle that's of the FlyingPawn type needs its own version of PIPCamera, so I had to duplicate the blueprint for PIPCamera. This lets me create new USceneCaptureComponent2D elements (say, screen_capture_2_ = UAirBlueprintLib::GetActorComponent<USceneCaptureComponent2D>(this, TEXT("SceneCaptureComponent2")); ), where SceneCaptureComponent2 is a part of PIPCamera2. This ensures that there are separate capture components for both vehicles. When getScreenshot() is called, it invokes the method APIPCamera::getCaptureComponent, and in here, it should be possible to return the capture component that's specific to whichever vehicle is requesting the image update.

[spencerh-b]

@saihv I have multiple drones spawned, but how am I able to control each pawn independently through an api? I am using the simple flight controller so it may be different than your's but I am quite stuck. Any suggestions

[saihv]

@spencerh-b Sorry about the really late reply: have you already figured this out? If not, please try the 'develop' branch of my fork at https://github.com/saihv/AirSim.git. I am also using SimpleFlight.

[spencerh-b]

@saihv I was able to figure it out. I appreciate that help. Not sure if I am doing it the most efficient way, but I can currently run around 40 drones in one sim.

[saihv]

Just out of curiosity, are you also recording images from the drones? Or are you mainly interested in only controlling them?

[spencerh-b]

I am using two stationary external drones to record the swarm of multi rotor's, but I'm not recording from each individual drone. So I guess only controlling them would be my primary concern

[sytelus]

The feature to record images from multiple cameras simultaneously is now available. You can, for example, record scene and depth images simultaneously.

Please see the doc: https://github.com/Microsoft/AirSim/blob/master/docs/settings.md#recording

[spencerh-b]

@sytelus thank you