Dynamically calling REST APIs in Data Factory under Fabric

[Anonymous]

I am attempting to call dynamically an open API in Data Factory under Fabric. I connected to a REST API where I provided names and ids. I have the following pipeline:

My Lookup activity configuration:

My Foreach activity configuration:

My Copy activity (source):

My Copy (destination) activity:

I would like to send 5 .json files (naming based on location_name) to Lakehouse, but I encounter difficulties exporting multiple files.

Please advise on the configuration highlighted in red.

Thanks!

[Anonymous]

@ajarora Thanks! I have just suceeded. Missing brackets were the case.

My current implementation involves obtaining data from multiple APIs simultaneously every day. I want to implement a process that does not rewrite or overwrite new data but appends it, taking into consideration the possibility of duplicates in consecutive API calls. Is it possible to achieve this using Data Factory in Fabric?

[ajarora]

Hello. Glad to know things worked.

For your follow up ask, if your ask is to use copy and write new data into lakehouse tables, we support "append" and "overwrite" mode already. But to handle duplicates you actually need UPSERT capability. This is a new functionality we are exploring, and if you are interested, can you please vote on this existing fabric idea about it: https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=87f3d168-6022-ee11-a81c-6045bdc01ce4

If I misunderstood your ask, please clarify.

[Anonymous]

Basically, I have just looked at that 'append' and 'overwrite' concern Lakehouse Tables, but not Lakehouse Files (which is my case).

There are various options of Copy Behavior. Should I use them? Or any workarounds in terms of UPSERT logic?

[ajarora]

I dont see how you can use "Files" to be able to maintain a tabular data, ie rows with unique keys. Please ping me offline, I can guide you further on how to proceed on that. But hoping the original issue you reported is addressed.

[ajarora]

Your destination activity looks incorrect, did you provide your concat logic via the "Dynamic content" button that shows up on hover over the "file name" textbox ?

One handy way for debugging expression (dynamic content) is to check the activity input of your activity run, and see if the value got "resolved" properly to the desired value at runtime or not.