Expert opinion/tips regarding incremental refresh with Semantic models/Dataflow

[AutoJL]

Hi fellow community!

I need some expert advice on the topic above mentioned.

To start off I am unsure if this post should be located in the "Service" forum since it also regards Dataflows. If so let me know and I will duplicate/move the question there.

Let me put some context in order to better explain the situation.

I have a powerbi report that is built from plain txt files (actually .log files that contain a mixture of plain text as well as flattened jsons). I am storing this files in either Sharepoint/Azure blob storage. In order to enhance this report I want to make use of the incremental refresh feature since this .log files are pretty heavy: 100k+ lines each file with multiple files per day and per system to analyze and multplie systems to store info and extrac insights from. As you can imagine all the data transformations get very heavy, specially as multiple days worth of information are stored ( and since its log information past files are never modified).

I have been searching some forums/blogs online to fully get grasp on how incremental refresh works (and of course the MS Learn documentation) , specially considering that my datasources cant fold querys to make this easier. As it happens theres some information that this is still something that can be overcome ( like this for example: Power BI incremental refresh with binary files).

I have tested with the Semantinc model Incremental refresh and it definetely looks like it works and im confident it will also work if using Dataflows.

My questions to you experts come here:

What should I should use?

Any specific recomendations on wether to use the Semantic model vs the Dataflows Incremental Refresh?

Any restrictions I need to take into account?

One thing to take into account is that I do have access to Premium Per User workspace/license in my organization.

Any help, tips, suggestions, related content to enhance knowledge on this topic and situations (Blogs/Youtube videos, etc) is more than welcome.

Many thanks!

[AutoJL · Accepted solution]

Hi everyone!

Let me link to the post in the other forum as it contains a more detail solution along the lines of what @KonradZawel said.

Solved: Re: Expert opinion/tips regarding incremental refr... - Microsoft Fabric Community

Many thanks!

[KonradZawel]

For heavy log files, both Semantic models and Dataflows with incremental refresh can work. Use Semantic models for complex relationships and Dataflows for centralized, reusable ETL. Premium Per User is great for scaling.

[AutoJL]

Thanks for the help @KonradZawel , much appreciated!

[Anonymous]

Hi, @AutoJL 

Note that incremental refresh is only possible with data sources that support collapsed refresh. In my personal opinion, ".Log" files don't enable collapsed refresh, my possible solution would be to use a scripting tool to convert it to .csv file and then use incremental refresh. If you have currently completed the incremental refresh of the semantic model, it is recommended that you go to the Service platform to test whether you can complete the normal refresh, as for the use of Dataflows, there is not a lot of understanding of this feature, you can go to the true "Service" forum to find more professional support (Note: Here is Desktop forum, not Service). Thank you for your understanding.

Service - Microsoft Fabric Community

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

How to get your questions answered quickly --  How to provide sample data in the Power BI Forum