Parquet with timestamp prior 1900- Spark conf

[Mathan1787]

Hi,

I am trying to copying data from azure sql database to lakehouse. I have used copy activity to move the data into parquet files but when i try to open them in notebook, I am getting an error message. 

I have to add below code.

spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED 

spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED

I have tried to set this in spark conf in workspace setting to have it set for my entire workspace but i cant find them in drop down list so i have to manually add it in all the notebook. could you help me here. thanks

[DennesTorres]

Hi,

If this works but you can't set for the entire workspace because it's not available in the dropdown, should you register this as an issue, asking for correction?

You can using this link: https://community.fabric.microsoft.com/t5/Issues/idb-p/Issues

Kind Regards,

Dennes

[DennesTorres · Accepted solution]

Hi,

Let me fix this previous reply:

The drop down are open, you can add configurations even if they are not present in the dropdown. If they are wrong, the spark session will not start and will result in an error message.

Kind Regards,

Dennes