Fabric Failed to create session for executing notebook

[UdaySutar]

The pipeline contains a notebook activity, and while it runs successfully at times, it occasionally fails, generating the following error.Please provide the reason of failure and how to fix this permanantly.

"error": {
"ename": "Exception",
"evalue": "Failed to create session for executing notebook. SessionId: Notebook: Notebook_",
"traceback": [
"Exception: Failed to create session for executing notebook. SessionId: Notebook: Notebook_",
"--> SparkCoreError/UnexpectedSessionState: Livy session has failed. Error code: SparkCoreError/UnexpectedSessionState. SessionInfo.State from SparkCore is Error: Encountered an unexpected session state Error while waiting for session to become Idle. Error description: [plugins.. WorkspaceType:<> CCID:<>] Attempt=[1]/[3]Cluster was in terminal state=[Cancelled] before it reached 'Ready' state. Cluster job has WorkspaceName=[], SpecName=[], and JobId=[].. Source: SparkCoreService."
]

[V-yubandi-msft]

Hi @UdaySutar ,

We noticed we haven't received a response from you yet, so we wanted to follow up and ensure the solution we provided addressed your issue. If you require any further assistance or have additional questions, please let us know.

Your feedback is valuable to us, and we look forward to hearing from you soon.

[V-yubandi-msft]

Hi @UdaySutar ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

 

Thank you.

[V-yubandi-msft]

Hi @UdaySutar ,

Can you please confirm if the solution worked for you. If it did, please consider marking the answer 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details.

Thank you.

[Srisakthi]

Hi @UdaySutar ,

Step 1:  Goto your Workspace settings -> Spark settings -> High Concurrency then enable for notebooks and pipeline run options.

Step 2 : In your notebook attach to high concurrency session(use when you want to run manually)

Step 3: In your data pipeline -> notebook activity -> settings -> Advanced settings -> give your session tag name(anything)

So now when you run your pipeline, if no high concurrency session available it will create high concurrency session and in one session it can execute 5 notebooks.

Reference : https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-noteboo...

Regards,

Srisakthi

[UdaySutar]

Hi @Srisakthi 
this is failing for scheduled pipeline. can we connect on this .

[Srisakthi]

Hi @UdaySutar ,

We can connect.

Regards,

Srisakthi

[V-yubandi-msft]

Hi @UdaySutar ,

The error message:

Failed to create session for executing notebook… Cluster was in terminal state=[Cancelled]"

This issue often occurs during scheduled executions because the Spark cluster is provisioned on demand. If the cluster fails to initialize in time, it may timeout or terminate unexpectedly.

To improve the reliability of scheduled pipeline runs, consider configuring the environment to use High Concurrency Mode with a session tag. This allows the pipeline to reuse an existing Spark session, reducing cold start delays that often disrupt scheduled tasks.

By using an active high concurrency session, the scheduled pipeline avoids the overhead of starting a new cluster each time, ensuring smoother execution and better stability.

I hope this helps....Please Accept as solution if this meets your needs and a Kudos would be appreciated.

Yugandhar.

[Srisakthi]

Hi @UdaySutar ,

Might be it is trying to spin up to multiple sessions to run the notebooks. Have you trued session tags and high concurrency session.? It can help you saving costs, startup time and it can avoid creating multiple sessions.

Regards,

Srisakthi

[UdaySutar]

Hi @Srisakthi - can you explain in detail each steps to proceed  ?

[V-yubandi-msft]

Hi @UdaySutar ,

Thank you for engaging with the Microsoft Fabric Community.  The Spark runtime in Microsoft Fabric couldn't become ready and was shut down before execution. This happened due to limited resources, excessive job queue times, or failures in cluster setup and provisioning.

To resolve the notebook execution issue in Microsoft Fabric, make sure the Spark pool has enough resources and is properly configured.

1. If sessions fail, increase the timeout duration to allow more startup time.
2. If problems continue, restart the notebook block or terminate old sessions that might be causing conflicts.
3. Scaling up resources in the Spark pool can improve performance, and checking for Fabric service outages ensures the issue isn't due to system disruptions

I found similar issues in the community that might be useful for your case, so it’s worth checking them out.

Solved: In Fabric Failed to create session for executing n... - Microsoft Fabric Community

Solved: Unable to start spark session: Livy session has fa... - Microsoft Fabric Community

If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.