LLM +
Automation
Framework
Guide for
Table of Contents
I. Introduction ........................................................................................................1
I.1 Overview & Challenge ................................................................................................................................................ 1
I.2 Benefits ........................................................................................................................................................................ 2
II. Prerequisite ......................................................................................................2
III.1 Runtime Dependencies ........................................................................................................................................... 3
III.2 Host Python Backend using Docker ....................................................................................................................... 3
III. Framework & Solution .....................................................................................4
II.1 Solution Overview ..................................................................................................................................................... 4
II.2 Solution Architecture .............................................................................................................................................. 10
II.3 Framework Components ........................................................................................................................................ 11
II.4 Reusable Components ............................................................................................................................................ 12
IV. Quick Start Use Cases .................................................................................... 13
IV.1 Conversational (Default) ........................................................................................................................................ 13
IV.2 Transactional .......................................................................................................................................................... 15
IV.3 Transactional with human in the loop ................................................................................................................. 17
V. Appendix ......................................................................................................... 20
V.1 Future Enhancements............................................................................................................................................. 20
V.2 Known Issues and Limitations ............................................................................................................................... 20
V.3 Additional sources of documentation .................................................................................................................. 20
[Type here]
I. Introduction
I.1 Overview & Challenge
Leverage this LLM Framework template along with its Generative AI powered
custom activities to make LLM App & Automation Development much easier,
more flexible, more reliable and built for enterprise level usage with
automation.
We've seen how powerful large language models can be, whether it's for generating
professional looking essays, responding to emails, extracting entities from
unstructured sources, generating works of art, music, etc - that's why we fondly call
them generative AI.
Its power relies on the huge amount and quality of data it's trained with, that
ironically is also where the limiting factor is. But through other integrations it can
make these models more powerful and reliable.
The challenge with large language models by itself is it's built to be general
purpose. It has been trained on millions of pieces of text so it can "understand
semantically" the human language.
It might be able to answer specific questions based on the information it was
trained on, but in reality, large language models don’t know of the underlying
context that language describes, as the technology aims to generate text that
sounds nice, grammatically, and semantically, but they don’t really have some kind
of objectivity or fact-based context aside from just satisfying statistical consistency
with the prompt.
So essentially, without the proper integrations or framework, LLMs can perpetuate
and amplify harmful biases through hallucinations and might, in turn, reinforce
today's pandemic issue of misinformation and disinformation.
I.2 Benefits
By marrying UiPath with Generative AI capabilities (specifically LLMs) through this
solution, it can help you achieve the following:
1. Extend the generative capabilities of large language models by providing
context from any types of sources.
2. Achieve full automation mileage whether its attended or unattended
automations, by supplementing the output derived from LLM with custom
actions.
3. Address general concerns around information accuracy, by leveraging
automation to help speed up adoption at an enterprise level.
4. Quickly build your own Generative AI powered automation using Large
Language Models, using this framework that is both flexible and reusable.
5. This template leverages custom connectors/API built on top of LangChain
framework which is an industry standard in LLM app development.
II. Prerequisite
The custom activity package used throughout the framework is the main pre-
requisite to utilize the template for your LLM powered projects. This section depicts
the steps to set up the custom activities included in the package.
See Appendix ‘V.3. Additional sources of documentation’ for full video tutorials
III.1 Runtime Dependencies
o Docker - Download Docker Desktop | Docker
o OpenAI API Key (https://platform.openai.com/) or Claude API Key (Anthropic
Console)
o Pinecone Account - https://app.pinecone.io/
A. Register and create a project.
B. Create/Name your index.
C. Index Dimensions should be 1536 (OpenAI output embeddings dimension).
III.2 Host Python Backend using Docker
The following step portrays docker container running on a local machine. In a
production scenario it’s best to host the API on a proper remote app server.
a. Download & Install Docker - https://www.docker.com/products/docker-desktop/
b. Make sure Docker is running by starting up the application. Verify docker is running
by checking docker icon in the right icons panel
c. Run below command in command line
docker run -p 5000:5000 zell12/qblangchainapi:latest
Where first 5000 -> external port accessed by client; and second 5000 -> internal port from code e.g. waitress-server --
listen=0.0.0.0:5000 app:app
d. Verify if http://localhost:5000 is running in the browser, which means the API is
running.
If hosted on a remote webserver, localhost will be replaced with the remote host name or IP
III. Framework & Solution
II.1 Solution Overview
The solution comprises of 2 main components:
A. UiPath Studio Template (LLM Framework Studio Template)
B. Custom Activity Package (LLM Framework Activities)
A. LLM Framework Studio Template
This template orchestrates the reusability of the solution across a wide array
of use cases powered up large language models, which can either be
Conversational (chat based/context aware conversations), Transactional (queue
based) and Transactional with human in the loop.
Along with this reusability, there is an important consideration for flexibility in the
framework so that it can cater to different types of downstream process which may
or may not involve UI/API automation.
It is created to fit all of the best practices regarding logging, exception handling,
application initialization, and others, being ready to tackle a complex conversational
and transactional business scenario.
The template contains several pre-made State containers for initializing
applications, retrieving input data, processing it, and ending the transaction. All of
these States are connected through multiple Transitions which cover almost every
need in a standard automation scenario. There are also multipleinvoked workflows,
each handling particular aspects of the project.
B. Custom Activities
LLM Powered Custom Activities built using LangChain Framework. These
activities build up the core elements to enable context-aware and memory
backed interaction with your favorite large language models.
a. Ingest Knowledge Source – Create and serialize vector database out of
provided sources such as csv, pdf, YouTube video links, word documents,
etc.
Properties
- Authentication
API Endpoint (String)
This is the endpoint where the backend API is hosted (either locally or on a remote app
server) + the api route = <API_Endpoint_BaseUrl>/api/vectordb
For instance, if docker container is ran in local machine where the bot is, then the API
endpoint will be the, where <API_Endpoint_BaseUrl> = http://localhost:5000
Resulting endpoint will be http://localhost:5000/api/vectordb
Embeddings API Key (String)
This is using OpenAI Embeddings so the key will come from https://platform.openai.com >
Account > View API Keys (API keys - OpenAI API)
Pinecone API Key (String)
API Key for the vector database. Must have a Pinecone Index created either at
https://app.pinecone.io/ or from Azure Resources. Fetch API Key from Pinecone Project >
API Keys. The following video guide includes tutorial on how to setup and get required
properties from Pinecone. https://youtu.be/xCqTXhPVakY
- Input
Source (String)
File path or URL sources. Base Path (e.g. c:\temp\file.csv -> file.csv or
https://youtube.com/watch?v=dafsf790 -> watch?v=dafsf790) will be used as a
namespace value in the vector store
- Options
Pinecone Environment (String)
The environment name of the Pinecone vector database e.g. us-east1-gcp
Pinecone Index Name (String)
The index name of the Pinecone vector database. Pinecone project > Indexes
- Output
Output Context (String)
Reference (i.e. namespace) to the vector embeddings or documents that will be used to
generate context using semantic search.
a. Converse With Context – Context aware LLM chain with conversational
buffer memory. Multi-input (llm model, context docs, memory/history,
prompt template) LLM chain.
Properties
- Authentication
API Endpoint (String)
This is the endpoint where the backend API is hosted (either locally or on a remote app
server) + the api route = <API_Endpoint_BaseUrl>/api/llmchain
For instance, if docker container is ran in local machine where the bot is, then the API
endpoint will be the, where <API_Endpoint_BaseUrl> = http://localhost:5000
Resulting endpoint will be http://localhost:5000/api/llmchain
Embeddings API Key (String)
Pinecone API Key (String)
LLM API Key
API Key for the chosen LLM provider. Get API key from the LLM provider’s portal.
OpenAI -> https://platform.openai.com > Account > View API Keys (API keys - OpenAI API)
Anthropic -> https://console.anthropic.com > Account > API Keys
- Input
Source or Context (String)
Message
[Any] Question or instruction [to the provided context]
Memory
Serialized memory file injected into the LLM chain.
- Options
Pinecone Environment (String)
Pinecone Index Name (String)
LLM Provider
LLM provider that will be used for the completions. Defaults to OpenAIGPT.
LLM Model Name
Specific model name for chose LLM provider.
Defaults to gpt-4-0613.
OpenAI GPT Models -> https://platform.openai.com/account/rate-limits
Anthropic Claude Models -> https://www-
files.anthropic.com/production/images/model_pricing_july2023.pdf
- Output
AI Message
Context-aware completions Output
Updated Memory
Same reference to input memory, with updated context after the LLM chain run
a. Fetch Knowledge Source – Fetch document/s from vector database for
inspection purposes
Properties
- Authentication
API Endpoint (String)
Embeddings API Key (String)
Pinecone API Key (String)
- Input
Source (String)
- Options
Pinecone Environment (String)
Pinecone Index Name (String)
- Output
Args (String)
Embeddings generation JSON string arguments. Includes keys - env, index_name,
namespace
Doc Content Array (String[])
Content of documents fetched from vector store
Doc Count (String)
Count of documents fetched from the vector store
C. Config Mapping
Global variables that is being used throughout the framework are being
mapped from the config.xlsx values in the 00_InitializeSource.xaml
II.2 Solution Architecture
Below is a high-level solution/technical architecture of the different gears or moving
parts powering up the solution. (Zoom in to 200% to view in more clarity)
II.3 Framework Components
Similar to REFramework, this template leverages state machines to flexibly handle
the flow for various types of use cases. The studio template includes some
components of REFramework stored in Framework\REFRamework folder, while the
other core framework components are stored in the root, Framework, Forms and
Data folder which are enumerated below:
a. LLMFramework.xaml – This framework revolves around standard usages of
LLM for various real-world use cases, which can generally be categorized as
Transactional (Input – Process – Output) and Conversational (Message –
Response)
b. Framework Folder – Core set of files that builds up the framework
b.1. 00_InitializeSource
Connect to various sources, convert into embeddings and store into a vector database, to be
used as context for the prompt template. Currently supported sources: CSV, DOCX, PDF,
YOUTUBE VIDEOS
b.2. 10_InteractLLM.xaml
Initialize Prompt Template, Fetch or Restore stored memory, (Re)Initialize LLM chain or
agent, LLM Interrogation, Interaction Flow Rules
b.3. 15_ActionCenter.xaml
Customizable action center form for your specific use case where the LLM output would
require human validation.
b.4. 20_PerformActions.xaml
Add process flows to interact with ANY downstream systems either through UI or API using
the response from the LLM.
b.5. 25_PostProcessResponse.xaml
Add process flows to perform any post processing step before completing the current
transaction or conversation.
c. Data Folder – Contains configuration files, temporary data, sample files
d. Forms Folder – Scenario specific. Conversational. Sample Forms
II.4 Reusable Components
Reusable components are stored in the Framework\Reusable components folder
which currently consists of the following:
e. AfterResumeWorkflow.xaml
Customizable Post Resume Workflow e.g.- Verifying task action/button
executed i.e. submit, reject, etc.- Post processing of validated data based on
the task action
f. InteractionFlowRulesValidation.xaml
Use this workflow to determine the next step after the LLM interaction.
IV. Quick Start Use Cases
IV.1 Conversational (Default)
Step1: In Project Settings , make sure ‘Supports Persistence’ is disabled.
Step 2: Publish Project to Orchestrator. Make sure the customizable fields are
filled up or just remove other samples xamls that a not relevant for this
deployment.
Step 3: Create process in Orchestrator. Select Sample-Chatmode.xaml as entry
point.
Step 4: Run from Assistant
Sample Scenario: Chat With Your Own Data
Sample-ChatMode.xaml
THIS CAN BE DEPLOYED AS IS, OR CUSTOMIZED. MARK THIS AS MAIN OR CHOOSE
THIS AS ENTRY POINT WHEN DEPLOYING (ATTENDED PROCESS). THIS REQUIRES
ROBOT VERSION >= 23.4
Leverage context aware LLM Framework with memory to ingest data source and
answer any questions (in a conversational manner) related or unrelated to the
ingested source.
Related Artifacts (Forms, Workflows, Triggers)
MAIN Framework -> Framework\LLMFramework.xaml
Framework Folder -> Framework\*
Forms & Triggers -> Forms\Forms-Chat\*
End Sample Scenario
IV.2 Transactional
Step 1: In Project Settings , make sure ‘Supports Persistence’ is enabled.
Step 2: Create process in Orchestrator. Select LLMFramework.xaml (Default Main)
as entry point.
Step 3: [IMPORTANT]: Create necessary Queues, Storage Buckets, Queue Trigger,
in Orchestrator as defined in Config.xlsx
Step 4: Run the process as unattended robot in Orchestrator and start consuming
queues.
Sample Scenario: AR PO Extraction & Analytics
Sample-TransactionMode1.xaml
-> Invoice Dispatcher -> Queue -> DU
-> Data Service -> LLM Framework [LLM Interaction (Analyze) > ERP (Act) > LLM
Powered AP Analytics Dashboard (Respond)]
Related Artifacts
MAIN Framework -> Framework\LLMFramework.xaml
Framework Folder -> Framework\*
End Sample Scenario
IV.3 Transactional with human in the loop
Step 1: In Project Settings , same as with Transactional, ‘Supports Persistence’ is
enabled.
Step 2: Also, ‘Wait for Form Task and Resume’ MUST BE ADDED to 1.5 wait for
human action state in the LLMFramework.Xaml (See Appendix: Known
Issues and Limitations for more details on the reason)
Step 2: Publish Project to Orchestrator
Step 3: Create process in Orchestrator. Select LLMFramework.xaml (Default Main)
as entry point.
Step 4 [IMPORTANT]: Create necessary Queues, Storage Buckets, Queue Trigger,
in Orchestrator as defined in Config.xlsx
Step 5: Run the process as unattended robot in Orchestrator and start consuming
queues.
Sample Scenario: Contracts Indemnification Review
Sample-TransactionMode2.xaml
THIS IS A SIMULATION OF A QUEUE BASED TRANSACTIONAL PROCESS.
LLMFRAMEWORK WORKS LIKE REFRAMEWORK AND DEPLOYED AS THE MAIN XAML.
AFTER RUNNING THIS DISPATCHER, RUN OR DEPLOY LLMFRAMEWORK.XAML
THIS DISPATCHER XAML SIMULATES below steps:
> Upload to Storage Bucket
> Add queue item
AFTER SIMULATING DISPATCHER, RUN LLMFRAMEWORK.xaml to consume the
transactions, using the following parameters
> in_TransactionMode = TRUE
> in_UseAgent = FALSE
> in_Query (Prompt Template) = “Identify if the document is a one-sided
indemnification or a mutual indemnification agreement? Provide examples from
the text to justify your answer. If both parties agree to indemnify each other it is
mutual indemnification. If only one party is required to indemnify the other (
example supplier indemnifies company, even if company has to notify the supplier)
it is one-sided indemnification. Provide examples of clauses to justify your answer.”
> in_Source = BLANK. Comes from GetTransaction Data - TransactionItem
Reference
Indemnification in a contract is a contractual clause that shifts liability, costs, and
expenses from one party to another. After the LLM agent is used to identify if an
MSA document has one-sided or two-sided indemnification (with extracted excerpts
from the contract) it will push to human for further review via the custom action
center form
Related Artifacts
MAIN Framework -> Framework\LLMFramework.xaml
Framework Folder -> Framework\*
End Sample Scenario
V. Appendix
V.1 Future Enhancements
➢ UiPath Integration Services Custom Connection for the LLM Framework
APIs/Activities
➢ Enable web page knowledge ingestion and cascading URL web scraping from
base URL.
➢ Prompt Customization/Pre-loaded prompts/prompt store
➢ Include other LLM options LLaMA2
➢ Include support for output parsers: unstructured input -> structured output
V.2 Known Issues and Limitations
As the framework is still in its nascent stage, there are still some known bugs and
limitations that are acknowledged, such as below.
➢ Persistence support needs to be enabled in Studio and the Wait activity needs to
be added when using the HITL flow, otherwise, adding the Wait activity without
Persistence enabled would keep the project from building or enabling
Persistence while using Chat Mode would throw Persistence Bookmarks error
during runtime.
V.3 Additional sources of documentation
Supporting guides and video tutorials
Additional Process Documentation
Full How to Power up your Automations with https://youtu.be/X0FmJoT8Mms
Guide GenAI using LLM Framework Template in
UiPath
Custom How to Setup Python API Backend for https://youtu.be/xCqTXhPVakY
LLM Framework Custom LangChain
Activity
Activities in UiPath
Setup
