####################################################################

Vector Theoritical Notes

####################################################################

=========================
Vector Embeddings
=========================

If you've ever used applications such as voice assistants, chatbots, language

translators, recommendation systems, anomaly detection, or video search and
recognition, you've implicitly used vector embeddings features.

Oracle AI Vector Search stores vector embeddings, which are mathematical vector
representations of data points.
These vector embeddings describe the semantic meaning behind content such as words,
documents, audio tracks, or images. As an example, while doing text based searches,
vector search is often considered better than keyword search as vector search is
based on the meaning and context behind the words and not the actual words
themselves.

Vector embeddings are a way of representing almost any kind of data, such as text,
images, videos, users, or music as points in a multidimensional space where the
locations of those points in space, and proximity to others, are semantically
meaningful.

=========================
Similarity Search
=========================

Searching semantic similarity in a data set is now equivalent to searching nearest

neighbors in a vector space instead of using traditional keyword searches.

Example:
~~~~~~~
The distance between dog and wolf in this vector space is shorter than the distance
between dog and kitten.
In this space, a dog is more similar to a wolf than it is to a kitten.

Vector data tends to be unevenly distributed and clustered into groups that are
semantically related.
Doing a similarity search based on a given query vector is equivalent to retrieving
the K-nearest vectors to your query vector in your vector space.

Basically, you need to find an ordered list of vectors by ranking them, where the
first row in the list is the closest or most similar vector to the query vector,
the second row in the list is the second closest vector to the query vector, and so
on.

When doing a similarity search, the relative order of distances is what really
matters rather than the actual distance.

Approximate searches using vector indexes can limit the searches to specific
clusters, whereas exact searches visit vectors across all clusters.

=========================
Vector Embedding Models
=========================

Vector embeddings are created via neural networks.

Most modern vector embeddings use a transformer model but convolutional neural
networks can also be used.

Depending on the type of your data, you can use different pretrained, open-source
models to create vector embeddings.

For example:
~~~~~~~~~~~
For textual data --> "sentence transformers" transform words, sentences, or
paragraphs into vector embeddings.
For visual data --> you can use Residual Network (ResNet) to generate vector
embeddings.
For audio data --> you can use the visual spectrogram representation of the audio
data to fall back into the visual data case.

Each model also determines the number of dimensions for your vectors.

For example:
~~~~~~~~~~~~
Cohere's embedding model 'embed-english-v3.0' has 1024 dimensions.
OpenAI's embedding model 'text-embedding-3-large' has 3072 dimensions.
Hugging Face's embedding model 'all-MiniLM-L6-v2' has 384 dimensions.

Of course, you can always create your own model that is trained with your own data
set.

==================================================
Import Embedding Models into Oracle Database
==================================================

You can generate vector embeddings outside the Oracle Database using pretrained
open-source embeddings models or your own embeddings models.

you also have the option to import those models directly into the Oracle Database
if they are compatible with the Open Neural Network Exchange (ONNX) standard.

Oracle Database implements an ONNX runtime directly within the database. This
allows you to generate vector embeddings directly within the Oracle Database using
SQL.

==================================================
Why Use Oracle AI Vector Search?
==================================================

One of the biggest benefits of Oracle AI Vector Search is that semantic search on
unstructured data can be combined with relational search on business data in one
single system.

This is not only powerful but also significantly more effective because you don't
need to add a specialized vector database, eliminating the pain of data
fragmentation between multiple systems.

==================================================
Oracle AI Vector Search Workflow
==================================================

Vector embeddings are generated by passing unstructured data through an embedding

model.
Vector embeddings can then be stored alongside business data in relational tables
and vector indexes can optionally be created.
Once you have the vector representations of your unstructured data stored in your
database table(s), a sample of unstructured data
can be passed through the embedding model to create a query vector.

With the query vector, you can perform similarity searches against the vectors that
are already stored in the database,
in combination with relational queries if desired.

To form a complete Retrieval Augmented Generation (RAG) pipeline, it is also

possible to make a call to a generative Large Language Model (LLM)
as part of the query step.

Primary workflow steps:

~~~~~~~~~~~~~~~~~~~~~~~
1. Generate Vector Embeddings from Your Unstructured Data
You can perform this step either outside or within Oracle Database.

2. Store Vector Embeddings, Unstructured Data, and Relational Business Data in

Oracle Database
You store the resulting vector embeddings and associated unstructured data with
your relational business data in Oracle Database.

3. Create Vector Indexes

You may want to create vector indexes on your vector embeddings. This is
beneficial for running similarity searches over huge vector spaces.
Vector indexes are a class of specialized indexing data structures that are
designed to accelerate similarity searches using high-dimensional vectors.
They use techniques such as clustering, partitioning, and neighbor graphs to
group vectors representing similar items, which drastically reduces the search
space, thereby making the search process extremely efficient.

4. Query Data with Similarity Searches

You can then use Oracle AI Vector Search native SQL operations to combine
similarity with relational searches to retrieve relevant data.

5. Generate a Prompt and Send it to an LLM for a Full RAG Inference

You can use the similarity search results to generate a prompt and send it to
your generative LLM of choice for a complete RAG pipeline.

==================================================
Generate Vector Embeddings
==================================================

You must generate vector embeddings from your unstructured data either outside or
within Oracle Database.
To get vector embeddings, you can either use ONNX embedding machine learning models
or access third-party REST APIs.

Put differently, You can generate vectors by calling services outside the database
or generate vectors directly from within the database after you have imported
pretrained embedding models.

***********************************************************************************
*
Import Pretrained Models in ONNX Format for Vector Generation Within the Database:
***********************************************************************************
*

You can download pretrained embedding machine learning models and then convert them
into ONNX format, if they are not already in ONNX format, and then import the ONNX
format models into the Oracle Database and then generate vector embeddings from
your data within the database.

-------------------------------------------
Import ONNX Models and Generate Embeddings
-------------------------------------------

If the pretrained model is in ONNX format already, then simply import it and
generate vector embeddings.

$ sqlplus sys/<password>@<pdb_service_name> as sysdba

SQL> GRANT DB_DEVELOPER_ROLE TO dmuser identified by <password>; (Use dmuser only
and not any other user)
SQL> GRANT create mining model TO dmuser;
SQL> CREATE OR REPLACE DIRECTORY DM_DUMP as '<work directory path>';
SQL> GRANT READ ON DIRECTORY dm_dump TO dmuser;
SQL> GRANT WRITE ON DIRECTORY dm_dump TO dmuser;
SQL> exec DBMS_VECTOR.DROP_ONNX_MODEL(model_name => 'doc_model', force => true);

SQL> CONN dmuser/<password>@<pdbname>;

SQL> EXECUTE DBMS_VECTOR.LOAD_ONNX_MODEL('DM_DUMP', 'my_embedding_model.onnx',
'doc_model'); <-- If the ONNX model to be imported already includes an output
tensor named embeddingOutput and an input string tensor named data, JSON metadata
is unnecessary. Embedding models converted from OML4Py follow this convention and
can be imported without the JSON metadata.

(or) Alternately, you can load the ONNX embedding model by specifying the JSON
metadata.

SQL> EXECUTE DBMS_VECTOR.LOAD_ONNX_MODEL('DM_DUMP', 'my_embedding_model.onnx',

'doc_model', JSON('{"function" : "embedding", "embeddingOutput" : "embedding",
"input": {"input": ["DATA"]}}'));

1. Query USER_MINING_MODEL_ATTRIBUTES view.

``````````````````````````````````````````````
SELECT model_name, attribute_name, attribute_type, data_type, vector_info
FROM user_mining_model_attributes
WHERE model_name = 'DOC_MODEL'
ORDER BY ATTRIBUTE_NAME;

2. Query USER_MINING_MODELS view.

`````````````````````````````````
SELECT MODEL_NAME, MINING_FUNCTION, ALGORITHM,
ALGORITHM_TYPE, MODEL_SIZE
FROM user_mining_models
WHERE model_name = 'DOC_MODEL'
ORDER BY MODEL_NAME;

3. Check model statistics and more details:

```````````````````````````````````````````
SELECT * FROM DM$VMDOC_MODEL ORDER BY NAME;
SELECT * FROM DM$VPDOC_MODEL ORDER BY NAME;
SELECT * FROM DM$VJDOC_MODEL;

4. Generate Embeddings:
```````````````````````
SELECT TO_VECTOR(VECTOR_EMBEDDING(doc_model USING 'hello' as data)) AS embedding;

Alternatively, using DBMS_DATA_MINING package also, you can import the Models.

CONN dmuser/<password>@<pdbname>;
DECLARE
m_blob BLOB default empty_blob();
m_src_loc BFILE ;
BEGIN
DBMS_LOB.createtemporary (m_blob, FALSE);
m_src_loc := BFILENAME('DM_DUMP', 'my_embedding_model.onnx');
DBMS_LOB.fileopen (m_src_loc, DBMS_LOB.file_readonly);
DBMS_LOB.loadfromfile (m_blob, m_src_loc, DBMS_LOB.getlength (m_src_loc));
DBMS_LOB.CLOSE(m_src_loc);
DBMS_DATA_MINING.import_onnx_model ('doc_model', m_blob, JSON('{"function" :
"embedding", "embeddingOutput" : "embedding", "input": {"input": ["DATA"]}}'));
DBMS_LOB.freetemporary (m_blob);
END;
/
(or)
DECLARE
model_source BLOB := NULL;
BEGIN
-- get BLOB holding onnx model
model_source := DBMS_CLOUD.GET_OBJECT(
credential_name => 'myCredential',
object_uri => 'https://objectstorage.us-phoenix -1.oraclecloud.com/' ||
'n/namespace -string/b/bucketname/o/myONNXmodel.onnx');

DBMS_DATA_MINING.IMPORT_ONNX_MODEL(
"myonnxmodel",
model_source,
JSON('{ function : "embedding" })
);
END;
/

-------------------------------------------
Convert Pretrained Models to ONNX Format
-------------------------------------------

If the pretrained model is NOT in ONNX format, then convert it to ONNX format so
that SQL understands it.

OML4Py enables the use of text transformers from Hugging Face by converting the
pretrained models into ONNX format models.
OML4Py also adds the necessary tokenization and post-processing.
The resulting ONNX pipeline is then imported into the database and can be used to
generate embeddings for AI Vector Search.

Note: This feature will only work on OML4Py client. It is not supported on the
OML4Py server.

If you do not have a pretrained embedding model in ONNX-format, Oracle offers a

Python utility package that downloads pretrained models from an external source,
converts the model to ONNX format augmented with pre-processing and post-processing
steps, and imports the resulting ONNX-format model into Oracle Database.
Use the DBMS_VECTOR.LOAD_ONNX_MODEL procedure to import the file as a mining model.
Then leverage the in-database ONNX Runtime with the ONNX model to produce vector
embeddings.

At a high level, the Python utility package performs the following tasks:
1. Downloads the pretrained model from external source to your system
2. Augments the model with pre-processing and post-processing steps and creates a
new ONNX model
3. Validates the augmented ONNX model
4. Loads into the database as a mining model or optionally exports to a file

The Python utility can take any of the models in the preconfigured list as input.
Alternatively, you can use the built-in template that contains common
configurations for certain groups of models such as text-based models.

Note:This feature is available with the OML4Py client only.

Currently, the Python package supports only the Text Transformers and the Model
size should be less than 1GB. If it is more than 1 GB, then try Quantization to
reduce the size. And, the Tokenizers must be either BERT, GPT2, SENTENCEPIECE, or
ROBERTA.

Preconfigured list of models are common models from external resource repositories
that are provided with the Python utility. The preconfigured models have an
existing specification. Users can create their own specification using the text
template as a starting point. To get a list of all model names in the preconfigured
list, you can use the show_preconfigured function.

The Python utility package provides built-in text template for you to configure the
pretrained models with pre-processing and post-processing operations. The template
has a default specification for the pretrained models. This specification can be
changed or augmented to create custom configurations. The text template uses Mean
Pooling and Normalization as post-processing operations by default.

The Python utility package provides the following classes:

1. EmbeddingModelConfig
2. EmbeddingModel

To use the Python utility, ensure that you have the following:
1. OML4Py Client running on Linux X64 for On-Premises Databases
2. Python 3.12 (the earlier versions are not compatible)

Conversion Code is as follows:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ python3
PYTHON PROMPT> from oml.utils import EmbeddingModel, EmbeddingModelConfig <--
load/import the python classes
PYTHON PROMPT> EmbeddingModelConfig.show_preconfigured() <-- To see the list of
preconfigured models.
PYTHON PROMPT> EmbeddingModelConfig.show_templates() <--- To see the list of
available templates.

-- Choose any of the below methods for conversion

-- Method1: Generate an ONNX file from the preconfigureded model "sentence-
transformers/all-MiniLM-L6-v2"
PYTHON PROMPT> em = EmbeddingModel(model_name="sentence-transformers/all-MiniLM-L6-
v2")
PYTHON PROMPT> em.export2file("your_preconfig_file_name",output_dir=".")

-- Method2: Generate an ONNX model from the preconfigured model "sentence-

transformers/all-MiniLM-L6-v2" in the database:
PYTHON PROMPT> em = EmbeddingModel(model_name="sentence-transformers/all-MiniLM-L6-
v2")
PYTHON PROMPT> em.export2db("your_preconfig_model_name")

Note:ONNX files are only created when export2file is used. If export2db is used, no
ONNX files will be generated.
Note: The Python utility package validates the embedding text model before you can
run them using ONNX Runtime. Oracle supports ONNX embedding models that conform to
string as input and float32 [<vector dimension>] as output.

Note: DBMS_VECTOR.LOAD_ONNX_MODEL or DBMS_DATA_MINING.IMPORT_ONNX_MODEL are only

needed if export2file was used instead of export2db.

-- Method3: Generate an ONNX file using the provided text template:

PYTHON PROMPT> config =
EmbeddingModelConfig.from_template("text",max_seq_length=512)
PYTHON PROMPT> em = EmbeddingModel(model_name="intfloat/e5-small-v2",config=config)
PYTHON PROMPT> em.export2file("your_template_file_name",output_dir=".")
PYTHON PROMPT> exit()

--------------------------------------------------------------
Python Classes to Convert Pretrained Models to ONNX Models
--------------------------------------------------------------

Explore the functions and attributes of the EmbeddingModelConfig class and

EmbeddingModel class within Python. These classes are designed to configure
pretrained embedding models.

The EmbeddingModelConfig class contains the properties required for the package to
perform downloading, exporting, augmenting, validation, and storing of an ONNX
model. The class provides access to configuration properties using the dot
operator. As a convenience, well-known configurations are provided as templates.

This example illustrates the preconfigured embedding model that comes with the
Python package. You can use this model without any additional configurations.

"sentence-transformers/distiluse-base-multilingual-cased-v2": {
"max_seq_length": 128,
"do_lower_case": false,
"post_processors":[{"name":"Pooling","type":"mean"},
{"name":"Dense","in_features":768, "out_features":512, "bias":true,
"activation_function":"Tanh"}],
"quantize_model":true,
"distance_metrics": ["COSINE"],
"languages": ["ar", "bg", "ca", "cs", "dk", "d", "us", "el", "et", "fa",
"sf", "f", "frc", "gu", "iw", "hi", "hr", "hu", "hy", "in", "i", "ja", "ko", "lt",
"lv", "mk", "mr", "ms", "n", "nl", "pl", "pt", "ptb", "ro", "ru", "sk", "sl", "sq",
"lsr", "s", "th", "tr", "uk", "ur", "vn", "zhs", "zht"]
}

***********************************************************************************
*
Generate Vector Embeddings Using Vector Utilities Leveraging Third-Party REST APIs:
***********************************************************************************
*

Oracle AI Vector Search offers vector utilities (SQL and PL/SQL tools) to
automatically generate vector embeddings from your unstructured data.
You can call either Vector Utility SQL functions or Vector Utility PL/SQL packages
to transform unstructured data into vector embeddings.

--------------------------------------------------------------
Understand the Stages of Data Transformations
--------------------------------------------------------------
Your input data may travel through different stages before turning into a vector

For example, you may first transform textual data (such as a PDF document) into
plain text, then break the resulting text into smaller pieces of text (chunks) to
finally create vector embeddings on each chunk.

Prepare: Text and Chunks

~~~~~~~~~~~~~~~~~~~~~~~~
This stage prepares your unstructured data to ensure that it is in a format that
can be processed by vector embedding models.

To prepare large unstructured textual data (for example, a PDF or Word document),
you first transform the data into plain text and then pass the resulting text
through Chunker. The chunker then splits the text into smaller chunks using a
process known as chunking. A single document may be split into many chunks, each
transformed into a vector.

Chunks are pieces of words (to capture specific words or word pieces), sentences
(to capture a specific meaning), or paragraphs (to capture broader themes). Later,
you will learn about several chunking parameters and techniques to define so that
each chunk contains relevant and meaningful context.

Embed: Tokens and Vectors

~~~~~~~~~~~~~~~~~~~~~~~~~~
You now pass the extracted chunks as input to Model (a declared vector embedding
model) for generating vector embeddings on each chunk. This stage makes these
chunks searchable.

The chunks are first passed through Tokenizer of the model to split into words or
word pieces, known as tokens. The model then embeds each token into a vector
representation.

Tokenizers used by embedding models frequently have limits on the size of the input
text they can deal with, so chunking is needed to avoid loss of text when
generating embeddings. If input text is larger than the maximum input limit imposed
by the model, then the text gets truncated unless it is split up into appropriate-
sized segments or chunks. Chunking is not needed for smaller documents, text
strings, or summarized texts that meet this maximum input limit.

The chunker must select a text size that approximates the maximum input limit of
your model. The actual number of tokens depends on the specified tokenizer for the
model, which typically uses a vocabulary list of words, numbers, punctuations, and
pieces of tokens.

A vocabulary contains a set of tokens that are collected during a statistical

training process. Each tokenizer uses a different vocabulary format to process text
into tokens, that is, the BERT multilingual model uses Word-Piece encoding and the
GPT model uses Byte-Pair encoding.
For example, the BERT model tokenizes the following four words:
Embedding usecase for chunking

as the following eight tokens and also includes ## (number signs) to indicate non-
initial pieces of words:
Em ##bedd ##ing use ##case for chunk ##ing

Vocabulary files are included as part of a model's distribution. You can supply a
vocabulary file (recognized by your model's tokenizer) to the chunker beforehand,
so that it can correctly estimate the token count of your input data when chunking.

Populate and Query: Vector Indexes

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Finally, you store the semantic content of your input data as vectors in Vector
Index. You can now implement combined similarity and relational queries to retrieve
relevant results using the extracted vectors.

--------------------------------------------------------------
Use SQL Functions to Generate Embeddings
--------------------------------------------------------------
Choose to implement Vector Utility SQL functions to perform parallel or on-the-fly
chunking and embedding operations, within Oracle Database.

Choose to implement Vector Utility SQL functions to perform parallel or on-the-fly

chunking and embedding operations, within Oracle Database.

Vector Utility SQL functions are intended for a direct and quick interaction with
data, within pure SQL.

To get chunks, this function uses the in-house implementation with Oracle Database.
To get an embedding, this function uses ONNX embedding models that you load into
the database (and not third-party REST providers).

VECTOR_CHUNKS
~~~~~~~~~~~~~
Use the VECTOR_CHUNKS SQL function if you want to split plain text into chunks
(pieces of words, sentences, or paragraphs) in preparation for the generation of
embeddings, to be used with a vector index.

For example, you can use this function to build a standalone Text Chunking system
that lets you break down a large amount of text into smaller yet semantically
meaningful chunks. You can experiment with your chunks by inspecting and
accordingly amending the chunking results and then proceed further.

For detailed information, see VECTOR_CHUNKS in Oracle Database SQL Language

Reference.

VECTOR_EMBEDDING
~~~~~~~~~~~~~~~~~
Use the VECTOR_EMBEDDING function if you want to generate a single vector embedding
for different data types.

For example, you can use this function in information-retrieval applications or

chatbots, where you can generate a query vector on the fly from a user's natural
language text input.

For detailed information, see VECTOR_EMBEDDING in Oracle Database SQL Language

Reference.
--------------------------------------------------------------
Use PL/SQL Packages to Generate Embeddings
--------------------------------------------------------------
Choose to implement Vector Utility PL/SQL packages to perform chunking, embedding,
and text generation operations along with text processing and similarity search,
within and outside Oracle Database.

Vector Utility PL/SQL APIs work with both the ONNX embedding models (loaded into
the database) and third-party REST providers, such as Cohere, Google AI, Hugging
Face, Oracle Cloud Infrastructure (OCI) Generative AI, OpenAI, or Vertex AI.

Supplied Chainable Utility Functions

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
These are intended to be a set of chainable and flexible "stages" through which you
pass your input data to transform into a different representation, including
vectors.

You can combine a set of chainable utility (UTL) functions together in an end-to-
end pipeline.

Each pipeline or transformation chain can include a single function or a

combination of functions, which are applied to source documents as they are
transformed into other representations (text, chunks, summary, or vector). These
functions are chained together, such that the output from one function is used as
an input for the next.

Each chainable utility function performs a specific task of transforming data into
other representations, such as converting data to text, converting text to chunks,
or converting the extracted chunks to embeddings.

At a high level, the supplied chainable utility functions include:

UTL_TO_TEXT() --> Converts a document (for example, MS Word, HTML, or

PDF) to plain text
UTL_TO_CHUNKS() --> Converts plain text to chunks. Calls the VECTOR_CHUNKS
SQL function under the hood.
UTL_TO_EMBEDDING() --> Converts plain text to a single embedding (VECTOR).
Calls the ONNX embedding model that you load into the database. It can also Call
the specified third-party embedding model through REST API
UTL_TO_EMBEDDINGS() --> Converts an array of chunks (VECTOR_ARRAY_T) to an
array of embeddings (VECTOR_ARRAY_T)
UTL_TO_SUMMARY() --> Converts plain text to a summary
UTL_TO_GENERATE_TEXT() --> Generates text for a prompt or input string. Calls the
specified third-party text generation model.

Chainable utility functions are designed to be flexible and modular. You can create
transformation chains in various sequences, depending on your use case.

For example, you can directly extract vectors from a PDF file by creating a chain
of the UTL_TO_TEXT, UTL_TO_CHUNKS, and UTL_TO_EMBEDDINGS chainable utility
functions.

As shown in the following diagram, a file-to-text-to-chunks-to-embeddings chain

performs a set of operations in this order:
1. Converts a PDF file to plain text (using UTL_TO_TEXT)
2. Splits the resulting text into appropriate-sized chunks (using UTL_TO_CHUNKS)
3. Generates vector embeddings on each chunk (using UTL_TO_EMBEDDINGS)

FILE --> TEXT --> CHUNKS --> EMBEDDINGS

Common Use Cases:
`````````````````
Document to vectors:
^^^^^^^^^^^^^^^^^^^
A common use case might be to automatically generate vectors from documents. You
can convert a set of documents to plain text, split the resulting text into smaller
chunks to finally generate embeddings on each chunk, in a single file-to-text-to-
chunks-to-embeddings chain

Document to vectors, with chunking and summarization:

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Another use case might be to generate a short summary of documents and then
automatically extract vectors from that summary.

After generating the summary, you can either generate a single vector (using
UTL_TO_EMBEDDING) or chunk it and then generate multiple vectors (using
UTL_TO_EMBEDDINGS).

You can convert the document to plain text, summarize the text into a concise gist
to finally create a single embedding on the summarized text, in a file-to-text-to-
summary-to-embedding chain.

You can convert the document to plain text, summarize the text into a gist, split
the gist into chunks to finally create multiple embeddings on each summarized
chunk, in a file-to-text-to-summary-to-chunks-to-embeddings chain.

While both the chunking and summarization techniques make text smaller, they do so
in different ways. Chunking just breaks the text into smaller pieces, whereas
summarization extracts a salient meaning and context of that text into free-form
paragraphs or bullet points.

By summarizing the entire document and appending the summary to each chunk, you get
the best of both worlds, that is, an individual piece that also has a high-level
understanding of the overall document.

Prompt to text:
^^^^^^^^^^^^^^
You can generate a response directly based on a prompt.
A prompt can be an input string, such as a question that you ask a Large Language
Model (LLM). For example, "What is Oracle Text?".
A prompt can also be a command, such as "Summarize the following ...", "Draft an
email asking for ...", or "Rewrite the following ...", and can include results from
a search.

Other usecases are Text to vector (text-to-embedding chain), Text to chunks (text-
to-chunks chain), Text to summary (text-to-summary chain).

About Vector Helper Procedures

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Vector helper procedures let you configure authentication credentials and language-
specific data, for use in chainable utility functions.
At a high level, the supplied vector helper procedures include

Credential helper procedures to securely manage authentication credentials, which

are used to access third-party providers when making REST API calls

CREATE_CREDENTIAL --> Creates a credential name for securely storing user

authentication credentials in the database.
DROP_CREDENTIAL --> Drops an existing credential name.

Chunker helper procedures to manage custom vocabulary and language data, which are
used when chunking user data
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CREATE_VOCABULARY --> Loads your own vocabulary file into the database.
DROP_VOCABULARY --> Removes the specified vocabulary data from the
database.
CREATE_LANG_DATA --> Loads your own language data file (abbreviation tokens) into
the database.
DROP_LANG_DATA --> Removes abbreviation data for a given language from the
database.

Supplied Vector Utility PL/SQL Packages

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Use either a lightweight DBMS_VECTOR package or a more advanced DBMS_VECTOR_CHAIN
package with full capabilities.

DBMS_VECTOR:
This package simplifies common operations with Oracle AI Vector Search, such as
chunking text into smaller segments, extracting vector embeddings from user data,
or generating text for a given prompt.

Refer the Table Chart to see what can be called within Database and outside
Database.
https://docs.oracle.com/en/database/oracle/oracle-database/23/vecse/supplied-
vector-utility-pl-sql-packages.html

DBMS_VECTOR_CHAIN:
This package provides chunking and embedding functions along with some text
generation and summarization capabilities. It is more suitable for text processing
with similarity search, using functionality that can be pipelined together for an
end-to-end search.

This package requires you to install the CONTEXT component of Oracle Text, an
Oracle Database technology that provides indexing, term extraction, text analysis,
text summarization, word and theme searching, and other utilities.

Refer the Table Chart to see what can be called within Database and outside
Database.
https://docs.oracle.com/en/database/oracle/oracle-database/23/vecse/supplied-
vector-utility-pl-sql-packages.html

Due to underlying dependance on the text processing capabilities of Oracle Text,

note that both the UTL_TO_TEXT and UTL_TO_SUMMARY chainable utility functions and
all the chunker helper procedures are available only in this package through Oracle
Text.

Supported Third-Party Provider Operations

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The supported third-party REST providers are:
* Cohere
* Generative AI
* Google AI
* Hugging Face
* OpenAI
 * Vertex AI

The corresponding REST calls allowed for each operation are:

* DBMS_VECTOR.UTL_TO_EMBEDDING and DBMS_VECTOR.UTL_TO_EMBEDDINGS
* DBMS_VECTOR_CHAIN.UTL_TO_EMBEDDING and DBMS_VECTOR_CHAIN.UTL_TO_EMBEDDINGS
* DBMS_VECTOR_CHAIN.UTL_TO_SUMMARY
* DBMS_VECTOR.UTL_TO_GENERATE_TEXT
* DBMS_VECTOR_CHAIN.UTL_TO_GENERATE_TEXT

---------------------------------------
Validate JSON Input Parameters
---------------------------------------

You can optionally validate the structure of your JSON input to the DBMS_VECTOR.UTL
and DBMS_VECTOR_CHAIN.UTL functions, which use JSON to define their input
parameters.

The JSON data is schemaless, so the amount of validation that Vector Utility
package APIs do at runtime is minimal for better performance. The APIs validate
only the mandatory JSON parameters, that is, the parameters that you supply for the
APIs to run (not optional JSON parameters and attributes).

Before calling an API, you can use subprograms in the DBMS_JSON_SCHEMA package to
test whether the input data to be specified in the PARAMS clause is valid with
respect to a given JSON schema. This offers more flexibility and also ensures that
only schema-valid data is inserted in a JSON column.

Refer to the list of JSON input parameters in the link:

https://docs.oracle.com/en/database/oracle/oracle-database/23/vecse/validate-json-
input-parameters.html

==================================================
Generate Vector Embeddings (Examples)
==================================================

Convert Text String to Embedding

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can vectorize text strings like this for chatbots or information-retrieval
applications, where you want to directly convert a user's input text to a query
vector and then run it against vector index for a fast similarity search.

You can perform a text-to-embedding transformation using the UTL_TO_EMBEDDING

PL/SQL API (note the singular "embedding") or the VECTOR_EMBEDDING SQL function.
Both UTL_TO_EMBEDDING and VECTOR_EMBEDDING directly return a VECTOR type (not an
array).

If you want to access a third-party embedding model, then you can use
UTL_TO_EMBEDDING from either the DBMS_VECTOR or DBMS_VECTOR_CHAIN package.
If you are using an ONNX format embedding model, then you can use both
UTL_TO_EMBEDDING and VECTOR_EMBEDDING functions.

Example: To generate a vector embedding with "hello" as the input

------------------------------------------------------------------

DROP USER docuser cascade;

CREATE USER docuser identified by docuser DEFAULT TABLESPACE tbs1 quota unlimited
on tbs1;
GRANT DB_DEVELOPER_ROLE, create credential to docuser;
CONN docuser/password@CDB_PDB
EXEC UTL_HTTP.SET_PROXY('<proxy-hostname>:<proxy-port>');

BEGIN
DBMS_NETWORK_ACL_ADMIN.APPEND_HOST_ACE(
host => '*',
ace => xs$ace_type(privilege_list => xs$name_list('connect'),
principal_name => 'docuser',
principal_type => xs_acl.ptype_db));
END;
/

Using Cohere, Google AI, Hugging Face, OpenAI, and Vertex AI:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-- Run DBMS_VECTOR.CREATE_CREDENTIAL to create and store a credential.
-- Cohere, Google AI, Hugging Face, OpenAI, and Vertex AI require the following
authentication parameters/syntax:
-- { "access_token": "<access token>" }

exec dbms_vector.drop_credential('<credential name>');

declare
jo json_object_t;
begin
jo := json_object_t();
jo.put('access_token', 'AbabA1B123aBc123AbabAb123a1a2ab');
dbms_vector.create_credential(
credential_name => 'HF_CRED',
params => json(jo.to_string));
end;
/

var params clob;

exec :params := '
{
"provider": "<REST provider>",
"credential_name": "<credential name>",
"url": "<REST endpoint URL for embedding service>",
"model": "<embedding model name>"
}';

select dbms_vector.utl_to_embedding('hello', json(:params)) from dual

(or)
declare
input clob;
params clob;
v vector;
begin
input := 'hello';
params := '
{
"provider": "<REST provider>",
"credential_name": "<credential name>",
"url": "<REST endpoint URL for embedding service>",
"model": "<embedding model name>"
}';
v := dbms_vector.utl_to_embedding(input, json(params));
dbms_output.put_line(vector_serialize(v));
exception
 when OTHERS THEN
DBMS_OUTPUT.PUT_LINE (SQLERRM);
DBMS_OUTPUT.PUT_LINE (SQLCODE);
end;
/

Note: Replace the provider, credential_name, url, and model values

Refer to the link for actual json syntax for different providers
https://docs.oracle.com/en/database/oracle/oracle-database/23/vecse/convert-text-
string-embedding.html

Using Generative AI:

^^^^^^^^^^^^^^^^^^^
Run DBMS_VECTOR.CREATE_CREDENTIAL to create and store an OCI credential (OCI_CRED).

Generative AI requires the following authentication parameters:

{
"user_ocid": "<user ocid>",
"tenancy_ocid": "<tenancy ocid>",
"compartment_ocid": "<compartment ocid>",
"private_key": "<private key>",
"fingerprint": "<fingerprint>"
}

exec dbms_vector.drop_credential('OCI_CRED');
declare
jo json_object_t;
begin
jo := json_object_t();
jo.put('user_ocid','<user ocid>');
jo.put('tenancy_ocid','<tenancy ocid>');
jo.put('compartment_ocid','<compartment ocid>');
jo.put('private_key','<private key>');
jo.put('fingerprint','<fingerprint>');
dbms_output.put_line(jo.to_string);
dbms_vector.create_credential(
credential_name => 'OCI_CRED',
params => json(jo.to_string));
end;
/

var params clob;

exec :params := '
{
"provider": "ocigenai",
"credential_name": "OCI_CRED",
"url": "<REST endpoint URL for embedding service>",
"model": "<REST provider embedding model name>"
}';
select dbms_vector.utl_to_embedding('hello', json(:params)) from dual;
(or)
declare
input clob;
params clob;
v vector;
begin
input := 'hello;
params := '
{
 "provider": "ocigenai",
"credential_name": "OCI_CRED",
"url": "<REST endpoint URL for embedding service>",
"model": "<REST provider embedding model name>"
}';
v := dbms_vector.utl_to_embedding(input, json(params));
dbms_output.put_line(vector_serialize(v));
exception
when OTHERS THEN
DBMS_OUTPUT.PUT_LINE (SQLERRM);
DBMS_OUTPUT.PUT_LINE (SQLCODE);
end;
/

Note: Replace the url and model values. Optionally, you can specify additional REST
provider-specific parameters

If you are using a declared embedding model, then call either VECTOR_EMBEDDING or
UTL_TO_EMBEDDING
a. Load your ONNX model into Oracle Database.
b. Call VECTOR_EMBEDDING or UTL_TO_EMBEDDING:
SELECT TO_VECTOR(VECTOR_EMBEDDING(doc_model USING 'hello' as data)) AS
embedding;
(or)
var params clob; exec :params := '{"provider":"database", "model":"doc_model"}';
select dbms_vector.utl_to_embedding('hello', json(:params)) from dual;

Convert File to Text to Chunks to Embeddings

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Using a set of functions from the DBMS_VECTOR_CHAIN package, you first convert a
PDF file to text (UTL_TO_TEXT), split the text into chunks (UTL_TO_CHUNKS), and
then create vector embeddings on each chunk (UTL_TO_EMBEDDINGS).

To generate embeddings using a declared embedding model, through step-by-step

transformation chains:
-----------------------------------------------------------------------------------
------------------
drop table documentation_tab purge;
CREATE TABLE documentation_tab (id number, data blob);
INSERT INTO documentation_tab values(1, to_blob(bfilename('VEC_DUMP', 'database-
concepts23ai.pdf')));
commit;
SELECT dbms_lob.getlength(t.data) from documentation_tab t;
SELECT dbms_vector_chain.utl_to_text(dt.data) from documentation_tab dt;
SELECT ct.* from documentation_tab dt,
dbms_vector_chain.utl_to_chunks(dbms_vector_chain.utl_to_text(dt.data)) ct;
SELECT ct.* from documentation_tab dt,
dbms_vector_chain.utl_to_chunks(dbms_vector_chain.utl_to_text(dt.data), JSON('{"by"
: "words", "max" : "100", "overlap" : "0", "split" : "recursively", "language" :
"american", "normalize" : "all"}')) ct;
EXECUTE dbms_vector.drop_onnx_model(model_name => 'doc_model', force => true);
EXECUTE dbms_vector.load_onnx_model('VEC_DUMP', 'my_embedding_model.onnx',
'doc_model', JSON('{"function" : "embedding", "embeddingOutput" : "embedding" ,
"input": {"input": ["DATA"]}}'));

Replace my_embedding_model.onnx with an ONNX export of your embedding model.

Here, doc_model specifies the name under which the imported model is stored in
Oracle Database.
var embed_params clob;
exec :embed_params := '{"provider":"database", "model":"doc_model"}';
SELECT et.* from
documentation_tab dt,
dbms_vector_chain.utl_to_embeddings(
dbms_vector_chain.utl_to_chunks(dbms_vector_chain.utl_to_text(dt.data)),
json(:embed_params)) et;

Convert File to Embeddings

~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can directly extract vector embeddings from a PDF document, using a single-step
statement.

This statement creates a relational table (doc_chunks) from unstructured text

chunks and the corresponding vector embeddings:

CREATE TABLE doc_chunks as

(select dt.id doc_id, et.embed_id, et.embed_data, to_vector(et.embed_vector)
embed_vector
from
documentation_tab dt,
dbms_vector_chain.utl_to_embeddings(
dbms_vector_chain.utl_to_chunks(dbms_vector_chain.utl_to_text(dt.data),
json('{"normalize":"all"}')),
json('{"provider":"database", "model":"doc_model"}')) t,
JSON_TABLE(t.column_value, '$[*]' COLUMNS (embed_id NUMBER PATH '$.embed_id',
embed_data VARCHAR2(4000) PATH '$.embed_data', embed_vector CLOB PATH
'$.embed_vector')) et
);

Generate and Use Embeddings for End-to-End Search

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In this example, you first generate embeddings from textual content by using an
ONNX model, and then populate and query a vector index. At query time, you also
vectorize the query criteria on the fly
This example covers the entire workflow of Oracle AI Vector Search.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
conn sys/password@CDB_PDB as sysdba

CREATE TABLESPACE tbs1

DATAFILE 'tbs5.dbf' SIZE 20G AUTOEXTEND ON
EXTENT MANAGEMENT LOCAL
SEGMENT SPACE MANAGEMENT AUTO;

drop user docuser cascade;

create user docuser identified by docuser DEFAULT TABLESPACE tbs1 quota unlimited
on tbs1;
grant DB_DEVELOPER_ROLE to docuser;

create or replace directory VEC_DUMP as '/my_local_dir/';

grant read, write on directory VEC_DUMP to docuser;
commit;

conn docuser/password@CDB_PDB;
SET ECHO ON
SET FEEDBACK 1
SET NUMWIDTH 10
SET LINESIZE 80
SET TRIMSPOOL ON
SET TAB OFF
SET PAGESIZE 10000
SET LONG 10000

drop table documentation_tab purge;

create table documentation_tab (id number, text clob);

insert into documentation_tab values (1,

'Analytics empowers business analysts and consumers with modern, AI-powered,
self-service analytics capabilities for data preparation, visualization, enterprise
reporting, augmented analysis, and natural language processing.
Oracle Analytics Cloud is a scalable and secure public cloud service that
provides capabilities to explore and perform collaborative analytics for you, your
workgroup, and your enterprise.

Oracle Analytics Cloud is available on Oracle Cloud Infrastructure Gen 2 in

several regions in North America, EMEA, APAC, and LAD when you subscribe through
Universal Credits. You can subscribe to Professional Edition or Enterprise
Edition.');

insert into documentation_tab values (3,

'Generative AI Data Science is a fully managed and serverless platform for data
science teams to build, train, and manage machine learning models in the Oracle
Cloud Infrastructure.');

insert into documentation_tab values (4,

'Language allows you to perform sophisticated text analysis at scale. Using the
pretrained and custom models, you can process unstructured text to extract insights
without data science expertise.
Pretrained models include sentiment analysis, key phrase extraction, text
classification, and named entity recognition. You can also train custom models for
named entity recognition and text
classification with domain specific datasets. Additionally, you can translate
text across numerous languages.');

insert into documentation_tab values (5,

'When you work with Oracle Cloud Infrastructure, one of the first steps is to
set up a virtual cloud network (VCN) for your cloud resources. This topic gives you
an overview of Oracle Cloud
Infrastructure Networking components and typical scenarios for using a VCN. A
virtual, private network that you set up in Oracle data centers. It closely
resembles a traditional network, with
firewall rules and specific types of communication gateways that you can
choose to use. A VCN resides in a single Oracle Cloud Infrastructure region and
covers one or more CIDR blocks
(IPv4 and IPv6, if enabled). See Allowed VCN Size and Address Ranges. The
terms virtual cloud network, VCN, and cloud network are used interchangeably in
this documentation.
For more information, see VCNs and Subnets.');

insert into documentation_tab values (6,

'NetSuite banking offers several processing options to accurately track your
income. You can record deposits to your bank accounts to capture customer payments
and other monies received in the
course of doing business. For a deposit, you can select payments received for
existing transactions, add funds not related to transaction payments, and record
any cash received back from the bank.');
commit;

EXECUTE dbms_vector.drop_onnx_model(model_name => 'doc_model', force => true);

EXECUTE dbms_vector.load_onnx_model(
'VEC_DUMP',
'my_embedding_model.onnx',
'doc_model',
json('{"function" : "embedding", "embeddingOutput" : "embedding" , "input":
{"input": ["DATA"]}}')
);

create table doc_chunks as (

SELECT d.id id,
row_number() over (partition by d.id order by d.id) chunk_id,
vc.chunk_offset chunk_offset,
vc.chunk_length chunk_length,
vc.chunk_text chunk,
vector_embedding(doc_model using vc.chunk_text as data) vector
FROM documentation_tab d,
vector_chunks(d.text by words max 100 overlap 10 split RECURSIVELY) vc
);

desc doc_chunks;
set linesize 100
set long 1000
col id for 999
col chunk_id for 99999
col chunk_offset for 99999
col chunk_length for 99999
col chunk for a30
col vector for a100
select id, chunk_id, chunk_offset, chunk_length, chunk from doc_chunks;
select vector from doc_chunks where rownum <= 1;

create vector index vidx on doc_chunks (vector)

organization neighbor partitions
with target accuracy 95
distance EUCLIDEAN parameters (
type IVF,
neighbor partitions 2);

select id, vector_distance(

vector,
vector_embedding(doc_model using 'machine learning models' as data),
EUCLIDEAN) results
FROM doc_chunks order by results;

select id, vector_distance(

vector,
vector_embedding(doc_model using 'gen ai' as data),
EUCLIDEAN) results
FROM doc_chunks order by results;

select id, vector_distance(

vector,
vector_embedding(doc_model using 'computing networks' as data),
MANHATTAN) results
FROM doc_chunks order by results;
select id, vector_distance(
vector,
vector_embedding(doc_model using 'banking, money' as data),
MANHATTAN) results
FROM doc_chunks order by results;
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

==================================================
Perform Text Processing: PL/SQL Examples
==================================================

Convert Text String to Summary

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
conn sys/password@CDB_PDB as sysdba

CREATE TABLESPACE tbs1

DATAFILE 'tbs5.dbf' SIZE 20G AUTOEXTEND ON
EXTENT MANAGEMENT LOCAL
SEGMENT SPACE MANAGEMENT AUTO;

DROP USER docuser cascade;

CREATE USER docuser identified by docuser DEFAULT TABLESPACE tbs1 quota unlimited
on tbs1;
GRANT DB_DEVELOPER_ROLE, create credential to docuser;

CONN docuser/password@CDB_PDB

SET ECHO ON
SET FEEDBACK 1
SET NUMWIDTH 10
SET LINESIZE 80
SET TRIMSPOOL ON
SET TAB OFF
SET PAGESIZE 10000
SET LONG 10000
EXEC UTL_HTTP.SET_PROXY('<proxy-hostname>:<proxy-port>');

BEGIN
DBMS_NETWORK_ACL_ADMIN.APPEND_HOST_ACE(
host => '*',
ace => xs$ace_type(privilege_list => xs$name_list('connect'),
principal_name => 'docuser',
principal_type => xs_acl.ptype_db));
END;
/

exec dbms_vector_chain.drop_credential('<credential name>');

declare
jo json_object_t;
begin
jo := json_object_t();
jo.put('access_token', '<access token>');
dbms_vector_chain.create_credential(
 credential_name => '<credential name>',
params => json(jo.to_string));
end;
/

declare
jo json_object_t;
begin
jo := json_object_t();
jo.put('access_token', 'AbabA1B123aBc123AbabAb123a1a2ab');
dbms_vector_chain.create_credential(
credential_name => 'HF_CRED',
params => json(jo.to_string));
end;
/

-- select example

var params clob;

exec :params := '
{
"provider": "<REST provider>",
"credential_name": "<credential name>",
"url": "<REST endpoint URL for text summarization service>",
"model": "<REST provider text summarization model name>"
}';

select dbms_vector_chain.utl_to_summary(
'A transaction is a logical, atomic unit of work that contains one or more SQL
statements.
An RDBMS must be able to group SQL statements so that they are either all
committed, which means they are applied to the database, or all rolled back,
which
means they are undone.
An illustration of the need for transactions is a funds transfer from a savings
account to
a checking account. The transfer consists of the following separate operations:
1. Decrease the savings account.
2. Increase the checking account.
3. Record the transaction in the transaction journal.
Oracle Database guarantees that all three operations succeed or fail as a unit.
For
example, if a hardware failure prevents a statement in the transaction from
executing,
then the other statements must be rolled back.
Transactions set Oracle Database apart from a file system. If you
perform an atomic operation that updates several files, and if the system fails
halfway
through, then the files will not be consistent. In contrast, a transaction
moves an
Oracle database from one consistent state to another. The basic principle of a
transaction is "all or nothing": an atomic operation succeeds or fails as a
whole.',
json(:params)) from dual;

-- PL/SQL example

declare
input clob;
 params clob;
output clob;
begin
input := 'A transaction is a logical, atomic unit of work that contains one or
more SQL
statements.
An RDBMS must be able to group SQL statements so that they are either all
committed, which means they are applied to the database, or all rolled back,
which
means they are undone.
An illustration of the need for transactions is a funds transfer from a savings
account to
a checking account. The transfer consists of the following separate operations:
1. Decrease the savings account.
2. Increase the checking account.
3. Record the transaction in the transaction journal.
Oracle Database guarantees that all three operations succeed or fail as a unit.
For
example, if a hardware failure prevents a statement in the transaction from
executing,
then the other statements must be rolled back.
Transactions set Oracle Database apart from a file system. If you
perform an atomic operation that updates several files, and if the system fails
halfway
through, then the files will not be consistent. In contrast, a transaction
moves an
Oracle database from one consistent state to another. The basic principle of a
transaction is "all or nothing": an atomic operation succeeds or fails as a
whole.';
params := '
{
"provider": "<REST provider>",
"credential_name": "<credential name>",
"url": "<REST endpoint URL for text summarization service>",
"model": "<REST provider text summarization model name>"
}';
output := dbms_vector_chain.utl_to_summary(input, json(params));
dbms_output.put_line(output);
if output is not null then
dbms_lob.freetemporary(output);
end if;
exception
when OTHERS THEN
DBMS_OUTPUT.PUT_LINE (SQLERRM);
DBMS_OUTPUT.PUT_LINE (SQLCODE);
end;
/

Note: Replace the provider, credential_name, url, and model values as per your
Provider like Cohere, Google AI, Hugging Face, OpenAI, Vertex AI

For Generative AI, follow the below code:

{
"user_ocid": "<user ocid>",
"tenancy_ocid": "<tenancy ocid>",
"compartment_ocid": "<compartment ocid>",
"private_key": "<private key>",
"fingerprint": "<fingerprint>"
}

exec dbms_vector_chain.drop_credential('OCI_CRED');
declare
jo json_object_t;
begin
jo := json_object_t();
jo.put('user_ocid','<user ocid>');
jo.put('tenancy_ocid','<tenancy ocid>');
jo.put('compartment_ocid','<compartment ocid>');
jo.put('private_key','<private key>');
jo.put('fingerprint','<fingerprint>');
dbms_output.put_line(jo.to_string);
dbms_vector_chain.create_credential(
credential_name => 'OCI_CRED',
params => json(jo.to_string));
end;
/

declare
jo json_object_t;
begin
jo := json_object_t();
jo.put('user_ocid','ocid1.user.oc1..aabbalbbaa1112233aabbaabb1111222aa1111bb');

jo.put('tenancy_ocid','ocid1.tenancy.oc1..aaaaalbbbb1112233aaaabbaa1111222aaa111a')
;

jo.put('compartment_ocid','ocid1.compartment.oc1..ababalabab1112233abababab1111222a
ba11ab');
jo.put('private_key','AAAaaaBBB11112222333...AAA111AAABBB222aaa1a/+');
jo.put('fingerprint','01:1a:a1:aa:12:a1:12:1a:ab:12:01:ab:a1:12:ab:1a');
dbms_output.put_line(jo.to_string);
dbms_vector_chain.create_credential(
credential_name => 'OCI_CRED',
params => json(jo.to_string));
end;
/

-- select example

var params clob;

exec :params := '
{
"provider": "ocigenai",
"credential_name": "OCI_CRED",
"url": "<REST endpoint URL for text summarization service>",
"model": "<REST provider text summarization model name>"
}';

select dbms_vector_chain.utl_to_summary(
'A transaction is a logical, atomic unit of work that contains one or more SQL
statements.
An RDBMS must be able to group SQL statements so that they are either all
committed, which means they are applied to the database, or all rolled back,
which
means they are undone.
An illustration of the need for transactions is a funds transfer from a savings
account to
 a checking account. The transfer consists of the following separate operations:
1. Decrease the savings account.
2. Increase the checking account.
3. Record the transaction in the transaction journal.
Oracle Database guarantees that all three operations succeed or fail as a unit.
For
example, if a hardware failure prevents a statement in the transaction from
executing,
then the other statements must be rolled back.
Transactions set Oracle Database apart from a file system. If you
perform an atomic operation that updates several files, and if the system fails
halfway
through, then the files will not be consistent. In contrast, a transaction
moves an
Oracle database from one consistent state to another. The basic principle of a
transaction is all or nothing: an atomic operation succeeds or fails as a
whole.',
json(:params)) from dual;

-- PL/SQL example

declare
input clob;
params clob;
output clob;
begin
input := 'A transaction is a logical, atomic unit of work that contains one or
more SQL
statements.
An RDBMS must be able to group SQL statements so that they are either all
committed, which means they are applied to the database, or all rolled back,
which
means they are undone.
An illustration of the need for transactions is a funds transfer from a savings
account to
a checking account. The transfer consists of the following separate operations:
1. Decrease the savings account.
2. Increase the checking account.
3. Record the transaction in the transaction journal.
Oracle Database guarantees that all three operations succeed or fail as a unit.
For
example, if a hardware failure prevents a statement in the transaction from
executing,
then the other statements must be rolled back.
Transactions set Oracle Database apart from a file system. If you
perform an atomic operation that updates several files, and if the system fails
halfway
through, then the files will not be consistent. In contrast, a transaction
moves an
Oracle database from one consistent state to another. The basic principle of a
transaction is all or nothing: an atomic operation succeeds or fails as a
whole.';
params := '
{
"provider": "ocigenai",
"credential_name": "OCI_CRED",
"url": "<REST endpoint URL for text summarization service>",
"model": "<REST provider text summarization model name>"
}';
 output := dbms_vector_chain.utl_to_summary(input, json(params));
dbms_output.put_line(output);
if output is not null then
dbms_lob.freetemporary(output);
end if;
exception
when OTHERS THEN
DBMS_OUTPUT.PUT_LINE (SQLERRM);
DBMS_OUTPUT.PUT_LINE (SQLCODE);
end;
/

Replace the url and model values. Optionally, you can specify additional REST
provider-specific parameters.
{
"provider": "ocigenai",
"credential_name": "OCI_CRED",
"url": "https://generativeai.oci.example.com/summarizeText",
"model": "summarize.modelname",
"length": "MEDIUM",
"format": "PARAGRAPH",
"temperature": 1.0
}
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Create and Use Custom Vocabulary

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can see how to create and use your own vocabulary of tokens when chunking data.
You use the chunker helper function CREATE_VOCABULARY from the DBMS_VECTOR_CHAIN
package to load custom vocabulary.
This vocabulary file contains a list of tokens, recognized by your model's
tokenizer.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
conn sys/password@CDB_PDB as sysdba

CREATE TABLESPACE tbs1

DATAFILE 'tbs5.dbf' SIZE 20G AUTOEXTEND ON
EXTENT MANAGEMENT LOCAL
SEGMENT SPACE MANAGEMENT AUTO;

drop user docuser cascade;

create user docuser identified by docuser DEFAULT TABLESPACE tbs1 quota unlimited
on tbs1;
grant DB_DEVELOPER_ROLE to docuser;

create or replace directory VEC_DUMP as '/my_local_dir/';

grant read, write on directory VEC_DUMP to docuser;
commit;

Transfer the vocabulary file for your required model to the VEC_DUMP directory.
For example, if using WordPiece tokenization, you can download and transfer the
vocab.txt vocabulary file for "bert-base-uncased".

conn docuser/password@CDB_PDB;

SET ECHO ON
SET FEEDBACK 1
SET NUMWIDTH 10
SET LINESIZE 80
SET TRIMSPOOL ON
SET TAB OFF
SET PAGESIZE 10000
SET LONG 10000

CREATE TABLE doc_vocabtab(token nvarchar2(64))

ORGANIZATION EXTERNAL
(default directory VEC_DUMP
ACCESS PARAMETERS (RECORDS DELIMITED BY NEWLINE)
location ('bert-vocabulary-uncased.txt'));

DECLARE
vocab_params clob := '{"table_name" : "doc_vocabtab",
"column_name" : "token",
"vocabulary_name" : "doc_vocab",
"format" : "bert",
"cased" : false}';
BEGIN
dbms_vector_chain.create_vocabulary(json(vocab_params));
END;
/

After loading the token vocabulary, you can now use the BY VOCABULARY chunking mode
(with VECTOR_CHUNKS or UTL_TO_CHUNKS) to split data by counting the number of
tokens.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Create and Use Custom Language Data

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can create and use your own language-specific conditions (such as common
abbreviations) when chunking data.

Here, you use the chunker helper function CREATE_LANG_DATA from the
DBMS_VECTOR_CHAIN package to load the data file for Simplified Chinese. This data
file contains abbreviation tokens for your chosen language.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
conn sys/password@CDB_PDB as sysdba

CREATE TABLESPACE tbs1

DATAFILE 'tbs5.dbf' SIZE 20G AUTOEXTEND ON
EXTENT MANAGEMENT LOCAL
SEGMENT SPACE MANAGEMENT AUTO;

drop user docuser cascade;

create user docuser identified by docuser DEFAULT TABLESPACE tbs1 quota unlimited
on tbs1;
grant DB_DEVELOPER_ROLE to docuser;

create or replace directory VEC_DUMP as '/my_local_dir/';

grant read, write on directory VEC_DUMP to docuser;
commit;

conn docuser/password@CDB_PDB;
SET ECHO ON
SET FEEDBACK 1
SET NUMWIDTH 10
SET LINESIZE 80
SET TRIMSPOOL ON
SET TAB OFF
SET PAGESIZE 10000
SET LONG 10000

CREATE TABLE doc_langtab(token nvarchar2(64))

ORGANIZATION EXTERNAL
(default directory VEC_DUMP
ACCESS PARAMETERS (RECORDS DELIMITED BY NEWLINE CHARACTERSET AL32UTF8)
location ('dreoszhs.txt'));

DECLARE
lang_params clob := '{"table_name" : "doc_langtab",
"column_name" : "token",
"language" : "simplified chinese",
"preference_name" : "doc_lang_data"}';
BEGIN
dbms_vector_chain.create_lang_data(json(lang_params));
END;
/

After loading the language data, you can now use language-specific chunking by
specifying the LANGUAGE chunking parameter with VECTOR_CHUNKS or UTL_TO_CHUNKS.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Convert Text to Chunks With Custom Chunking Specifications

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A chunked output, especially for long and complex documents, sometimes loses
contextual meaning or coherence with its parent content. In this example, you can
see how to refine your chunks by applying custom chunking specifications.

Here, you use the VECTOR_CHUNKS SQL function or the UTL_TO_CHUNKS() PL/SQL function
from the DBMS_VECTOR_CHAIN package.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
conn sys/password@CDB_PDB as sysdba

CREATE TABLESPACE tbs1

DATAFILE 'tbs5.dbf' SIZE 20G AUTOEXTEND ON
EXTENT MANAGEMENT LOCAL
SEGMENT SPACE MANAGEMENT AUTO;

DROP USER docuser cascade;

CREATE USER docuser identified by docuser DEFAULT TABLESPACE tbs1 quota unlimited
on tbs1;
GRANT DB_DEVELOPER_ROLE to docuser;

CONN docuser/password@CDB_PDB

SET ECHO ON
SET FEEDBACK 1
SET NUMWIDTH 10
SET LINESIZE 80
SET TRIMSPOOL ON
SET TAB OFF
SET PAGESIZE 10000
SET LONG 10000

DROP TABLE IF EXISTS documentation_tab;

CREATE TABLE documentation_tab (

id NUMBER,
text VARCHAR2(2000));

INSERT INTO documentation_tab VALUES(1,

'Oracle AI Vector Search stores and indexes vector embeddings'||
' for fast retrieval and similarity search.'||CHR(10)||CHR(10)||
' About Oracle AI Vector Search'||CHR(10)||
' Vector Indexes are a new classification of specialized indexes'||
' that are designed for Artificial Intelligence (AI) workloads that allow'||
' you to query data based on semantics, rather than keywords.'||CHR(10)||CHR(10)||
' Why Use Oracle AI Vector Search?'||CHR(10)||
' The biggest benefit of Oracle AI Vector Search is that semantic search'||
' on unstructured data can be combined with relational search on business'||
' data in one single system.'||CHR(10));

COMMIT;

SET LINESIZE 1000;

SET PAGESIZE 200;
COLUMN doc FORMAT 999;
COLUMN id FORMAT 999;
COLUMN pos FORMAT 999;
COLUMN siz FORMAT 999;
COLUMN txt FORMAT a60;
COLUMN data FORMAT a80;

-- Call the VECTOR_CHUNKS SQL function and specify the following custom chunking
parameters:
SELECT D.id doc, C.chunk_offset pos, C.chunk_length siz, C.chunk_text txt
FROM documentation_tab D, VECTOR_CHUNKS(D.text
BY words
MAX 50
OVERLAP 0
SPLIT BY recursively
LANGUAGE american
NORMALIZE all) C;

-- To call the same operation using the UTL_TO_CHUNKS function from the
DBMS_VECTOR_CHAIN package, run:
SELECT D.id doc,
JSON_VALUE(C.column_value, '$.chunk_id' RETURNING NUMBER) AS id,
JSON_VALUE(C.column_value, '$.chunk_offset' RETURNING NUMBER) AS pos,
JSON_VALUE(C.column_value, '$.chunk_length' RETURNING NUMBER) AS siz,
JSON_VALUE(C.column_value, '$.chunk_data') AS txt
FROM documentation_tab D,
dbms_vector_chain.utl_to_chunks(D.text,
JSON('{"by":"words",
"max":"50",
"overlap":"0",
"split":"recursively",
"language":"american",
 "normalize":"all"}')) C;

-- To further clean up the chunking results, supply JSON parameters for

UTL_TO_CHUNKS:
SELECT C.*
FROM documentation_tab D,
dbms_vector_chain.utl_to_chunks(D.text,
JSON('{"by":"words",
"max":"50",
"overlap":"0",
"split":"recursively",
"language":"american",
"normalize":"all"}')) C;
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Explore Chunking Techniques and Examples

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Go through the link for detailed examples. One example I am showing it below:

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-- create a relational table

DROP TABLE IF EXISTS documentation_tab;

CREATE TABLE documentation_tab (
id NUMBER,
text CLOB);

-- create a local directory and store the document into the table

CREATE OR REPLACE DIRECTORY VEC_DUMP AS '/my_local_dir/';

CREATE OR REPLACE PROCEDURE my_clob_from_file(
p_dir in varchar2,
p_file in varchar2,
p_id in number
) AS
dest_loc CLOB;
v_bfile bfile := null;
v_lang_context number := dbms_lob.default_lang_ctx;
v_dest_offset integer := 1;
v_src_offset integer := 1;
v_warning number;
BEGIN
insert into documentation_tab values(p_id,empty_clob()) returning text
into dest_loc;

v_bfile := BFileName(p_dir, p_file);

dbms_lob.open(v_bfile, dbms_lob.lob_readonly);
dbms_lob.loadClobFromFile(
dest_loc,
v_bfile,
dbms_lob.lobmaxsize,
v_dest_offset,
v_src_offset,
873,
v_lang_context,
v_warning);
dbms_lob.close(v_bfile);
END my_clob_from_file;
/

show errors;

-- transform clob into chunks

exec my_clob_from_file('VEC_DUMP', 'ChineseDoc.txt', 1);

SELECT rownum as id, C.chunk_offset pos, C.chunk_length as siz,

REPLACE(SUBSTR(C.chunk_text,1,15),CHR(10),'_') as beg,
'...' as rng,
REPLACE(SUBSTR(C.chunk_text,-15),CHR(10),'_') as end
FROM documentation_tab D, VECTOR_CHUNKS(to_char(D.text) BY words
MAX 40
OVERLAP 0
SPLIT BY sentence
LANGUAGE "simplified chinese"
NORMALIZE none) C;
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Generate Text for a Prompt: PL/SQL Example

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In this example, you can see how to generate text for a given prompt by accessing
third-party text generation models.

A prompt can be an input string (such as a question that you ask an LLM or a
command), and can include results from a search.

You can use the described functions either from the DBMS_VECTOR or
DBMS_VECTOR_CHAIN package, depending on your use case.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
conn sys/password@CDB_PDB as sysdba

CREATE TABLESPACE tbs1

DATAFILE 'tbs5.dbf' SIZE 20G AUTOEXTEND ON
EXTENT MANAGEMENT LOCAL
SEGMENT SPACE MANAGEMENT AUTO;

DROP USER docuser cascade;

CREATE USER docuser identified by docuser DEFAULT TABLESPACE tbs1 quota unlimited
on tbs1;
GRANT DB_DEVELOPER_ROLE, create credential to docuser;

CONN docuser/password@CDB_PDB

SET ECHO ON
SET FEEDBACK 1
SET NUMWIDTH 10
SET LINESIZE 80
SET TRIMSPOOL ON
SET TAB OFF
SET PAGESIZE 10000
SET LONG 10000
EXEC UTL_HTTP.SET_PROXY('<proxy-hostname>:<proxy-port>');
BEGIN
DBMS_NETWORK_ACL_ADMIN.APPEND_HOST_ACE(
host => '*',
ace => xs$ace_type(privilege_list => xs$name_list('connect'),
principal_name => 'docuser',
principal_type => xs_acl.ptype_db));
END;
/

exec dbms_vector_chain.drop_credential('<credential name>');

declare
jo json_object_t;
begin
jo := json_object_t();
jo.put('access_token', '<access token>');
dbms_vector_chain.create_credential(
credential_name => '<credential name>',
params => json(jo.to_string));
end;
/

declare
jo json_object_t;
begin
jo := json_object_t();
jo.put('access_token', 'AbabA1B123aBc123AbabAb123a1a2ab');
dbms_vector_chain.create_credential(
credential_name => 'HF_CRED',
params => json(jo.to_string));
end;
/

-- select example

var params clob;

exec :params := '
{
"provider": "<REST provider>",
"credential_name": "<credential name>",
"url": "<REST endpoint URL for text generation service>",
"model": "<REST provider text generation model name>"
}';

select dbms_vector_chain.utl_to_generate_text(
'What is Oracle Text?',json(:params)) from dual;

-- PL/SQL example

declare
input clob;
params clob;
output clob;
begin
input := 'What is Oracle Text?';
params := '
{
"provider": "<REST provider>",
"credential_name": "<credential name>",
"url": "<REST endpoint URL for text generation service>",
 "model": "<REST provider text generation model name>"
}';
output := dbms_vector_chain.utl_to_generate_text(input, json(params));
dbms_output.put_line(output);
if output is not null then
dbms_lob.freetemporary(output);
end if;
exception
when OTHERS THEN
DBMS_OUTPUT.PUT_LINE (SQLERRM);
DBMS_OUTPUT.PUT_LINE (SQLCODE);
end;
/

Refer to the below link for the JSON syntax for various providers.
https://docs.oracle.com/en/database/oracle/oracle-database/23/vecse/generate-text-
prompt-pl-sql-example.html

###########################################################
Vector Indexes
###########################################################

There are two ways to make vector searches faster:

- Reduce the search scope by clustering vectors (nearest neighbors) into
structures based on certain attributes and restricting the search to closest
clusters.
- Reduce the vector size by reducing the number of bits representing vectors
values.

Oracle AI Vector Search supports the following categories of vector indexing

methods based on approximate nearest-neighbors (ANN) search:
- In-Memory Neighbor Graph Vector Index
- Neighbor Partition Vector Index

The distance function used to create and search the index should be the one
recommended by the embedding model used to create the vectors.
You can specify this distance function at the time of index creation or when you
perform a similarity search using the VECTOR_DISTANCE() function.
If you use a different distance function than the one used to create the index, an
exact match is triggered because you cannot use the index in this case.

Oracle AI Vector Search indexes supports the same distance metrics as the
VECTOR_DISTANCE() function.
COSINE is the default metric if you do not specify any metric at the time of index
creation or during a similarity search
using the VECTOR_DISTANCE() function.

You should always define the distance metric in an index based on the distance
metric used by the embedding model you are using

In-Memory Neighbor Graph Vector Index

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Hierarchical Navigable Small World (HNSW) is the only type of In-Memory Neighbor
Graph vector index supported.
HNSW graphs are very efficient indexes for vector approximate similarity search.
HNSW graphs are structured using principles from small world networks along with
layered hierarchical organization.
CREATE VECTOR INDEX vector_index_name
ON table_name ( vector_column )
[ GLOBAL ] ORGANIZATION INMEMORY NEIGHBOR GRAPH
[ WITH ] [ DISTANCE metric name ]
[ WITH TARGET ACCURACY percentage_value ]
[ PARAMETERS ( TYPE
{ HNSW , { NEIGHBORS max_closest_vectors_connected
| M max_closest_vectors_connected }

, EFCONSTRUCTION max_candidates_to_consider
|
IVF , { NEIGHBOR PARTITIONS number_of_partitions
| SAMPLE_PER_PARTITION number_of_samples
| MIN_VECTORS_PER_PARTITION
min_number_of_vectors_per_partition }
}
]
[ PARALLEL degree_of_parallelism ]

Neighbor Partition Vector Index

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Inverted File Flat (IVF) index is the only type of Neighbor Partition vector index
supported.
Inverted File Flat Index (IVF Flat or simply IVF) is a partitioned-based index
which balance high search quality with reasonable speed.

The default type of index created for a Neighbor Partition vector index is Inverted
File Flat (IVF) vector index. The IVF index is a technique designed to enhance
search efficiency by narrowing the search area through the use of neighbor
partitions or clusters.

CREATE VECTOR INDEX <vector index name>

ON <table name> ( <vector column> )
[GLOBAL] ORGANIZATION NEIGHBOR PARTITIONS
[WITH] [DISTANCE <metric name>]
[WITH TARGET ACCURACY <percentage value>
[PARAMETERS ( TYPE IVF, { NEIGHBOR PARTITIONS <number of partitions> |
SAMPLE_PER_PARTITION
<number of samples> | MIN_VECTORS_PER_PARTITION <minimum number of vectors per
partition>
})]]
[PARALLEL <degree of parallelism>];

CREATE VECTOR INDEX galaxies_ivf_idx ON galaxies (embedding) ORGANIZATION NEIGHBOR

PARTITIONS
DISTANCE COSINE
WITH TARGET ACCURACY 95;

CREATE VECTOR INDEX galaxies_ivf_idx ON galaxies (embedding) ORGANIZATION NEIGHBOR

PARTITIONS
DISTANCE COSINE
WITH TARGET ACCURACY 90 PARAMETERS (type IVF, neighbor partitions 10);

###########################################################
Work with Retrieval Augmented Generation
###########################################################
Oracle AI Vector Search supports Retrieval Augmented Generation (RAG) to enable
sophisticated queries that can combine vectors
with relational data, graph data, spatial data, and JSON collections. By
communicating with LLMs through the implementation of RAG,
the knowledge of LLMs is increased with business data found through AI Vector
Search.

Compliment LLMs with Oracle AI Vector Search

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Retrieval Augmented Generation (RAG) is an approach developed to address the
limitations of LLMs.

Using Retrieval Augmented Generation (RAG) can mitigate the inaccuracies and
hallucinations faced when using Large Language Models (LLMs).
Oracle AI Vector Search enables RAG through the use of popular frameworks and
PL/SQL APIs.

The primary problem with Large Language Models (LLMs) like GPT (Generative
Pretrained Transformer) is that they generate responses
based solely on the patterns and data they were trained on up to the point of their
last update.

This means that they inherently lack the ability to access or incorporate new,
real-time information after their training is cut off,
potentially limiting their responses to outdated or incomplete information.

LLMs do not know about your private company data.

Consequently, LLMs can make up answers (hallucinate) when they do not have enough
relevant and up-to-date facts.

By providing your LLM with up-to-date facts from your company, you can minimize the
probability that an LLM will make up answers (hallucinate).

Read the RAG WorkFlow:

https://docs.oracle.com/en/database/oracle/oracle-database/23/vecse/compliment-
llms-oracle-ai-vector-search.html

RAG combines the strengths of pretrained language models with the ability to
retrieve recent and accurate information from a dataset or
database in real-time during the generation of responses.

Here is how RAG improves upon the issues with traditional LLMs:

Access to External and Private Information:

``````````````````````````````````````````````
RAG can pull in data from external and private sources during its response
generation process. This allows it to provide answers that are up-to-date and
grounded in the latest available information, which is crucial for queries
requiring current knowledge or specific details not included in its original
training data.

Factually More Accurate and Detailed Responses:

``````````````````````````````````````````````
While traditional LLMs are trained on older data, RAG incorporates real-time
retrieved information, meaning that generated responses are not only contextually
rich but also factually more up-to-date and accurate as time goes on. This is
particularly beneficial for queries that require precision and detail, such as
scientific facts, historical data, or specific statistics.
Reduced Hallucination:
```````````````````````
LLMs can sometimes "hallucinate" information, as in generate plausible but false or
unverified content. RAG mitigates this by grounding responses in retrieved
documents, thereby enhancing the reliability of the information provided.

Oracle AI Vector Search enables RAG within Oracle Database using the
DBMS_VECTOR_CHAIN PL/SQL package.
You can also implement RAG externally by using popular frameworks such as
LangChain.

LangChain is a popular open source framework that encapsulates popular LLMs, vector
databases, document stores, and embedding models.
DBMS_VECTOR_CHAIN is a PL/SQL package that provides the ability to create RAG
solutions, all within the database. With DBMS_VECTOR_CHAIN, your
data never needs to leave the security of Oracle Database.

>>>>>>>>>>>>>>>>>>>>>>>>>>
From Chapter4, I am going through Online. Later, I will take notes and update
here.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@
Vector Keywords
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@

Vector Data Type

Vector Dimensions
Vector Distance -- "DOT", "COSINE", "EUCLIDEAN"
Vector Index -- Vector Pool, In-Memory Neighbor Graph Vector Index (Hierarchical
Navigable Small World (HNSW)) , <-- use these indexes for running similarity
searches over huge vector spaces.
Similarity Search
Clustered
Open Neural Network Exchange (ONNX)

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@
Important Points for Speech
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@

One of the biggest benefits of Oracle AI Vector Search is that semantic search on
unstructured data can be combined with relational search on business data in one
single system.

This is not only powerful but also significantly more effective because you don't
need to add a specialized vector database, eliminating the pain of data
fragmentation between multiple systems.