Skip to content

[FEATURE]Setup Spark Connector Module #1721

@vamsimanohar

Description

@vamsimanohar

Is your feature request related to a problem?
Add spark datasource support

  1. Setup Basic Spark Connector Module.
  2. Support EMR connector
  3. Support Livy connector

Proposed Approches:

  1. Synchronous
    image (2)

  2. Asynchronous
    image (3)

Comparison

  Synchronous Async with Opensearch index
Long Execution/Waiting Time Yes (min time for query ~30sec) No
Query limitations Queries which need timeout more than 5 mins won't work NA
Index management required? Yes Yes

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions